Migrate pmtiles from Python->Go

[orblivion]

See the description of the associated milestone for some technical background.

This task will be for migrating the pmtiles file serving from Python to Go. The pmtiles files contain the vector map that makes up the visible, scrollable part of the map (i.e. not the search index).

One detail to avoid confusion: You would think nginx would just handle this as a static file. However, protomaps.js needs subsets of the file to show parts of the map. The usual way to request a subset of a file is a Range header. Sandstorm being as oppressive as it is, doesn't pass on range headers (they weren't against it, they just never got around to implementing it). So instead of range headers, I hacked around it by specifying the range in the query string. This hack requires implementing the file endpoint myself. (See the corresponding frontend hack in dependencies/protomaps.js/protomaps.patch if you're curious)

So we have an endpoint in the Python code to return a subset of a given pmtiles file:

desert-atlas/server.py

Lines 596 to 611 in b97fe81

	if url.path.endswith('.pmtiles'):
	fname = url.path.split('/')[-1]
	qs = urllib.parse.parse_qs(url.query)
	self.send_response(HTTPStatus.PARTIAL_CONTENT)
	self.send_header('Content-Type','application/pbf')
	# Hack, until we get range headers in Sandstorm
	if 'rangeFirst' in qs:
	first, last = int(qs['rangeFirst'][0]), int(qs['rangeLast'][0])
	else:
	first, last = 0, len(filemaps[fname])
	# In case we want to go back to the original for a second
	#first, last = byterange(self.headers['Range'])
	self.send_header('Content-Length',str(last-first+1))
	self.end_headers()
	self.wfile.write(filemaps[fname][first:last+1])
	return

• I think this endpoint should be implemented as a "side-by-side" endpoint (i.e. directly accessible from nginx).

But there's a small catch. Part of the implementation of this endpoint is the filemaps global variable which contains the pmtiles files as mmaped data. If you search the python file, you'll find this variable referred to in a couple other places. Both of these, I think, should be replaced by "internal" calls (i.e. Python server calls Go server):

• The function update_filemaps() which looks at the filesystem for any newly downloaded or deleted pmtiles files and updates the filemaps variable accordingly.

• I.e. when a region is downloaded or deleted in the Python server, instead of calling update_filemaps() to update its list of pmtiles files, it should call an endpoint on the Go server that tells it to update its list of pmtiles files. There is an mmap module for Go as there is for Python, so just do the equivalent on that side.

• A couple times when it uses a list comprehension to get all of the names of the files from filemaps.

• Replace this with another endpoint in the Go server that Python calls to get the list of the filenames.

I'm open to other ideas, but this is what comes to mind as a way to extract a piece of independent functionality. You could make a case that this only touches part of the functionality, since Python will still add the .pmtiles to the filesystem in the download_map() function but I think that's probably okay to migrate later. The filesystem can be another interface between Python and Go.

• Please add tests and stuff as well of course. However, I would not take the time to write tests for the internal http handler functions as such, since http tests are annoying and we'll be stripping those handlers away soon anyway. But everything else, go ahead and test.