CLI tool to convert raster to pmtiles

[daniel-j-h]

The protomaps format allows for raster tile hierarchies.

We've been working with bundling up raster tiles recently and ended up using the python library from https://github.com/protomaps/PMTiles to do so (see also example in https://github.com/protomaps/PMTiles/blob/e1228f5df1a4ac852e0117dc08c8813089a5c8af/python/examples/create_raster_example.py).

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Should the convert sub-command be extended to support raster, too?

[bdon]

It's a good feature request - anything that interacts with GeoTIFF or raster formats will probably depend on GDAL, which is out of scope for go-pmtiles. I've heard the best outputs come from rasterio and rio-mbtiles here: https://docs.protomaps.com/pmtiles/create#geotiff after which you can use go-pmtiles convert

so maybe we just need to make a rio-pmtiles equivalent?

[bmcbride]

A rio-pmtiles equivalent to directly convert GeoTIFF to PMTiles would be awesome, especially if the functionality was able to be added to go-pmtiles, giving the user the option to convert either MBTiles or GeoTIFF.

[bdon]

It's unlikely this would be part of go-pmtiles because it would need to depend on GDAL. A python package installed through pip run via a rasterio command e.g. rio pmtiles ... I think would be the easiest path.

[bdon]

issue transferred to main PMTiles repository

[mabhub]

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Isn't it already kinda simple by the use of mb-util which allow to convert tiles file tree to mbtiles, then converting it to pmtiles with pmtiles convert?

$ mb-util my-tiles-root-directory my_tiles.mbtiles
$ pmtiles convert my_tiles.mbtiles my_tiles.pmtiles

[neodescis]

What would have made that workflow simpler is a cli tool like pmtiles convert but instead of converting mbtiles into pmtiles it would take e.g. a slippy map tile hierarchy as in z/x/y.ext or a GeoTIF raster and turn it into a pmtiles file.

Isn't it already kinda simple by the use of mb-util which allow to convert tiles file tree to mbtiles, then converting it to pmtiles with pmtiles convert?

$ mb-util my-tiles-root-directory my_tiles.mbtiles
$ pmtiles convert my_tiles.mbtiles my_tiles.pmtiles

This is the route I went to convert some terrain tiles. It is indeed pretty simple, but:

1. This is not an obvious path to take for someone who doesn't know anything about mbtiles
2. It seems bad for pmtiles to depend on mb-util like this to get archives created

[larsmaxfield]

@bdon what would you think of PR which implements a solution like how @neodescis and @mabhub describe above?

Specifically, converting raster to .pmtiles with a temporary .mbtiles file which is automatically created and deleted? I'm doing a similar approach in a script — I could look into writing a function in PMTiles.

[bdon]

@JesseCrocker contributed this script: https://gist.github.com/JesseCrocker/4fee23a262cdd454d14e95f2fb25137f

I would prefer a CLI that does not need an intermediate mbtiles because that will be slower for large archives. Could you try the Gist above?

[larsmaxfield]

@bdon thanks for the feedback. The gist wasn't applicable for me because my need is to convert a z/x/y.ext directory of tiles (a.k.a. "file tree" or "slippy map tile hierarchy").

I created a disk_to_pmtiles method which directly reads the tiles and writes them to PMTiles — here's the fork diff: main...larsmaxfield:PMTiles:feature/disk_to_pmtiles.

The disk_to_pmtiles method uses elements of the original disk_to_mbtiles from mbutil combined with your mbtiles_to_pmtiles. It supports different tiling schemes like disk_to_mbtiles does, and I added an 'auto' feature for maxzoom whereby the maximum zoom is determined from the directory.

Let me know if you're OK with a PR for this.

[bdon]

I originally stopped development on the python CLI functions because https://github.com/protomaps/go-pmtiles replaces it and should be much faster for large tilesets. disk_to_pmtiles sounds useful if you have a small to medium size tileset, but likely doesn't work well for tens of millions of tiles. Does that match your use case?

[larsmaxfield]

Ah I see. The Python functions fit nice into my Python-based pipeline.

Indeed my use case is for tilesets at most up to z=10. And indeed disk_to_pmtiles is not appropriate for large tilesets. It takes my computer roughly 7 minutes at z=9, which extrapolates to a few hundred hours for z=15.

[bdon]

Well I think it will be useful for the small use cases like that. Can you open the PR?

[larsmaxfield]

Nice. I agree.