Allow replacing fetch tile and decode image for local Dem manager

[HarelM]

I've added the ability to replace decode image and fetch tile.
I've also moved some stuff around and did some minor renaming.
Let me know what you think.

This can facilitate for #212 for the case of local dem manager, but not for remote dem manager.

[HarelM]

CC: @acalcutt

[HarelM]

The following script seems to work as expected I think, I tried to use the exported parts of this library

import { writeFileSync } from "fs";
import type { DemTile, Encoding } from "./dist/types";
import {default as mlcontour} from "./dist/index.mjs";
import { PNG } from "pngjs";

async function decodeImageNode(
    blob: Blob,
    encoding: Encoding,
    abortController: AbortController,
  ): Promise<DemTile> {
    const buffer = await blob.arrayBuffer();
    const png = PNG.sync.read(Buffer.from(buffer));
    const parsed = mlcontour.decodeParsedImage(png.width, png.height, encoding, png.data as any as Uint8ClampedArray);
    if (Boolean(abortController?.signal?.aborted)) return null as any as DemTile;
    return parsed;
  }

let manager = new mlcontour.LocalDemManager({
    demUrlPattern: "https://www.example.com/{z}/{x}/{y}.png", // This is the URL of the DEM tiles
    cacheSize: 100,
    encoding: "mapbox",
    maxzoom: 12,
    timeoutMs: 10000,
    decodeImage: decodeImageNode,
});

manager.fetchContourTile(12,2446,1655, {levels: [10]}, new AbortController()).then((tile) => {
    writeFileSync("12-2446-1655.mvt", Buffer.from(tile.arrayBuffer));
});

[msbarry]

Looks good overall! A couple of small suggestions, but can you also run npm run format and add a test with a local DEM manager with custom function for fetch and resolve?

[HarelM]

I've done the formatting and added a e2e test, although it might be better to move it to "local-dem-manager.test.ts" as a unit test for that class, not sure, let me know what you prefer...

[msbarry]

Thank you! Do you think it would make sense to upstream a top-level node.js-ready entrypoint like #359 (comment) ? Or would it be easier to just to keep it in the other repo?

[HarelM]

I think @acalcutt is working on a script that can allow running this using a shell script to be added to the bin folder of this package.
Note that the constructor arguments change is a breaking change in terms of public API so you might need to release a major version.

[acalcutt]

I made this code to get all the tiles under a specific parent tile which is based off similar code to the first post that @HarelM had given me
https://github.com/acalcutt/maplibre-contour-pmtiles/blob/main/src/generate-countour-tile-batch.ts
Usage: npx tsx ./src/generate-countour-tile-batch.ts --x <x> --y <y> --z <z> --sFile <sFile> --sEncoding <sEncoding> --sMaxZoom <sMaxZoom> --increment <increment> --oMaxZoom <oMaxZoom> --oDir <oDir>"

and this batch script the gets all tiles at a specific level and runs the above script for each in parallel
https://github.com/acalcutt/maplibre-contour-pmtiles/blob/main/scripts/generate_contour_tiles.sh

Usage: ./generate_contour_tiles.sh --sFile <path> --oDir <path> [options]
 Options:
  --increment <value> Increment value (default: 10)
  --sMaxZoom <value> Source Max Zoom (default: 8)
  --sEncoding <encoding> Source Encoding (default: mapbox) (must be 'mapbox' or 'terrarium')
  --sFile <path>  TerrainRGB or Terrarium PMTiles File Path or URL (REQUIRED)
  --oDir <path>  Output Directory (REQUIRED)
  --oMaxZoom <value> Output Max Zoom (default: 8)
  --oMinZoom <value> Output Min Zoom (default: 5)
  -v|--verbose  Enable verbose output
  -h|--help  Show this usage statement

I used it to make this z5-z11 contour at 10m
increments https://tiles.wifidb.net/data/JAXA_AW3D30_2024_contour_z5-Z11_vector/#5/0/0
(though I am making a new one z5-z12 with slightly lower increments for the lower zoom levels)

It shouldn't be hard to move that to this new method of fetch tile method I wouldn't think.

[acalcutt]

Would it be possible to override 'fetchTile' in the same way as decodeImage?

[HarelM]

It is part of this PR, yes.

[acalcutt]

ah, nice. i think i see in the tests how it would be used

[acalcutt]

I was trying to use your maplibre-contour changes with this code,
https://github.com/WifiDB/maplibre-contour/blob/ea60d710750c389ed05c0443a0873b18b1373faa/scripts/generate-countour-tile-pyramid.ts
but it keeps complaining about

SyntaxError: The requested module '../dist/index' does not provide an export named 'default'
    at ModuleJob._instantiate (node:internal/modules/esm/module_job:132:21)
    at async ModuleJob.run (node:internal/modules/esm/module_job:214:5)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:113:12)

but when i look at those index files the do seem like they have a default export... I did try to delete dist and recreated but it didn't help

The other thing is I noticed $url only seems to be the only parameter for getTile,. It would be nice if zxy parameters where given to so it doesn't need to be split out of the url

[HarelM]

Did you make sure to rebuild dist folder?
I think I managed to get things working as to what I wrote in the comment here:
#359 (comment)

I'll take a look later tonight to see if I can make it work.
@msbarry any chance you can publish the new version to npm?

[HarelM]

I've updated the above comment to something that is working for me, I'm not sure what's not well defined in terms of typescript, but it can run using tsx, vscode doesn't like what I wrote though...

[acalcutt]

Nice, I was able to get my local pmtiles script to work with this change

I modified the code I posted above that builds all tiles under a parent tile to work with this new feature
https://github.com/WifiDB/maplibre-contour/blob/fetch_test/scripts/generate-countour-tile-pyramid.ts

I also put the node/pmtiles functions I had made into it's own file that can be referenced
https://github.com/WifiDB/maplibre-contour/blob/fetch_test/scripts/node-pmtiles-adapter.ts

In 'generate-countour-tile-pyramid.ts' I added a switch if demUrl starts with pmtiles:// so it should be able to either work with pmtiles or a regular http(s) url https://github.com/WifiDB/maplibre-contour/blob/fetch_test/scripts/generate-countour-tile-pyramid.ts#L218-L240,

it should start with something like this for pmtiles
npx tsx generate-countour-tile-pyramid.ts --x 17 --y 6 --z 5 --demUrl pmtiles:///path/to/dem.pmtiles --sEncoding terrarium --sMaxZoom 8 --increment 10 --oMaxZoom 8 --oDir ./out
or
npx tsx generate-countour-tile-pyramid.ts --x 17 --y 6 --z 5 --demUrl pmtiles://https://wifidb.net/demo/pmtiles/sources/gebco_terrarium0-8.pmtiles --sEncoding terrarium --sMaxZoom 8 --increment 10 --oMaxZoom 8 --oDir ./out

or for a tile url
npx tsx generate-countour-tile-pyramid.ts --x 17 --y 6 --z 5 --demUrl https://url/of/dem/source/{z}/{x}/{y}.png --sEncoding terrarium --sMaxZoom 8 --increment 10 --oMaxZoom 8 --oDir ./out

[msbarry]

OK I just published a 0.1.0 release: https://www.npmjs.com/package/maplibre-contour/v/0.1.0

[HarelM]

Thanks @msbarry! I was able to create a new project (external to maplibre-contour) and use the following code to generate a tile locally.
Note that the types are used from the dist folder which can create a breaking change if you change internal implementation.
I would advise to use something like dts-bundle-generator or an equivalent to bundle all the types into a single index.d.ts file so that everything is exported from the same path (I wish typescript folks would implement that already and avoid the need to write it in every package I maintain, but that's a different story).

import { writeFileSync } from "fs";
import mlcontour from "maplibre-contour";
import { DemTile, Encoding } from "maplibre-contour/dist/types";
import { PNG } from "pngjs";

async function decodeImageNode(
    blob: Blob,
    encoding: Encoding,
    abortController: AbortController,
  ): Promise<DemTile> {
    const buffer = await blob.arrayBuffer();
    const png = PNG.sync.read(Buffer.from(buffer));
    const parsed = mlcontour.decodeParsedImage(png.width, png.height, encoding, png.data as any as Uint8ClampedArray);
    if (Boolean(abortController?.signal?.aborted)) return null as unknown as DemTile;
    return parsed;
  }

let manager = new mlcontour.LocalDemManager({
    demUrlPattern: "https://www.expample.com/{z}/{x}/{y}.png", // This is the URL of the DEM tiles
    cacheSize: 100,
    encoding: "mapbox",
    maxzoom: 12,
    timeoutMs: 10000,
    decodeImage: decodeImageNode,
});

manager.fetchContourTile(12,2446,1655, {levels: [10]}, new AbortController()).then((tile) => {
    writeFileSync("12-2446-1655.mvt", Buffer.from(tile.arrayBuffer));
});