Derive progress % from logs & running from container

[musicformellons]

I've successfully ran planetiler on a planet file using large RAM machines. I'm now trying to save some costs by running on smaller RAM machines. The process still runs but using like 1.5% of CPU, free RAM is at 1.5%. I'm wondering if this process is still making progress. Can you help me interpret the logs as in 'what defines best overall progress', and can I somehow derive an expected time to completion?

Here is a recent fragment of the logs:

{"level":"info","time":1709616588,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 817M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616588,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (408/53k) -> write( 0%)"}
{"level":"info","time":1709616598,"message":"stdOut log: \u001b[m\u001b[0m29:25:27 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   85/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  491/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616598,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 822M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616598,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (909/53k) -> write( 0%)"}
{"level":"info","time":1709616609,"message":"stdOut log: \u001b[m\u001b[0m29:25:37 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   76/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  589/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616609,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 824M/9.5G direct: 4M postGC: \u001b[34m898M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616609,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  0%) -> (439/53k) -> write( 0%)"}
{"level":"info","time":1709616619,"message":"stdOut log: \u001b[m\u001b[0m29:25:47 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   86/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  568/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616619,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 823M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616619,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  1%) -> (1.5k/53k) -> write( 0%)"}
{"level":"info","time":1709616629,"message":"stdOut log: \u001b[m\u001b[0m29:25:58 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%  108/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  795/s ] 55G   blocks: [   2M  86%    0/s ]"}
{"level":"info","time":1709616629,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 820M/9.5G direct: 4M postGC: \u001b[34m899M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616629,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 0%  1%) ->  (1k/53k) -> write( 0%)"}
{"level":"info","time":1709616639,"message":"stdOut log: \u001b[m\u001b[0m29:26:08 INF [osm_pass2] -  nodes: [  16B 100%    0/s ] 226G  ways: \u001b[32m[  13M   1%   79/s ]\u001b[0m rels: [    0   0%    0/s ] features: [ 700M  484/s ] 55G   blocks: [   2M  86%   <1/s ]"}
{"level":"info","time":1709616639,"message":"stdOut log:     cpus: \u001b[34m0\u001b[0m gc:  0% heap: 826M/9.5G direct: 4M postGC: \u001b[34m905M\u001b[0m relInfo: 351M mpGeoms: 151M "}
{"level":"info","time":1709616639,"message":"stdOut log:     read( 0%) ->   (11/13) -> process( 1%  1%) -> (1.1k/53k) -> write( 0%)"}

Another thing is that I'm running from docker container. I see instructions just mentioning running on straight java. Anything against running in a container (I do have the impression performance is worse than what I see in your performance table...)?

[msbarry]

The nodes: [ ... x% ... ] ways: [ ... y% ...] rels: [ ... z% ... ] part of the logs tells you how far you are through those 3 elements. When planetiler starts it also prints a list of the phases it's going to run (sort, archive write, etc.). There is similar logging from each of those phases. Planetiler doesn't try to guess how long they'll each take to give overall % complete though, although you could derive it from a few runs on the hardware you are working on.

There's also an argument to push metrics to a prometheus push gateway although I might get rid of that because I don't really think anyone is using it.

Another thing is that I'm running from docker container. I see instructions just mentioning running on straight java. Anything against running in a container (I do have the impression performance is worse than what I see in your performance table...)?

Running in a container is fine, the instructions explain how to run using either java or a container. It might be slower on a machine where the container runs inside a vm (like a mac) but on linux the performance should be the same. Your performance is probably negatively impacted by other settings. What size container and arguments are you running with?

[musicformellons]

I'm running with this:

docker run -m 10g -e JAVA_TOOL_OPTIONS='-XX:+UseParallelGC -Xmx10g' -v /mnt/data:/data myPlaneTilerImage --force --building_merge_z13=true --output=/data/planet/planet.pmtiles --download --osm-path=/planet --bounds=planet --nodemap-type=array --storage=mmap

on a 231GB planet file using a 16GB RAM machine with 2.5TB harddisk.

Logs:

- ✓ 76G storage on /data (/dev/sdb) requested for read phase disk, 2.4T available
  - 76G used for temporary node location cache
  - 82k used for temporary multipolygon geometry cache
  - 1.7M used for temporary feature storage
- ✓ 2.6M storage on /data (/dev/sdb) requested for write phase disk, 2.4T available
  - 1.7M used for temporary feature storage
  - 899k used for archive output
- ✓ 76G storage on /data (/dev/sdb) requested for read phase, 2.4T available
  - 76G used for array node location cache (switch to sparsearray to reduce size)
  - 82k used for multipolygon way geometries
- ✓ 161k JVM heap requested for read phase, 9.5G available
  - 161k used for temporary profile storage
- Planetiler will use ~76G memory-mapped files for node locations and multipolygon geometries but the OS only
 has 1.1G available to cache pages, this may slow the import down. To speed up, run on a machine with more
 memory or reduce the -Xmx setting.

[msbarry]

Ok got it, it's going to be very slow reading OSM ways because it writes out the node location file, then does memory-mapped reads from it to get the location of each node in each way. The OS caches pages outside of the JVM heap, but there's only ~1GB available between the JVM heap and container memory limit to cache those files.

To run faster, you need to increase the difference between the container memory limit and JVM heap size, either by reducing -Xmx setting to maybe -Xmx6g or increase container memory limit to -m 15g or something.