nominatim replication backlog

[mickutz]

Hi all,

I'm a bit confused as to how "nominatim replication" works.

This is the relevant part of my .env file

NOMINATIM_REPLICATION_URL="https://planet.openstreetmap.org/replication/hour/"
NOMINATIM_REPLICATION_UPDATE_INTERVAL=7200
NOMINATIM_REPLICATION_RECHECK_INTERVAL=90

Initial --check-for-updates returned:

root@host:/home/nominatim/nominatim-planet# nominatim replication --check-for-updates
2021-06-27 12:58:41: Using project directory: /home/nominatim/nominatim-planet
2021-06-27 12:58:41: New data available (76561 => 77045).

Than ran:

nominatim replication -j 4 --osm2pgsql-cache 2000 --project-dir /home/nominatim/nominatim-planet/ --once

and this is a sanitised version of the output:

2021-06-27 12:59:33: Using project directory: /home/nominatim/nominatim-planet
2021-06-27 12:59:37  osm2pgsql version 1.4.1
2021-06-27 12:59:37  Database version: 12.7 (Ubuntu 12.7-0ubuntu0.20.04.1)
2021-06-27 12:59:37  PostGIS version: 3.0
2021-06-27 12:59:37  Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
2021-06-27 12:59:37  Parsing gazetteer style file '/usr/local/etc/nominatim/import-extratags.style'.
    Processing: Node(310k 310.0k/s) Way(0k 0.00k/s) Relation(0 0.0/s)
    ............
2021-06-27 13:02:31  Reading input files done in 174s (2m 54s).               
2021-06-27 13:02:31    Processed 419254 nodes in 52s - 8k/s
2021-06-27 13:02:31    Processed 57263 ways in 85s (1m 25s) - 674/s
2021-06-27 13:02:31    Processed 1545 relations in 37s - 42/s
2021-06-27 13:04:16  node cache: stored: 389213(100.00%), storage efficiency: 72.99% (dense blocks: 40, sparse nodes: 102772), hit rate: 20.97%
2021-06-27 13:04:16  osm2pgsql took 279s (4m 39s) overall.
2021-06-27 13:04:16: Starting indexing boundaries using 4 threads
2021-06-27 13:04:16: Starting boundaries rank 4 (using batch size 1)
2021-06-27 13:04:16: Done 0/0 in 0 @ 0.000 per second - FINISHED boundaries rank 4

......

2021-06-27 13:04:16: Starting boundaries rank 8 (using batch size 1)
2021-06-27 13:04:17: Done 2/2 in 0 @ 2.795 per second - FINISHED boundaries rank 8

2021-06-27 13:04:17: Starting boundaries rank 9 (using batch size 1)
2021-06-27 13:04:17: Done 0/0 in 0 @ 0.000 per second - FINISHED boundaries rank 9

......

2021-06-27 13:04:22: Starting boundaries rank 25 (using batch size 1)
2021-06-27 13:04:22: Done 2/2 in 0 @ 5.947 per second - FINISHED boundaries rank 25

2021-06-27 13:04:22: Starting indexing rank (0 to 30) using 4 threads
2021-06-27 13:04:22: Starting rank 1 (using batch size 1)
2021-06-27 13:04:22: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 1

.......

2021-06-27 13:04:22: Starting rank 5 (using batch size 1)
2021-06-27 13:04:22: Done 13/13 in 0 @ 394.274 per second - FINISHED rank 5

........

2021-06-27 13:06:53: Starting rank 29 (using batch size 1)
2021-06-27 13:06:53: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 29

2021-06-27 13:06:53: Starting rank 0 (using batch size 1)
2021-06-27 13:07:19: Done 1254/1254 in 25 @ 48.471 per second - FINISHED rank 0

2021-06-27 13:07:19: Starting interpolation lines (location_property_osmline) (using batch size 20)
2021-06-27 13:07:21: Done 27/27 in 1 @ 17.128 per second - FINISHED interpolation lines (location_property_osmline)

2021-06-27 13:07:21: Starting rank 30 (using batch size 20)
2021-06-27 13:10:23: Done 27255/27255 in 181 @ 149.783 per second - FINISHED rank 30

2021-06-27 13:10:23: Update completed. Import: 0:04:42. Indexing: 0:06:07 Total: 0:10:49. Remaining backlog: 20 days, 3:10:23.

A subsequent check for updates returns:

root@host:/home/nominatim/nominatim-planet# nominatim replication --check-for-updates
2021-06-27 13:16:23: Using project directory: /home/nominatim/nominatim-planet
2021-06-27 13:16:23: New data available (76563 => 77046).

Now, am I wrong in thinking it should've updated the data to today instead of doing just 2 blocks? I only want to run one update a week on my build, is there an easy way to achieve this?
Also, is there an issue running replication with /hour/ and then switching to /day/?

Import was done using 3.7.0 and still on the same version.

[lonvia]

Nominatim downloads data only up to a certain size limit. This can be set with NOMINATIM_REPLICATION_MAX_DIFF (size in MB). You can increase the value to get more data in one round. 8000 should for most cases be sufficient to catch up for a week. But you might still want to run the update in a loop until it does report that all data has been downloaded and applied. Do not set the value to more than 1/3 of your available RAM or you risk running out of memory in the process.

In my experience it is better to run the updates in smaller steps. So in the case of doing updates once a week, apply the 7 daily diffs sequentially. This is what automatically happens when you use the default setting of NOMINATIM_REPLICATION_MAX_DIFF with daily diffs.

You can switch from hourlies to dailies and back any time. Just change the NOMINATIM_REPLICATION_URL in your local settings and then run nominatim replication --init again.

[mickutz]

Thanks @lonvia , I appreciate it, that seems to do the job.

Is there a way to optimise the node processing stage, as that's the bottleneck now? Even though I'm running replication with -j 4, the machine is pretty much in idle state.