Nominatim catch-up updates do not finish ever

[Zverik]

I have installed Nominatim, did a full planet-240624 import, but on replication --catch-up it hangs while reading the data. It looks like #3445, but in some ways different.

Log file is:

izverev@Ubuntu-2204-jammy-amd64-base:/srv/nominatim$ NOMINATIM_REPLICATION_MAX_DIFF=10000 sudo -u www-data nominatim replication --catch-up --threads 20
sudo: unable to resolve host Ubuntu-2204-jammy-amd64-base: Name or service not known
[sudo] password for izverev: 
2024-07-08 19:15:35: Using project directory: /srv/nominatim
2024-07-08 19:15:39  osm2pgsql version 1.11.0
2024-07-08 19:15:39  Database version: 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1)
2024-07-08 19:15:39  PostGIS version: 3.2
2024-07-08 19:15:39  Loading properties from table '"public"."osm2pgsql_properties"'.
2024-07-08 19:15:39  Using flat node file '/srv/nominatim/flatnodes.bin' (same as on import).
2024-07-08 19:15:39  Using prefix 'planet_osm' (same as on import).
2024-07-08 19:15:39  Using style file '/usr/local/etc/nominatim/import-extratags.lua' (same as on import).
Processing: Node(420k 420.0k/s) Way(0k 0.00k/s) Relation(0 0.0/s)

(and stuck on this line and count).

In psql, this is the stuck query:

INSERT INTO osm2pgsql_changed_relations
SELECT r.id
FROM "public"."planet_osm_rels" r, osm2pgsql_changed_nodes n
WHERE r.parts && ARRAY[n.id]
AND r.parts[1:way_off] && ARRAY[n.id]

Software Environment (please complete the following information):

• Nominatim version: 4.4.0
• Postgresql version: 14
• Postgis version: 3.2
• OS: Ubuntu 22.04

Hardware Configuration (please complete the following information):

• RAM: 64 GB
• number of CPUs: 32
• type and size of disks: NVMe SSD 2 TB x2

Nominatim Configuration:

NOMINATIM_FLATNODE_FILE="/srv/nominatim/flatnodes.bin"
NOMINATIM_USE_US_TIGER_DATA=yes

# base URL of the replication service
NOMINATIM_REPLICATION_URL="https://planet.openstreetmap.org/replication/hour"
# How often upstream publishes diffs (in seconds)
NOMINATIM_REPLICATION_UPDATE_INTERVAL=3600
# How long to sleep if no update found yet (in seconds)
NOMINATIM_REPLICATION_RECHECK_INTERVAL=300

[lonvia]

That's an osm2pgsql issue. Because you imported with Nominatim 4.3.2, you are stuck with the old middle format and it looks like it still has issues with large diffs. Try a catch-up with very small diffs, something like NOMINATIM_REPLICATION_MAX_DIFF=500.

[Zverik]

Thanks Sarah. Small diffs didn't work, so reimporting the database from scratch with 4.4.0. Turns out updates for another server (that's on 4.3.2 and imported in January) already stuck three weeks ago :(

[lonvia]

The server stuck three weeks ago is likely #3445. This new server isn't because the planet is post vandalism. So I suspect you have two unrelated problems here. You could try running an 'ANALYSE' on the new database and see if that helps.

(Also note that I'm intentionally not closing this ticket. Still investigating what is exactly going on.)

[Zverik]

Right. I'll see how it goes and report back to you :) Until the second server is in production, we could run any tests on it.

[Zverik]

So. I have imported the planet again (same one) with Nominatim 4.4.0, and everything went fine, including the catch-up update. Probably because the planet file was generated after the vandalism wave. Feel free to close this ticket when you don't need it :) And thank you for the hint about differing osm2psql versions.

[lonvia]

I don't really know how to debug this. We need a more reproducible case. Closing for now.