Nominatim replication init computed date from last object on DB is wrong

[jhrvalente]

When starting the updates I've noticed that the date computed from replication init to retrieve the sequence from which to start the updates is wrong.

I've imported the full planet on 14th of January and executed the replication init command for the first time on 15th.

However, whenever I execute the command replication init it uses the same date (27th December 2022 which is wrong):

2023-01-20 10:52:52: Using project directory: /home/nominatim
2023-01-20 10:52:52: Initialising replication updates
2023-01-20 10:52:53: Updates initialised at sequence 3758 (2022-12-27 20:58:03+00:00)

On the website, on which I have the Nominatim api and the Nominatim-ui it also says 'Data last updated: 3 weeks ago'

What could be the cause for this?

[mtmail]

Do you still have the planet.osm.pbf file? One way to find out which file was imported is comparing the md5 sum of the file against https://planet.openstreetmap.org/pbf/

e45d4461341f865e6af735559d63d2a9  planet-221219.osm.pbf
2376329b4cc352f7d0cfc6c7c9a1d6fd  planet-221226.osm.pbf
52e0e2129a6b3c3a52479e92fef60b34  planet-230102.osm.pbf
1b692feca072ed34ef1298bc7afa51a9  planet-230109.osm.pbf

The planet dump file creation takes several days. The file dated 2022-12-26 was only available on planet.openstreetmap.org server December 31st. Then depending on which mirror* you used there might be another 1-2 day delay. That means a file you downloaded might have the latest data from up to 7 days ago.

*) mirror servers are faster, use mirrors whenever possible

[jhrvalente]

The md5 sum of the file is the following: 2376329B4CC352F7D0CFC6C7C9A1D6FD
So it corresponds to the file planet-221226.osm.pbf you posted above.

I have executed the command replication init two times:
The first I think it might have used the date 26th of December 2022 for the imports.
Today it used the date 27th of December 2022.
Each time it will add one day to the previous date.
But on the first update, when I ran the update just after the first replication init, supposing I did it on the 16th of January, shouldn't the date computed with replication init be 16th of January, as the last object added to the DB was updated with this date to the DB?

[mtmail]

The logic changed between Nominatim versions. I think the default is still that for daily replication only one day is run. So you'd have to run it 15 times, or rather until nominatim replication --check-for-updates return that there's no more updates (Unix exit code 1).

There's a new https://nominatim.org/release-docs/latest/admin/Update/#catch-up-mode I'd use that now until the database is up to date and then use the other commands for regular (cronjob, continuous or systemd) setup.

Daily planet updates can take hours. If the server is not yet used by other users try a high --threads value.

If a database is 2-3 months behind then we usally advise to do a full reimport instead: catching up all the days would take longer than a reimport.

[jhrvalente]

Thanks @mtmail for your response it was really helpful!
Can you also tell how much disk space it is expected for it to add to the disk daily with the updates?

[mtmail]

Expect database size to grow by 10% per year. The (temporary) daily downloads are about about 100-150MB for per day https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/replication/day/000/003/ but not all OSM changes are relevant for Nominatim.