Nominatim updates are stuck after "Processed relations" #3445
Comments
|
Script: |
|
Can you post relevant lines from the postgresql server log? -> we're looking for errors, warnings, out-of-memory or such. What does the query What is the output of |
|
pSQL, nothing in special, only when I stopped the script and restarted it.
|
|
|
|
This is an issue with the recent vandalism on OpenStreetMap. To work around this:
Further note that the vandalism issues are only present in the minutely and hourly diffs of planet replication. The daily diffs from the planet and Geofabrik should not be affected. Your setup is configured wrongly and consumes planet-wide diffs. In particular, in your bash script you need to set |
|
Hi @lonvia, thanks, I'll retry today and get back to you. BTW, I found the "old server" script log (that was running +1 year fine). That was a PG 13.
|
|
Hi, hello. Same behaviour here with the Spain and Germany maps since yesterday update at 4 AM. The Portugal update also had some problems but the machines returned to a good state after a few hours. |
|
Same remedy if the geofabrik daily diffs cause issues. Stop updates, kill copy (or simply restart postgres), run catch-up with a very large replication size. |
|
@lonvia question: I need to run
My need is update once per day. |
|
The "catch-up" ensures that you jump over the problematic edits in the updates. It only needs to be done once and then updates can continue as usual. (Until the next time it gets stuck, that is. The vandalism is sadly still not completely contained.) As for your use of planet updates: I suggest that you reimport the Brasil extract and start again from there. The year of applying planet updates will have added a lot of garbage to your database. |
|
@lonvia thanks, I did't know about the "export" in the daily script... so, basically I can drop the database, redo the:
and then just run daily:
It's ok then? I only will use Brazil data. |
|
@ArvyRogerio That looks good. |
lonvia
changed the title
|
Done. Re-imported and ran the update script. For now, looks fine. BTW, no more errors.
|
|
I have imported the planet on 2 July (240624), and the import still hangs on reading the osmchange. I've used all the suggestions from this thread. Nominatim 4.3.2 (rn updating to 4.4.0 to see how it goes). Could you maybe release 4.4.1 with the fix?.. |
|
Updated to 4.4.0, still get the same: (and stuck on this line and count). (I made #3470). |
|
So with 4.3.2 neither big nor small max_diff helped. First try I stopped after 22 hours: Second try started 3 hours ago, don't think it'll end: Am I missing anything, or it'd be better to reimport from scratch with 4.4.0? |
|
Have you made absolutely sure that no COPY or TRUNCATE operation is in progress before restarting the catch-up? The vandalism leaves a COPY operation ongoing which blocks everything, restarting leaves a TRUNCATE which blocks everything. They both need to be killed separately. |
|
Yes, there was this query ongoing: |
|
Two possible explanations then: (a) part of the vandalism already got committed, so this is actually doing the reverts or (b) doing the large diff lands you again in the middle of the next vandalism wave. Either way, a reimport might be in order. If you don't mind experimenting a bit, could you apply #3447 and see if that makes updates go through? The patch should be appliable to your 4.3.2 version. If it helps, I'll do a 4.4.1 release with this fix. Note that you still have to reimport if your current state is (a). The database will be likely messed up badly if the vandalised ways got through. Edit: after applying the patch, you need to run |
|
I've also had this problem (on v4.3.2.) and attempting to catch up as suggested with max diff 10000. I'm concerned my database may be "messed up badly if the vandalised ways got through". How can I find out if my database has been corrupted? |
|
If the catch-up goes through, you are fine. Otherwise you can try running the query Warning: the following is only for experienced users. You could try to get rid of them with |
|
Thanks for the suggestions. It looks like the update is stuck again but I'll wait a little longer. Details below If it is stuck I guess I'll need to reload the database? The original problem from Juneapplying the patch to placex_triggers.sqlI eventually found the file in the location below (ubuntu) and confirmed that 'nominatim refresh --functions' applied the change to the database. QueriesLooks OK, Result less than 20 and no records to update. Attempt to CatchupLooks to be stuck again. osm2pgsql queriesAll started at 2024-08-01 11:51:28 and 2 active copy from STDIN. |
Thanks for the tip! I'm currently trying this, but my server (running a Europe extract) is now on day three of running the catch-up command. @lonvia do you have any experience how long this should take? Even though I'd really like to avoid downtime, maybe a fresh import makes more sense at some point 🤔 CPU usage jumps approx. every hour from ~98% to ~105%, so it still seems to be doing something. Output is stuck at |
|
@martbock Given that you now will be 2 months behind on updates, a fresh start sounds more reasonable. It's also a good opportunity to switch to version 4.4 (and eventually the Python frontend). |
|
@RailsGuy I'm out of ideas here. I wouldn't have thought that relations are effected by the vandalism but obviously they are. It might just be that the vandalism has completely messed up Postgres' query planning. I've seen that before. At this point, given that you are now missing 2 months of updates, a reimport (please, do upgrade to 4.4) is advisable. |
@lonvia all right, that validates my gut feeling. I'll do a fresh import, then. Thanks a lot for your quick input, I appreciate it! |
Describe the bug
No docker (self host).
Since 2 weeks ago, "nominatim replication --once" is not completing.
Tried to full reinstall, same results.
First server was up +1 year, update every 2:00 AM.
Postgres appears to be in loop (COPY). More than 24 hours running.
Only one country (Brazil).
2024-06-16 16:57:45: Using project directory: /dados/nominatim/nominatim-planet
2024-06-16 16:58:20 osm2pgsql version 1.11.0
2024-06-16 16:58:20 Database version: 15.6 (Debian 15.6-0+deb12u1)
2024-06-16 16:58:20 PostGIS version: 3.3
2024-06-16 16:58:20 Loading properties from table '"public"."osm2pgsql_properties"'.
2024-06-16 16:58:20 Not using flat node file (same as on import).
2024-06-16 16:58:20 Using prefix 'planet_osm' (same as on import).
2024-06-16 16:58:20 Using style file '/usr/local/etc/nominatim/import-extratags.lua' (same as on import).
2024-06-16 17:02:45 Reading input files done in 265s (4m 25s).
2024-06-16 17:02:45 Processed 201681 nodes in 56s - 4k/s
2024-06-16 17:02:45 Processed 40896 ways in 188s (3m 8s) - 218/s
2024-06-16 17:02:45 Processed 4733 relations in 21s - 225/s
(Control+C)
KeyboardInterrupt
Postgres 15:
3351778 postgres 87.6 6.7 218:32.43 postgres: 15/main: nominatim nominatim 127.0.0.1(54044) COPY
(for more than 24 Hours - needed to restart postgres)
To Reproduce
nominatim replication --once
Software Environment (please complete the following information):
Hardware Configuration (please complete the following information):
The text was updated successfully, but these errors were encountered: