Importing Extracted Data taking too long

[StarGPS]

Hello everybody,
This is my first post in this honorable community.
I'm trying to have my own server for reverse-geocoding purposes on:
4 cores * 4 threads CPU, 16GB RAM, and 500GB HDD VM.
I was able to import the African extracted data relatively pretty quickly, but when I try to import the EU and African data together it is taking too long. It is now more than 48H and the import process is still unfinished. Following is what I get when I run the import command:

nominatim@nominatim:~/nominatim-project$ NOMINATIM_IMPORT_STYLE=street nominatim import --osm-file africa-latest.osm.pbf --osm-file europe-latest.osm.pbf --reverse-only --no-updates 2>&1 | tee setup.log
2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(152112k 0.51k/s)** Relation(0 0.0/s)

I can eventually wait, but I'm wondering if this is a normal speed for the "Way" part. As I tried before to import the EU+NA+AF and it ended with an error of the import process being aborted abnormally.

Thanks in advance for your help!

[mtmail]

For Europe you should use a flatnode file https://nominatim.org/release-docs/latest/admin/Import/#flatnode-files And of course faster hardware, more RAM, faster local harddrives always help.

[StarGPS]

Hi, Thanks for the quick reply! I'm indeed using a flatnode file to import data.

…

On Mon, Aug 21, 2023 at 1:29 PM mtmail ***@***.***> wrote: For Europe you should use a flatnode file https://nominatim.org/release-docs/latest/admin/Import/#flatnode-files And of course faster hardware, more RAM, faster local harddrives always help. — Reply to this email directly, view it on GitHub <#3160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKAY3Z5GURQD7HLEO6ORYA3XWPALRANCNFSM6AAAAAA3Y2Y3O4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[StarGPS]

So this is a normal speed for my hardware config?

[StarGPS]

2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(415901k 0.87k/s)** Relation(0 0.0/s)

+5 days now and it is still going!

[StarGPS]

Just an update to whom might be interested. Finally the Europe and Africa Extracted data has been imported successfully. Here is the timeline:

2023-08-18 07:38:44: Using project directory: /srv/nominatim/nominatim-project
2023-08-18 07:38:48: Creating database
2023-08-18 07:38:55: Setting up country tables
2023-08-18 07:38:57: Importing OSM data file
2023-08-18 07:38:57  osm2pgsql version 1.7.2
2023-08-18 07:38:57  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-08-18 07:38:57  PostGIS version: 3.2
2023-08-18 07:38:57  Parsing gazetteer style file '/usr/local/etc/nominatim/import-street.style'.
NOTICE:  table "place" does not exist, skipping
Processing: Node(4327447k 1412.8k/s) **Way(415901k 0.87k/s)** Relation(0 0.0/s)

Reading input files done in 587587s (163h 13m 7s).
2023-08-25 02:52:04    Processed 4327447553 nodes in 3063s (51m 3s) - 1413k/s
2023-08-25 02:52:04    Processed 522215618 ways in 532631s (147h 57m 11s) - 980/s
2023-08-25 02:52:04    Processed 7333836 relations in 51893s (14h 24m 53s) - 141/s

`2023-08-30 03:26:42: Setup website at /srv/nominatim/nominatim-project/website`

It took roughly 11 days and 20 hours to complete the process.

Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  489G   75G  393G  17% /

Although the import process in my case needed about 300GB disk space to process the import. In the end, most of the data has been dropped and only 50GB of storage is being used if you use the --reverse-only --no-updates parameters.

Patience and luck is needed here :)

[Papricka95]

@lonvia Hello! im to using nominatim for import osm data. Also i tried to import some files (asia-latest.osm.pbf and europe-latest.osm.pbf) and it was terribly long... I got killed process (status D from htop and low consuming resources of processors <0.2%).

My second try is import first file - asia-latest.osm.pbf. This import was success. But if i try to import europe-latest.osm.pbf as first file i got error and kill this process..

Also i try to "nominatim import --osm-file antarctica-latest.osm.pbf", after i try "nominatim add-data --file ukraine-latest.osm.pbf" i got error and stopping process add-data... And i try "nominatim add-data --file kazakhstan-latest.osm.pbf" and i have same error..

And as result i have so much lost time and i decided to import all planet and this import was success. Why did it happend ? I don't know..
If who know this problem or how it resolve, write to me, pls in this comment

[Papricka95]

I have configuration PC:
AMD Ryzen 7 5700G, 8 cores, 16 threads
32GB RAM,
SSD disks (2): 2TB, 128 GB for system

[mtmail]

@Papricka95 Open a new issue. Commenting on unrelated threads over a year old will get lost. No need to mention individual developers or mentioning your own email address. It's a public forum.

[Papricka95]

@mtmail okey, please, sorry! Do i need to delete my comments ?

[mtmail]

discussion moved to #3605

Do i need to delete my comments ?

That's not required. I just wanted to make sure the question is seen. As a comment to an answer from over a year ago it would get lost.