Data from another country in an existing database

[smsoousa]

How do I add data from another country in nominatim 3.5.2. I currently have the data for Brazil and now I want to add the data for Portugal.

[mtmail]

Have a look at https://nominatim.org/release-docs/3.5/admin/Advanced-Installations/

[smsoousa]

Yes, I already did that process, see my import_multiple_regions.sh file:

#!/bin/bash -xv

# Script to set up Nominatim database for multiple countries

# Steps to follow:

#     *) Get the pbf files from server

#     *) Set up sequence.state for updates

#     *) Merge the pbf files into a single file.

#     *) Setup nominatim db using 'setup.php --osm-file'

# Hint:
#
# Use "bashdb ./update_database.sh" and bashdb's "next" command for step-by-step
# execution.

# ******************************************************************************

touch2() { mkdir -p "$(dirname "$1")" && touch "$1" ; }

# ******************************************************************************
# Configuration section: Variables in this section should be set according to your requirements

# REPLACE WITH LIST OF YOUR "COUNTRIES":

COUNTRIES="europe/portugal"

# SET TO YOUR NOMINATIM build FOLDER PATH:

NOMINATIMBUILD="/srv/nominatim/Nominatim/build"
SETUPFILE="$NOMINATIMBUILD/utils/setup.php"
UPDATEFILE="$NOMINATIMBUILD/utils/update.php"

# SET TO YOUR update FOLDER PATH:

UPDATEDIR="/srv/nominatim/Nominatim/update"

# SET TO YOUR replication server URL:

BASEURL="https://download.geofabrik.de"
DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"

# End of configuration section
# ******************************************************************************

COMBINEFILES="osmium merge"

mkdir -p ${UPDATEDIR}
cd ${UPDATEDIR}
rm -rf tmp
mkdir -p tmp
cd tmp

for COUNTRY in $COUNTRIES;
do
    
    echo "===================================================================="
    echo "$COUNTRY"
    echo "===================================================================="
    DIR="$UPDATEDIR/$COUNTRY"
    FILE="$DIR/configuration.txt"
    DOWNURL="$BASEURL/$COUNTRY$DOWNCOUNTRYPOSTFIX"
    IMPORTFILE=$COUNTRY$DOWNCOUNTRYPOSTFIX
    IMPORTFILEPATH=${UPDATEDIR}/tmp/${IMPORTFILE}
    FILENAME=${COUNTRY//[\/]/_}
    

    touch2 $IMPORTFILEPATH
    wget ${DOWNURL} -O $IMPORTFILEPATH

    touch2 ${DIR}/sequence.state
    pyosmium-get-changes -O $IMPORTFILEPATH -f ${DIR}/sequence.state -v

    COMBINEFILES="${COMBINEFILES} ${IMPORTFILEPATH}"
    echo $IMPORTFILE
    echo "===================================================================="
done


echo "${COMBINEFILES} -o combined.osm.pbf"
${COMBINEFILES} -o combined.osm.pbf

echo "===================================================================="
echo "Setting up nominatim db"
#${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1

${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1
echo "===================================================================="

[smsoousa]

Unfortunately, the process didn't work out for me, I spent almost a day and a half processing the import, and even then, the data from Portugal didn't import.

HTTP request sent, awaiting response... 200 OK
Length: 306606981 (292M) [application/octet-stream]
Saving to: ‘/srv/nominatim/Nominatim/update/tmp/europe/portugal-latest.osm.pbf’

/srv/nominatim/Nominatim/update/tmp/europe/portugal 100%[=================================================================================================================>] 292.40M  45.8MB/s    in 6.6s    

2023-07-25 09:11:30 (44.4 MB/s) - ‘/srv/nominatim/Nominatim/update/tmp/europe/portugal-latest.osm.pbf’ saved [306606981/306606981]

2023-07-25 09:11:31 INFO: Using replication server at http://download.geofabrik.de/europe/portugal-updates
europe/portugal-latest.osm.pbf
====================================================================
osmium merge /srv/nominatim/Nominatim/update/tmp/south-america/brazil-latest.osm.pbf /srv/nominatim/Nominatim/update/tmp/europe/portugal-latest.osm.pbf -o combined.osm.pbf
====================================================================
Setting up nominatim db
'/srv/nominatim/Nominatim/build/osm2pgsql/osm2pgsql' --hstore --latlong --append --slim '--number-processes' 1 --cache 2000 --output gazetteer --style '/srv/nominatim/Nominatim/settings/import-full.style' --database nominatim --port 5432 '/srv/nominatim/Nominatim/update/tmp/combined.osm.pbf'
osm2pgsql version 1.2.0

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Parsing gazetteer style file '/srv/nominatim/Nominatim/settings/import-full.style'.
Using projection SRS 4326 (Latlong)

Reading in file: /srv/nominatim/Nominatim/update/tmp/combined.osm.pbf
Using PBF parser.
Processing: Node(301667k 5.3k/s) Way(23571k 1.91k/s) Relation(414660 132.7/s)  parse time: 72930s                         
Node stats: total(301667988), max(11065756920) in 57436s
Way stats: total(23571506), max(1191919682) in 12370s
Relation stats: total(414660), max(16124828) in 3124s
Stopping table: planet_osm_nodes
Stopped table: planet_osm_nodes in 0s
Stopping table: planet_osm_ways
Stopped table: planet_osm_ways in 0s
Stopping table: planet_osm_rels
Stopped table: planet_osm_rels in 0s

Osm2pgsql took 73304s overall
node cache: stored: 151546513(50.24%), storage efficiency: 57.81% (dense blocks: 6180, sparse nodes: 105758721), hit rate: 52.71%

I do not know what else to do

[mtmail]

I think the indexing step hasn't run yet. Can you run utils/update.php --index and the database queries

• SELECT country_code, count(*) FROM placex WHERE country_code in ('br', 'pt') GROUP BY country_code;
• SELECT count(*) FROM placex WHERE indexed_status > 0';

again?

[smsoousa]

postgres@vmi313965:~$ /srv/nominatim/Nominatim/build/utils/update.php --index
WARNING: Starting indexing rank (0 to 30) using 1 threads
WARNING: Starting rank 0
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 0

WARNING: Starting rank 1
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 1

WARNING: Starting rank 2
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 2

WARNING: Starting rank 3
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 3

WARNING: Starting rank 4
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 4

WARNING: Starting rank 5
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 5

WARNING: Starting rank 6
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 6

WARNING: Starting rank 7
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 7

WARNING: Starting rank 8
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 8

WARNING: Starting rank 9
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 9

WARNING: Starting rank 10
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 10

WARNING: Starting rank 11
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 11

WARNING: Starting rank 12
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 12

WARNING: Starting rank 13
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 13

WARNING: Starting rank 14
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 14

WARNING: Starting rank 15
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 15

WARNING: Starting rank 16
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 16

WARNING: Starting rank 17
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 17

WARNING: Starting rank 18
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 18

WARNING: Starting rank 19
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 19

WARNING: Starting rank 20
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 20

WARNING: Starting rank 21
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 21

WARNING: Starting rank 22
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 22

WARNING: Starting rank 23
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 23

WARNING: Starting rank 24
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 24

WARNING: Starting rank 25
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 25

WARNING: Starting rank 26
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 26

WARNING: Starting rank 27
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 27

WARNING: Starting rank 28
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 28

WARNING: Starting rank 29
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 29

WARNING: Starting interpolation lines (location_property_osmline)
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED interpolation lines (location_property_osmline)

WARNING: Starting rank 30
WARNING: Done 0/0 in 0 @ 0.000 per second - FINISHED rank 30

nominatim=# SELECT country_code, count(*) FROM placex WHERE country_code in ('br', 'pt') GROUP BY country_code;
 country_code |  count  
--------------+---------
 br           | 6990038
(1 row)

nominatim=# SELECT count(*) FROM placex WHERE indexed_status > 0';
nominatim'# 

[mtmail]

nominatim=# SELECT count(*) FROM placex WHERE indexed_status > 0';
nominatim'#

That didn't execute the query. (You forgot the ' before the 0)

I consulted with another Nominatim maintainer. If the indexed_status is all 0 then we don't know what is going on. Nominatim 3.5 is now three years old and a lot, actually most, of Nominatim has been rewritten since then. It's possible this issue was fixed in the meantime. Our recommendation is to start fresh with a Nominatim 4.2.3 install. Ideally on a separate server or create a full backup of your existing server first.