Using nominatim machine image replicas (AWS EC2)

[mickutz]

Hi all,

I've been working on a DR procedure based on AMIs of the EC2 instance nominatim is imported on and wondering whether someone has an opinion on the issue bellow.

The issue I'm having is that performance goes completely downhill when sending request to machines running on new instances based off these resulting AMIs, as in response times are going up to even over 60 seconds from < 1 sec on the original machine for non cached searches. The replicated machines are running on identical specs as well, so I'd expect to see pretty much like for like performance.

Indexes are fine:
$ nominatim admin --check-database
2021-07-07 16:13:46: Using project directory: /home/nominatim/nominatim-planet
2021-07-07 16:13:46: Checking database
Checking database connection ... OK
Checking for placex table ... OK
Checking for placex content ... OK
Checking that nominatim.so module is installed ... OK
Checking indexing status ... OK
Checking that database indexes are complete ... OK
Checking that all database indexes are valid ... OK
Checking TIGER external data table. ... OK

But, when running an explain on the query, this is what we get:

Limit  (cost=436592.02..436592.02 rows=1 width=29) (actual time=12104.144..12104.146 rows=1 loops=1)
   ->  Sort  (cost=436592.02..436635.35 rows=17334 width=29) (actual time=12094.285..12094.286 rows=1 loops=1)
         Sort Key: (st_distance('0101000020E610000096B20C71AC6B0240EA4CEB8035814840'::geometry, geometry))
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Index Scan using idx_placex_geometry on placex  (cost=0.80..436505.35 rows=17334 width=29) (actual time=677.896..12094.226 rows=50 loops=1)
               Index Cond: (geometry && st_expand('0101000020E610000096B20C71AC6B0240EA4CEB8035814840'::geometry, '0.006'::double precision))
               Filter: ((linked_place_id IS NULL) AND (rank_address >= 26) AND (rank_address <= 30) AND (class <> 'boundary'::text) AND (indexed_status = 0) AND ((name IS NOT NULL) OR (housenumber IS NOT NULL)OR ((rank_address >= 26) AND (rank_address <= 27))) AND (((rank_address >= 26) AND (rank_address <= 27)) OR (st_geometrytype(geometry) <> 'ST_LineString'::text)) AND st_dwithin('0101000020E610000096B20C71AC6B0240EA4CEB8035814840'::geometry, geometry, '0.006'::double precision) AND ((st_geometrytype(geometry) <> ALL ('{ST_Polygon,ST_MultiPolygon}'::text[])) OR st_dwithin('0101000020E610000096B20C71AC6B0240EA4CEB8035814840'::geometry, centroid, '0.006'::double precision)))
               Rows Removed by Filter: 96
 Planning Time: 0.420 ms
 JIT:
   Functions: 8
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 1.141 ms, Inlining 0.000 ms, Optimization 0.651 ms, Emission 8.984 ms, Total 10.775 ms
 Execution Time: 12105.357 ms

[mtmail]

Does warming up the database query cache (nominatim admin --warm) lead to any improvement? It runs 1000 queries against the database which usually gives the operating system and database time to put some indices into memory rather than reading from disk.

[mickutz]

This is what's causing the issue.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

The best solution is using fast snapshot restore, but it's not cheap at all at $600.

[mtmail]

Interesting, thanks for the feedback