Postgres server closed the connection unexpectedly during import, is that recoverable?

[alpoi-x]

Describe the bug
Initial planet import terminates at "Recompute word counts" stage with following stack trace (including previous terminal output).

2023-11-21 00:52:49: Done 2737883/2737883 in 457 @ 5979.073 per second - FINISHED postcodes (location_postcode)

2023-11-21 00:55:01: Post-process tables
2023-11-21 02:29:23: Create search index for default country names.
2023-11-21 02:30:04: Recompute word counts


Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 12, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 226, in nominatim
    return get_set_parser().run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 121, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/setup.py", line 149, in run
    tokenizer.update_statistics()
  File "/usr/local/lib/nominatim/lib-python/nominatim/tokenizer/icu_tokenizer.py", line 116, in update_statistics
    cur.execute("""UPDATE word
  File "/usr/local/lib/nominatim/lib-python/nominatim/db/connection.py", line 36, in execute
    super().execute(query, args)
  File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 146, in execute
    return super().execute(query, vars)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

To Reproduce

• Follow the installation and import process for ubuntu22 from here and here
• Wait 2 days
• Wake up to this 😢

Software Environment (please complete the following information):

• Nominatim version: 4.3.0-0 (6478409)
• Postgresql version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
• Postgis version: 3.2.0
• OS: Ubuntu22.04

Hardware Configuration (please complete the following information):
NB: using an aws ec2 r7g.4xlarge instance

• RAM: 128GB
• number of CPUs: 16
• type and size of disks: 3000GB GP3 with 16000 iops and 1000 throughput

Postgresql Configuration:
defaults + the recommended values on the tuning page

data_directory = '/var/lib/postgresql/14/main'
hba_file = '/etc/postgresql/14/main/pg_hba.conf'
ident_file = '/etc/postgresql/14/main/pg_ident.conf'
external_pid_file = '/var/run/postgresql/14-main.pid'
port = 5432
max_connections = 100
unix_socket_directories = '/var/run/postgresql'
ssl = on
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'
dynamic_shared_memory_type = posix
min_wal_size = 80MB
log_line_prefix = '%m [%p] %q%u@%d '
log_timezone = 'Etc/UTC'
cluster_name = '14/main'
stats_temp_directory = '/var/run/postgresql/14-main.pg_stat_tmp'
datestyle = 'iso, mdy'
timezone = 'Etc/UTC'
lc_messages = 'C.UTF-8'
lc_monetary = 'C.UTF-8'
lc_numeric = 'C.UTF-8'
lc_time = 'C.UTF-8'
default_text_search_config = 'pg_catalog.english'
include_dir = 'conf.d'
shared_buffers = 2GB
maintenance_work_mem = 10GB
autovacuum_work_mem = 2GB
work_mem = 50MB
effective_cache_size = 24GB
synchronous_commit = off
max_wal_size = 1GB
checkpoint_timeout = 10min
checkpoint_completion_target = 0.9

Nominatim Configuration:

NOMINATIM_FLATNODE_FILE=/home/ubuntu/nominatim-project/node_locations/flatnode.file
NOMINATIM_IMPORT_STYLE=extratags

Additional context
The rest of the import process went smoothly. I'm less worried about what went wrong here and more hopeful that the import is recoverable? Any help is appreciated 😁

[mtmail]

Check the postgresql server logfile for hints why the server might have stopped.

It's reconverable, the indexing step already finished. Go into the project directory, then nominatim import --continue db-postprocess. nominatim import -h shows all parameters. The --threads is only relevant for the indexing step. Expect it to take a couple of hours.