Too many postgres processes in bluk queries

[Adora2333]

Describe the bug
I have installed Nominatim in my local machine and imported the planet dataset. I use the GeoPy in python. I use a for loop to query the nominatim for full geographic locations. As the program runs, there are more and more processes on the server and
the program runs more and more slowly. Finally, my program crushed. Are there some methods such as parameters to shorten the runtime of postgres processes that are completed?

To Reproduce

    str_useragent="bhukg"+str(random.uniform(1, 100000))
    geolocator = Nominatim(user_agent=str_useragent, domain='127.0.0.1/nominatim', scheme='http')
    geocode = partial(geolocator.geocode, language="en")
    for i in tqdm(range(len(data))):
        temp=str(data['location'][i])
        aa=geopy_address(geocode,temp)
        if aa:
            print(str(i),'success')
            write_res([data['id'][i], aa],file)

Software Environment (please complete the following information):

• Nominatim version: 3.7.2
• Postgresql version: psql (PostgreSQL) 10.18 (Ubuntu 10.18-0ubuntu0.18.04.1)
• Postgis version:
• OS: Ubuntu 18.04

Hardware Configuration (please complete the following information):

• RAM: 128G
• number of CPUs: 96
• type and size of disks: SSD 1.8T
• bare metal/AWS/other cloud service:

Postgresql Configuration:
Our configurations are same to the documents.

shared_buffers = 2GB
maintenance_work_mem = 10GB
autovacuum_work_mem = 2GB
work_mem = 50MB
effective_cache_size = 24GB
synchronous_commit = off
max_wal_size = 1GB
checkpoint_timeout = 10min
checkpoint_completion_target = 0.9

[lonvia]

(Moved to discussions as this is rather a question about deployment than a Nominatim issue.)

How have you set up the webserver side of things, i.e. the part described in Deployment? Apache, nginx, php-fpm?

[Adora2333]

Sorry for the delay. I use Apache/2.4.29 (Ubuntu) to deployment.

[mtmail]

By default Apache will start one PHP process per request (mpm_prefork), that's not effective for high number of requests per second. Ideally you want longer running PHP processes handling 1000s of requests. You'd need to experiment with StartServers and other Apache setting (e.g. http://httpd.apache.org/docs/current/mod/worker.html https://www.microhost.com/docs/tutorial/tuning-of-your-apache-server/) and there's no one best setting for number of CPUs.

More straight forward, or rather easier to copy&paste) is to switch to nginx webserver + php_fpm as documented in https://www.nominatim.org/release-docs/3.7.2/appendix/Install-on-Ubuntu-18/#option-2-using-nginx There's still some settings one can tune, e.g. max_children but overall it will handle more requests per second with the base settings already. Don't forget to stop and disable the Apache2 webserver.

[Adora2333]

Thank you! I will try it.