Drastic increase in snapshot initialization time

[the-smooth-operator]

Time to start a fresh node from snapshot has quickly degraded.
Pre spring snapshot initialization time was about 20 mins. Post spring, up until today we were at about 15 minutes for a 8GB+ snapshots.
Today, snapshot initialization (about 11G/snapshot) seems to last for several hours even in top hardware.

We see a lot of IO:

However disk size stops to increase always at about 90 to 100GB of disk size:

Logs show slow process and only messages like:
info 2024-10-01T21:41:49.884 nodeos controller.cpp:2435 operator() ] Snapshot initialization 72% complete

[the-smooth-operator]

We have noticed that increasing the total available memory helps. However, the process never utilizes all the memory available.
I.e: if we set a limit of 16Gi of memory we only the process using up to see 2Gi.

[bhazzard]

Given that the state database has recently multiplied in size, 16GB memory is not sufficient. 192GB minimum, 256GB recommended for nodes with the simplest configuration. If you have sys-admin skills you can get away with significantly lower memory by utilizing tmpfs with swap file.

Some other config options that can help run in the most efficient way:
read-mode=irreversible
validation-mode=light
eos-vm-oc-enable=true