Run away process on guardian start

[RossWright-ZP]

Running a Guardian instance on EC2, 8GB ram, 8GB swap file. Suddenly started consuming 100% swap, memory and CPU. When we reboot the image or the entire server, once we restart the instance it immediately runs back up within 2-3 mins. All we see a increasing number of node processes that are eating up all resources until the server crashes within 5-7 minutes. Managed to get this from viewing the log before it crashed. We see dozens of instances of one issue before it crashes. We also see a lot of 500 error popups in the UI before it crashes. This started after we initialed a dozen policy imports with about 10 documents per policy via the API in serial.
. We are using 2.27.1.

[mattsmithies]

We have suffered many issues like this before, normally it's when the Guardian is overloaded by expected actions - as you described. If you try to push too much work to guardian without having a queue on top to manage the flow of data until the Guardian is ready to receive new state/actions.