Upgrade EC2 instance type when high load is expected

[DeD1rk]

What?

We can change the EC2 instance type of our server with just a simple reboot. We can use this to upgrade from a t3a.small to e.g. a t3a.2xlarge, so that we can handle short periods of high load (such as when event registrations open) better, without permanently paying for big servers.

Why?

We need to not crash on e.g. weekend registration opening. The most extreme event registration openings are easy to predict.

How?

When event registration is about to open, temporarily change the instance type in terraform. With terraform apply we can be back online in about 1-2 minutes if we skip the currently slow collectstatic step (see #3290). Doing this manually is probably fine.

I tried this out on staging. As a t3a.small, it handles +- 16 request per second to the homepage (unauthenticated). Upgrading to a 2xlarge (8 vcpus instead of 2), it handles 64. So scaling is pretty much perfectly linear.

We should think about how we can be sure to apply this in time. For example, it would be nice to upgrade early in the morning and downgrade again in the evening, so we should be aware of upcoming opening registrations a few days in advance to plan it.

[DeD1rk]

Update: this (together with the other optimizations we've done) worked flawlessly for the weekend registrations.

[ColonelPhantom]

Speaking of optimization, is the scaling with the number of vCPUs, or the number of workers? Seeing as we are probably spending most time blocked on the database rather than processing things in Python on CPU.

Of course, increasing the worker count is probably its own can of worms since each one consumes quite a bit of memory. So I'm not sure we can add more workers on a smaller VM.

[ColonelPhantom]

I just checked the statistics of the server with htop, and we're of course pretty idle at this time. Memory-wise, there seems to be a bit of room left (2.5G / 3.8G). There's 5 gunicorn workers at 226M to 288M. Additionally, there's 4 celery workers at 115M to 217M. So I think there's room to add a few more gunicorn processes. Additionally, do we need 4 Celery workers? Since I'd expect most Celery tasks to not be that latency-sensitive.