853: Service availability in AWS

[chriswilty]

Description

It came to my attention that the fargate service and associated infrastructure is costing us more than $3 a day. We can decrease that by around 70% if we bring the service down overnight, and not have the application up over weekends.

Currently there is no trivial way to achieve this using CDK, although there are some alpha constructs in development which we can use, together with a lambda function, to update the service to flip "desired task count" from zero to 1 in the morning and back to zero in the evening.

Resolves #853

Screenshots

Concerns

• We might want to show a different UI page outside of uptime hours, such as a generic "downtime / maintenance" message. I will assess how this looks once it's live in AWS, and spawn another issue if needed.
• I'm happy using the alpha constructs, as they seem robust; it's just that the API is subject to change so we may need to make minor adjustments whenever bumping to a new version.

Checklist

Have you done the following?

• Linked the relevant Issue
• Added tests
• Ensured the workflow steps are passing

[pmarsh-scottlogic]

what makes this a devdependency and not a standard dependency?

[chriswilty]

This is the CLI for running commands, such as "cdk synth", so it's not part of any built artifacts or build process. It's like using jest to run tests, or typescript to compile the code.

In contrast, aws-cdk-lib is used in our scripts that are run to produce the CloudFormation templates, which are then deployed to AWS.

[pmarsh-scottlogic]

obviously I don't understand half of it, but it reads well and seems sensible enough

[chriswilty]

@pmarsh-scottlogic Understood! It's more complex than I would've liked, but the idea is that we schedule a pair of events, one in morning and one in evening, and when they run, the scheduler turns them into an "action" to run a lambda function. That lambda function has one job to do: adjust the number of tasks/containers the service is running: to 1 in the morning, 0 in the evening.

But we need to adjust the service definition itself, because simply auto-scaling the running service to zero will cause Fargate to check its task definition and see that it needs at least one container running, so it immediately brings one up again. This is Fargate's job - it manages container deployment for us, optimising according to current traffic and our defined limits. But our "desired count" is the baseline to which our number of containers will return under "normal" load, and that desired count cannot be zero, as that would make no sense. Hence why this is a touch more complex than I would've liked, and hence the need for a lambda function to adjust the service definition.