Upgrade our stack to latest Python

[jimleroyer]

Description

As a software developer working on GCNotify stack,
I want the Python stack to be upgraded at latest supported version,
So that I can leverage the latest features of the language and better/safer code.

As a system ops working on GCNotify stack,
I want the Python stack to be upgraded at latest supported version,
So that I can leverage the latest performance improvement
And process notifications faster.

WHY are we building?

To get latest performance improvements, features and security updates of the Python language/runtime.

WHAT are we building?

Upgrade the stack to the latest possible version of Python. A few considerations:

• It is possible that we cannot upgrade to the latest of the latest (3.12 at the time of this writing for the stable version), as this might be blocked by a dependency library that did not release a version supporting the latest. For example, we might have to upgrade from version 3.10 to version 3.11, and wait a bit longer to migrate to version 3.12.
• The lambdas might have additional considerations that might limit the possible maximum Python version we can use, such as supported runtime and also monitoring libraries that need to support that environment (such as New Relic). Consider doing this work in Upgrade lambda and New Relic to latest supported version #354 .

VALUE created by our solution

Better performance, security and reliability.

Acceptance Criteria

• utils 2
• document-download-api 2
• admin 2
• api 3
• The Kubernetes components running on Python are upgraded to latest agreed and possible Python version.
• The lambdas components are done by task Upgrade lambda and New Relic to latest supported version #354 and not this task.
• Run rollercoaster tests
• Run load tests and compare performance to 3.10 (about 5500 emails / min max sustained)
• have a bugbash on it
• deploy and test on dev
• document in tricks and tips how we can deploy branches to dev

QA Steps

• Smoke tests are ran against newly upgraded lambdas.
• Performance tests are ran against newly upgraded lambdas.

[sastels]

Looking at moving document-download-api from 3.10 to 3.12
https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-document-download-api/195

need utils before document-download-api of course :/
https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-utils/326

[sastels]

also admin WIP cds-snc/notification-admin#1996

[sastels]

also started on api

[sastels]

fixing many broken tests 😭

[sastels]

Tests passing, ran admin / api / dd-api locally and all working! Going to give the changes another read through and then put for review.

[sastels]

Will convert this card to an epic and create separate cards for the 4 repos.
utils 2
document-download-api 2
admin 2
api 3

[sastels]

Documented how to build and run admin and dd-api on dev. dd-api crashed on start :/ This is a problem for Thursday Steve.

Thursday Steve should also break up this card.

[jimleroyer]

It seems the DD-API pod restart once in a while even without the Python upgrade. Steve will double-validate and create a card to follow up on that if that's the case.