Upgrade internal cluster(s)

[sfreudenthaler]

Let's get non-customer internal clusters upgraded if we can to validate findings from #30786

outcomes

• internal cluster updated
• quick and dirty runbook or documentation review
• Confirm or Deny if we're upgrading prod clusters on Jan 8 maintenance window

[cobbg]

The internal EKS cluster in the R&D account has been upgraded to EKS v1.31. Nodes in this cluster have also been updated to the latest Bottlerocket AMIs for version 1.31.

This upgrade was completed by pairing with @yolabingo and he is going to upgrade the internal-2 cluster on his own. I have also made some notes to use for upgrade documentation.

[yolabingo]

internal-2 cluster on Cloud account also updated to k8s 1.31, there were no apparent issues

[cobbg]

Internal documentation for EKS upgrades can be found in Confluence here.

[cobbg]

I do not have any significant reservations for upgrading EKS clusters during the next maintenance window on Wednesday, January 8, 2025. My only concern was that we are given a reasonable amount of time to test the updated Bottlerocket AMI running on the nodes through normal operations because these AMIs have been problematic in the past and we have to update them following an EKS upgrade.

Our normal procedure is to update the internal clusters' nodes to a recent AMI version the week prior to a maintenance window. Since this is a fairly new process, Todd has typically been updating AMIs the Friday prior to a maintenance window. This means that we're within the normal time frame of AMI updates and testing right now.

Therefore, I don't see any reason to not plan to upgrade the production EKS clusters during the upcoming maintenance window -- provided no one else has any concerns.

[cobbg]

After speaking with @yolabingo, he has come to the same conclusion regarding the EKS upgrades.

We will plan to upgrade production EKS clusters during the next maintenance window. If there's a problem between now and then, we can always postpone the upgrades. I'll also inform Jess so she can create all of the necessary tickets.

I'm going to close this issue and move it to done.

[sfreudenthaler]

This is great guys.  Appreciate the notes on the issue so I could follow along On Jan 3, 2025, at 14:57, Gregory Cobb ***@***.***> wrote: After speaking with @yolabingo, he has come to the same conclusion regarding the EKS upgrades. We will plan to upgrade production EKS clusters during the next maintenance window. If there's a problem between now and then, we can always postpone the upgrades. I'll also inform Jess so she can create all of the necessary tickets. I'm going to close this issue and move it to done. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>