Upgrade path from Nebari 2023.1.1 to latest 2023.11.1

[kalpanachinnappan]

Context

Our prod Nebari cluster is running on 2023.1.1, and we would like to upgrade it to the latest stable version, 2023.11.1.
What would be the safest upgrade path to take without data loss?

Thank you!

Value and/or benefit

Upgrade minor versions without affecting the cluster data.

Anything else?

No response

[rsignell]

We just recently went through this with the ESIP Nebari deployment on AWS, going from 2023.1.1 to 2023.10.1.

• We backed up all the user data using the Manual Backup Process.
• Saved each conda-store environment to conda environment yaml file so we can create it later.
• We upgraded our nebari conda environment to the latest version.
• We ran nebari upgrade -c nebari-config.yaml multiple times, answering all the questions each time.
• We did basically a new deployment with nebari deploy (got a new IP and needed to change the CNAME assignment)
• We restored all the user data.
• We generated a CSV file of usernames from the user data folder and used that to regenerate user accounts for everyone using keycloak import (we use github auth so we just needed the usernames)
• Modified the environment.yaml files to upgrade nebari-dask to latest version
• Rebuilt the conda-store environments using the updated conda environment files (manual process, one by one)
  Not sure if there was a way to do this without needing to redo the user accounts. Also exporting the user accounts would have been better as we lost people's email addresses, but we couldn't figure out how to do that.

@aktech ?

[dharhas]

Longer term, the approach I want to implement involves having a proper backup and restore. This is on the roadmap to build in January

• Backup everything in Nebari that has state
  • user folder data
  • keycloak data
  • conda-store data
  • anything else?
• deploy a clean install of the latest version of Nebari
• restore state from backups
• destroy previous nebari install

I think this will be a more robust solution than incremental update scripts, especially when jumping a large number of versions.

[rsignell]

@dharhas oh my, yes that would be an amazing improvement!

[marcelovilla]

Most likely you have already migrated upgraded your Nebari version. For posterity, I think @rsignell outlined a good approach for upgrading, taking into account the backup of existing data. This is specially necessary for AWS as version 2023.7.1 triggered Terraform to replace the VPC due to some changes on the AWS side of things.

As @dharhas mentioned, we're working on a backup and restore mechanism that will facilitate this process for users in the future. More information can be found on this RFD.

I'm closing this issue but feel free to reopen if necessary.