GitOps at scale patterns

[dtzar]

There is an abundance of guidance for GitOps fundamentals, however a lack of consolidated guidance with architectural design, considerations, performance/human review limitations for running GitOps patterns at scale.

I'd like to collaborate with other around creating general guidance here:

1. Important high-level questions for people to consider (e.g. organizational structure/autonomy desired, target scale)
2. Technical tips for implementation limits/roadblocks - (e.g. using high-level definitions (HLDs like kustomize & helm) versus manually extracting all HLDs into raw manifests)
3. Optimal human-manageability tips (e.g. reviewing deployments, Git PRs, etc.)
4. Code repository template structure considerations/examples

Tangent related discussion topic here: #95

Some relevant resources I (or others have) found:

1. CodeFresh.io - They have a "GitOps at Scale" course which points to use ArgoCD and their paid solution (course is also paid), Blog https://codefresh.io/blog/scaling-argo-cd-securely-in-2022
2. https://github.com/fluxcd/flux2-multi-tenancy
3. https://github.com/microsoft/multicluster-gitops
4. GitOps Con session at Kubecon NA 2022 - Hundreds of clusters sitting in a tree with ArgoCD Adobe
5. GitOps Con session - How Adobe planned for scale With Argo CD, Cluster API, and VCluster Another angle on Adobe story
6. GitOps Europe May 2022 - Managing Thousands of clusters and their workloads with flux
7. Intuit GitOps at scale - https://www.youtube.com/watch?v=ESQLqjbM8h0
8. https://codefresh.io/blog/how-to-model-your-gitops-environments-and-promote-releases-between-them/

[scottrigby]

Once folks start collaborating on this, please let us know. At that point, we can create a GitHub team and invite those who wish to and are able to be part of it ❤️

[moshloop]

I think we should consider phrasing this as patterns or cookbooks - guidance is normally something that requires context. With a pattern, the pros and cons of the approach can be highlighted so that people can pick the patterns that fit their context the best.

[scottrigby]

+1 for patterns 💖

[dtzar]

Sure, I already renamed the title of the discussion from guidance to patterns awhile back.

[roberthstrand]

Excellent suggestion!

Is this something we could have a kick-off meeting for, so to get some structured approach and scope the work a bit? I would love to help, but it is hard for me to keep track of everything happening unless there is some sort of method to the madness 😅