"Management" Clusters

[chris-short]

I'd like to start a discussion around this topic to get a better idea of what people are seeing/doing around the management clusters.

As folks look to minimize attack surfaces and maximize infrastructure usage, the concept of a GitOps management cluster comes to mind. This management node would be in charge of GitOps pipelines across a larger fleet of Kubernetes clusters handling all GitOps operations across the fleet. At least, this is how I understand the desired end state.

What information, if any, is already out there?
What suggestions would we give to folks considering this model?
What would the GitOps docs for Management Clusters look like?

[todaywasawesome]

Did a blog post recently about hub-and-spoke vs standalone gitops operators, specifically with Argo CD. I think it would apply globally. https://codefresh.io/argo-platform/scaling-argo-cd-securely-in-2022/

[chris-short]

This too: https://www.youtube.com/watch?v=OIWUyKuuytQ

[tsahiduek]

Hi,
Personally I see the stand-alone model as an ideal/preferred over the single/management-cluster model.
One way I can think of it, is similar to configuration management system that have a main service that is the source of truth of all instances that will be connected to it. With this analogy, the main service that holds the configuration can be a repository, an the instance can be a cluster that uses any kind of gitops agent (such as argo or flux), that is being sync with the main repository. This way, clusters are not aware of each other (and doesn’t need to), nor the configurations itself. This approach can also help in smooth transition into B/G cluster upgrades (which is something common I see today because of the challenge/in-ability to downgrade Kubernetes version), or having multiple immutable clusters on the fly (instead of dealing with very-large clusters with all the implication that comes out of this)
I also can share my experience from a company I’ve worked in:
We had multiple DCs across the globe with single Configuration management system.
Every new server was deployed with the CM agent installed and being pointed out to the DC main CM service. the CM service were synced with Git repository that holds the configurations for all the DCs. The CM didn’t managed the server, but it was the other way around - the server registered itself as part of it’s bootstrapping process
Also, each time we needed to build a new DC, we simply point it’s CM agent to the primary CM service that is being replicated from other region. All the configuration was managed in git, and was downloaded to the main CM server (automated of course). This way we achieved a process where all the CM config was stored in git, and every DC were pointing to the relevant directory in the repository as part of the DC creation process (which included CM main service deployment).
To me, the above CM process can be used in a similar way with gitops - Clusters bootstrap themselves as part of cluster creation process, and being synced with a git repository that holds the cluster configuration (whether it’s app of apps concept of Argo, or kustomization with Flux)
Hope this make sense

[williamcaban]

I consider is not one model vs the other, but what model has the more advantages for the use case. Some examples:

• Using GitOps to deploy the actual clusters, a "management" cluster might be the ideal scenario
• Using GitOps for cluster-wide configurations and cluster day-2 operations, it might be good to have a dedicated privileged instance in the cluster restricted to cluster admins GitOps activities
• Using GitOps for application deployment an operation, it might be good to have a tenant based unprivileged instance to isolate and decouple dependencies from a cluster level GitOps software
• Using GitOps to distribute where a tenant instance is assigned (and their workload), that might more suitable for a "management" cluster

Now, a "management cluster" could also be seen as a concept or role handled by a subset of clusters and that "management role" does the "management" tasks described in the previous list.

_W