CrossCluster State Mgmt Templates

[DinaBelova]

Problem Statement
The current HMC (in development) lacks a centralized way to manage and deploy state across the HMC clusters, making it difficult to maintain consistency and efficiently update configurations fleet-wide.

Epic Goal
Develop a ServiceTemplate system for HMC, that allows for centralized definition, configuration, and deployment of templates (beach head services) across multiple Kubernetes clusters in the fleet.

Key Features

• Centralized the ServiceTemplate definitions for fleet-wide deployment.
• Support for multi-cluster and multi-namespace deployments.
• Anticipates ServiceTemplate conflicts and provides method to resolve.
• Version control mechanisms of the templates
• Access control and security measures to the ServiceTemplates.
• This system will enable platform engineering teams to manage deployments more flexibly and respond to changes or issues across the entire Kubernetes estate efficiently, with greater efficiency of effort and time.

Major deliverables

• Deploy beach head services (BHS) through templates with a managed ServiceTemplate definition system.
• Deploy Service templates across all clusters, existing in cluster mgmt with access control.
• Deploy Service templates across a specific list of clusters in cluster mgmt which are matched through labels, with access control.
• Deploy namespace-specific ServiceTemplate deployments across clusters, which are matched through labels, with access control.

Who it benefits

• Customer Business
  • Ensures consistency across all HMC-managed clusters, leading to a more stable and predictable infrastructure.
  • Reduces the resources required to manage large-scale Kubernetes deployments (100+ clusters).
  • Improves operational efficiency through centralized management of service templates.
• Platform Engineering Teams
  • Simplifies the definition, management, and upgrades of ServiceTemplates (beach head services) across the entire cluster fleet.
  • Provides flexibility to create overrides for specific clusters, cluster namespaces or groups of clusters when needed.
  • Enables faster response to changes and issues across the Kubernetes estate.
• Mirantis
  • Delivers an enhanced product offering that addresses a critical need in large-scale Kubernetes management.
  • Reduces support overhead by enabling customers to manage their infrastructure more effectively.

Acceptance criteria
General: The ServiceTemplate Management system for HMC will demonstrate centralized management and deployment of templates across multiple Kubernetes clusters, supporting fleet-wide, label-based, and namespace-specific deployments. It should include version control, conflict resolution, and access control mechanisms.

• Centralized ServiceTemplate Definition:
  • Create a ServiceTemplate that can be deployed across multiple clusters.
  • Verify that the ServiceTemplate can be managed and updated from a central location.
• Multi-Cluster Deployment:
  • Deploy a ServiceTemplate across all clusters in HMC
  • Confirm that the deployment is consistent across all clusters.
• Specific set of Clusters Deployment:
  • Deploy a ServiceTemplate to a specific set of clusters based on defined labels.
  • Verify deployed to that specific set of clusters.
• Specific Namespace(s) Deployment: •
  • Deploy a ServiceTemplate to specific namespaces across multiple clusters.
  • Verify that the deployment only affects the specified namespaces.
• Conflict Resolution:
  • Create two versions of a ServiceTemplate with conflicting configurations.
  • Verify that the system correctly resolves the conflict based on some value.
• Versioning:
  • Implement a ServiceTemplate system that ensures each template version is immutable once created, with any modifications resulting in a new version.
• Access Control:
  • Verify that each role (platform lead, platform engineer, security team) can only perform actions on the ServiceTemplates according to their defined permissions and organizational boundaries.

Assumptions

• Platform Engineers and Leads can manage ServiceTemplate deployments by simply creating ServiceTemplate Deployment Custom Resources (CRs) in the management cluster. They do not need to interact with a Git repository as part of this process.
• Open-source tool integration: Given that we're leveraging an open-source tool (e.g., Sveltos), certain acceptance criteria, such as basic operations, can be fulfilled using kubernetes and the tool CLI.
• Support for GitOps: Some customers may prefer to use a GitOps-based configuration/state management system to manage 2A State Management Objects. Since our solution is based on Kubernetes Custom Resource Definitions (CRDs) and objects, we allow customers to utilize a GitOps system if they choose to do so. We do not impose restrictions on how these objects are managed.
• Customer flexibility: We recognize that some customers may opt out of the 2A state management solution entirely and bring their own systems (e.g., Argo, Flux, Helmfile, Kaptain). Although this reduces the operational simplicity that our system provides, we will ensure that opting out does not interfere with the cluster management functionality provided by 2A.

Out of scope

• API or UI development
• Performance optimization for large-scale deployments
• Validation of deployed services
• Development of actual beach head services
• Integration with observability systems