Initial proposal of the new Helmops controller.

It adds a new custom resource `HelmApp` (resource name open to debate) that describes a helm chart to be deployed.

The resource contains all the fealds from the classic `fleet.yaml` file plus a few new from the `GitRepo`
resource.

`HelmApp` yaml example:

```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmApp
metadata:
  name: sample1
  namespace: fleet-local
spec:
  helm:
    releaseName: testhelm
    repo: https://charts.bitnami.com/bitnami
    chart: postgresql
    version: 16.2.1
  insecureSkipTLSVerify: true
```

The implementation tries to share as much as possible from a `Bundle` spec inside the new resource, because it helps
to "transform" the `HelmApp` into a deployment (no coversion is needed for most of the spec).

The new controller was also implemented splitting the functionality into 2 controllers (similar to what we did for the `GitRepo` controller). This allows us to reuse most of the status handling code, as display fields in the status of the new resource are as similar as possible to have consistent user experience and to integrate with the UI in the same way the `GitRepo` does.

When a new `HelmApp` resource is applied it is transformed into a single `Bundle`, adding some extra fields to let the `Bundle` reconciler know that this is not a regular `Bundle` coming from a `GitRepo`.

Similar as we did for OCI storage, the `Bundle` created from a `HelmApp` does not contain resources. The helm chart to be deployed is downloaded by the agent.

Code for downloading the helm chart is reused from gitops, so the same formats are supported.
Insecure TLS skipping was added the the ChartURL and LoadDirectory functions in order to support this for gitops and helmops.

If we need a secret to access the helm repository we can use the `helmSecretName` field. This secret will be cloned to secrets under the `BundleDeployment` namespace (same as we did for the OCI storage secret handling).

The PR includes unit, integration (most of code is tested this way) and just one single e2e test so far just to test the whole feature together in a real cluster.

Note: This is an experimental feature. In order to activate the `HelmApp` reconciling and `Bundle` deployment you need to the the environment variable: `EXPERIMENTAL_HELM_OPS=true`

Refers to: rancher#2962

.github/scripts/deploy-fleet.sh

@@ -66,6 +66,8 @@ eventually helm upgrade --install fleet charts/fleet \
   $shards_settings \
   --set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
   --set-string extraEnv[0].value=true \
+  --set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
+  --set-string extraEnv[1].value=true \
   --set garbageCollectionInterval=1s \
   --set debug=true --set debugLevel=1
 

README.md

@@ -33,7 +33,7 @@ For instance, using Homebrew:
 brew install helm
 ```
 
-Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)
+Install the Helm App charts (there's two because we separate out CRDs for ultimate flexibility.)
 
 ```shell
 helm -n cattle-fleet-system install --create-namespace --wait \

charts/fleet-agent/templates/deployment.yaml

@@ -62,6 +62,8 @@ spec:
             - ALL
         {{- end }}
         volumeMounts:
+          - mountPath: /tmp
+            name: tmp
           - mountPath: /.kube
             name: kube
       - env:
@@ -89,6 +91,8 @@ spec:
             - ALL
         {{- end }}
       volumes:
+        - name: tmp
+          emptyDir: {}
         - name: kube
           emptyDir: {}
       serviceAccountName: fleet-agent

charts/fleet-crd/README.md

@@ -1,5 +1,5 @@
 # Fleet CRD Helm Chart
 
-Fleet CustomResourceDefinitions Helm chart is a requirement for the Fleet Helm Chart.
+Fleet CustomResourceDefinitions Helm chart is a requirement for the Helm App Chart.
 
 The Fleet documentation is centralized in the [doc website](https://fleet.rancher.io/).

charts/fleet/README.md

@@ -1,4 +1,4 @@
-# Fleet Helm Chart
+# Helm App Chart
 
 Fleet is GitOps at scale. Fleet is designed to manage multiple clusters.
 
@@ -21,7 +21,7 @@ Get helm if you don't have it. Helm 3 is just a CLI.
 
 ## Install Fleet
 
-Install the Fleet Helm charts (there are two because we separate out CRDs for ultimate flexibility.):
+Install the Helm App charts (there are two because we separate out CRDs for ultimate flexibility.):
 
 ```
 $ helm repo add fleet https://rancher.github.io/fleet-helm-charts/

charts/fleet/ci/debug-values.yaml

@@ -52,6 +52,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nobootstrap-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nodebug-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nogitops-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/templates/deployment_helmops.yaml

@@ -0,0 +1,131 @@
+{{- $shards := list (dict "id" "" "nodeSelector" dict) -}}
+{{- $uniqueShards := list -}}
+{{- if .Values.shards -}}
+  {{- range .Values.shards -}}
+    {{- if not (has .id $uniqueShards) -}}
+      {{- $shards = append $shards . -}}
+      {{- $uniqueShards = append $uniqueShards .id -}}
+    {{- end -}}
+  {{- end -}}
+{{- end -}}
+
+{{ range $shard := $shards }}
+{{- if $.Values.gitops.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: "helmops{{if $shard.id }}-shard-{{ $shard.id }}{{end}}"
+spec:
+  selector:
+    matchLabels:
+      app: "helmops"
+  template:
+    metadata:
+      labels:
+        app: "helmops"
+        fleet.cattle.io/shard-id: "{{ $shard.id }}"
+        {{- if empty $shard.id }}
+        fleet.cattle.io/shard-default: "true"
+        {{- end }}
+    spec:
+      serviceAccountName: helmops
+      containers:
+        - image: "{{ template "system_default_registry" $ }}{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
+          name: helmops
+          {{- if $.Values.metrics.enabled }}
+          ports:
+          - containerPort: 8081
+            name: metrics
+          {{- end }}
+          args:
+          - fleetcontroller
+          - helmops
+          {{- if $.Values.debug }}
+          - --debug
+          - --debug-level
+          - {{ quote $.Values.debugLevel }}
+          {{- end }}
+          {{- if $shard.id }}
+          - --shard-id
+          - {{ quote $shard.id }}
+          {{- end }}
+          {{- if not $.Values.metrics.enabled }}
+          - --disable-metrics
+          {{- end }}
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          {{- if $.Values.leaderElection.leaseDuration }}
+            - name: CATTLE_ELECTION_LEASE_DURATION
+              value: {{$.Values.leaderElection.leaseDuration}}
+          {{- end }}
+          {{- if $.Values.leaderElection.retryPeriod }}
+            - name: CATTLE_ELECTION_RETRY_PERIOD
+              value: {{$.Values.leaderElection.retryPeriod}}
+          {{- end }}
+          {{- if $.Values.leaderElection.renewDeadline }}
+            - name: CATTLE_ELECTION_RENEW_DEADLINE
+              value: {{$.Values.leaderElection.renewDeadline}}
+          {{- end }}
+          {{- if $.Values.proxy }}
+            - name: HTTP_PROXY
+              value: {{ $.Values.proxy }}
+            - name: HTTPS_PROXY
+              value: {{ $.Values.proxy }}
+            - name: NO_PROXY
+              value: {{ $.Values.noProxy }}
+          {{- end }}
+          {{- if $.Values.controller.reconciler.workers.gitrepo }}
+            - name: HELMOPS_RECONCILER_WORKERS
+              value: {{ quote $.Values.controller.reconciler.workers.gitrepo }}
+          {{- end }}
+{{- if $.Values.extraEnv }}
+{{ toYaml $.Values.extraEnv | indent 12}}
+{{- end }}
+          {{- if $.Values.debug }}
+            - name: CATTLE_DEV_MODE
+              value: "true"
+          {{- end }}
+          {{- if not $.Values.disableSecurityContext }}
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            privileged: false
+            capabilities:
+                drop:
+                - ALL
+          {{- end }}
+          volumeMounts:
+            - mountPath: /tmp
+              name: tmp
+      nodeSelector: {{ include "linux-node-selector" $shard.id | nindent 8 }}
+{{- if $.Values.nodeSelector }}
+{{ toYaml $.Values.nodeSelector | indent 8 }}
+{{- end }}
+{{- if $shard.nodeSelector -}}
+{{- range $key, $value := $shard.nodeSelector }}
+{{ $key | indent 8}}: {{ $value }}
+{{- end }}
+{{- end }}
+      tolerations: {{ include "linux-node-tolerations" $shard.id | nindent 8 }}
+{{- if $.Values.tolerations }}
+{{ toYaml $.Values.tolerations | indent 8 }}
+{{- end }}
+      {{- if $.Values.priorityClassName }}
+      priorityClassName: "{{$.Values.priorityClassName}}"
+      {{- end }}
+
+{{- if not $.Values.disableSecurityContext }}
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+{{- end }}
+      volumes:
+        - name: tmp
+          emptyDir: {}
+{{- end }}
+---
+{{- end }}

charts/fleet/templates/rbac_helmops.yaml

@@ -0,0 +1,97 @@
+{{- if .Values.gitops.enabled }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: helmops
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - 'secrets'
+    verbs:
+      - "create"
+      - "list"
+  - apiGroups:
+      - ""
+    resources:
+      - 'configmaps'
+    verbs:
+      - '*'
+  - apiGroups:
+      - "fleet.cattle.io"
+    resources:
+      - "helmapps"
+      - "helmapps/status"
+    verbs:
+      - "*"
+  - apiGroups:
+      - "fleet.cattle.io"
+    resources:
+      - "bundles"
+      - "bundledeployments"
+    verbs:
+      - list
+      - delete
+      - get
+      - watch
+      - update
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - 'events'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - serviceaccounts
+    verbs:
+      - "create"
+  - apiGroups:
+      - ""
+    resources:
+      - namespaces
+    verbs:
+      - "create"
+      - "delete"
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: helmops-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: helmops
+subjects:
+  - kind: ServiceAccount
+    name: helmops
+    namespace: {{ .Release.Namespace }}
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: helmops
+rules:
+  - apiGroups:
+      - "coordination.k8s.io"
+    resources:
+      - "leases"
+    verbs:
+      - "*"
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: helmops
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: helmops
+subjects:
+  - kind: ServiceAccount
+    name: helmops
+{{- end }}

charts/fleet/templates/serviceaccount_helmops.yaml

@@ -0,0 +1,6 @@
+{{- if .Values.gitops.enabled }}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: helmops
+{{- end }}

dev/setup-fleet

@@ -42,6 +42,8 @@ helm -n cattle-fleet-system upgrade --install --create-namespace --wait --reset-
   $shards_settings \
   --set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
   --set-string extraEnv[0].value=true \
+  --set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
+  --set-string extraEnv[1].value=true \
   --set garbageCollectionInterval=1s \
   --set debug=true --set debugLevel=1 fleet charts/fleet
 

e2e/assets/helmapp/helmapp.yaml

@@ -0,0 +1,13 @@
+apiVersion: fleet.cattle.io/v1alpha1
+kind: HelmApp
+metadata:
+  name: {{.Name}}
+  namespace: "fleet-local"
+spec:
+  helm:
+    releaseName: testhelm
+    repo: {{.Repo}}
+    chart: {{.Chart}}
+  namespace: {{.Namespace}}
+  helmSecretName: {{.HelmSecretName}}
+  insecureSkipTLSVerify: true