Initial proposal of the new Helmops controller.

It adds a new custom resource `HelmApp` (resource name open to debate) that describes a helm chart to be deployed.

The resource contains all the fealds from the classic `fleet.yaml` file plus a few new from the `GitRepo`
resource.

`HelmApp` yaml example:

```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmApp
metadata:
  name: sample1
  namespace: fleet-local
spec:
  helm:
    releaseName: testhelm
    repo: https://charts.bitnami.com/bitnami
    chart: postgresql
    version: 16.2.1
  insecureSkipTLSVerify: true
```

The implementation tries to share as much as possible from a `Bundle` spec inside the new resource, because it helps
to "transform" the `HelmApp` into a deployment (no coversion is needed for most of the spec).

The new controller was also implemented splitting the functionality into 2 controllers (similar to what we did for the `GitRepo` controller). This allows us to reuse most of the status handling code, as display fields in the status of the new resource are as similar as possible to have consistent user experience and to integrate with the UI in the same way the `GitRepo` does.

When a new `HelmApp` resource is applied it is transformed into a single `Bundle`, adding some extra fields to let the `Bundle` reconciler know that this is not a regular `Bundle` coming from a `GitRepo`.

Similar as we did for OCI storage, the `Bundle` created from a `HelmApp` does not contain resources. The helm chart to be deployed is downloaded by the agent.

Code for downloading the helm chart is reused from gitops, so the same formats are supported.
Insecure TLS skipping was added the the ChartURL and LoadDirectory functions in order to support this for gitops and helmops.

If we need a secret to access the helm repository we can use the `helmSecretName` field. This secret will be cloned to secrets under the `BundleDeployment` namespace (same as we did for the OCI storage secret handling).

The PR includes unit, integration (most of code is tested this way) and just one single e2e test so far just to test the whole feature together in a real cluster.

Note: This is an experimental feature. In order to activate the `HelmApp` reconciling and `Bundle` deployment you need to the the environment variable: `EXPERIMENTAL_HELM_OPS=true`

Refers to: rancher#2962

.github/scripts/deploy-fleet.sh

@@ -66,6 +66,8 @@ eventually helm upgrade --install fleet charts/fleet \
   $shards_settings \
   --set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
   --set-string extraEnv[0].value=true \
+  --set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
+  --set-string extraEnv[1].value=true \
   --set garbageCollectionInterval=1s \
   --set debug=true --set debugLevel=1
 

charts/fleet-agent/templates/deployment.yaml

@@ -62,6 +62,8 @@ spec:
             - ALL
         {{- end }}
         volumeMounts:
+          - mountPath: /tmp
+            name: tmp
           - mountPath: /.kube
             name: kube
       - env:
@@ -89,6 +91,8 @@ spec:
             - ALL
         {{- end }}
       volumes:
+        - name: tmp
+          emptyDir: {}
         - name: kube
           emptyDir: {}
       serviceAccountName: fleet-agent

charts/fleet/ci/debug-values.yaml

@@ -52,6 +52,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nobootstrap-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nodebug-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/ci/nogitops-values.yaml

@@ -51,6 +51,8 @@ controller:
 extraEnv:
   - name: EXPERIMENTAL_OCI_STORAGE
     value: "true"
+  - name: EXPERIMENTAL_HELM_OPS
+    value: "true"
 
 shards:
   - id: shard0

charts/fleet/templates/deployment_helmops.yaml

@@ -0,0 +1,131 @@
+{{- $shards := list (dict "id" "" "nodeSelector" dict) -}}
+{{- $uniqueShards := list -}}
+{{- if .Values.shards -}}
+  {{- range .Values.shards -}}
+    {{- if not (has .id $uniqueShards) -}}
+      {{- $shards = append $shards . -}}
+      {{- $uniqueShards = append $uniqueShards .id -}}
+    {{- end -}}
+  {{- end -}}
+{{- end -}}
+
+{{ range $shard := $shards }}
+{{- if has (dict "name" "EXPERIMENTAL_HELM_OPS" "value" "true") $.Values.extraEnv }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: "helmops{{if $shard.id }}-shard-{{ $shard.id }}{{end}}"
+spec:
+  selector:
+    matchLabels:
+      app: "helmops"
+  template:
+    metadata:
+      labels:
+        app: "helmops"
+        fleet.cattle.io/shard-id: "{{ $shard.id }}"
+        {{- if empty $shard.id }}
+        fleet.cattle.io/shard-default: "true"
+        {{- end }}
+    spec:
+      serviceAccountName: helmops
+      containers:
+        - image: "{{ template "system_default_registry" $ }}{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
+          name: helmops
+          {{- if $.Values.metrics.enabled }}
+          ports:
+          - containerPort: 8081
+            name: metrics
+          {{- end }}
+          args:
+          - fleetcontroller
+          - helmops
+          {{- if $.Values.debug }}
+          - --debug
+          - --debug-level
+          - {{ quote $.Values.debugLevel }}
+          {{- end }}
+          {{- if $shard.id }}
+          - --shard-id
+          - {{ quote $shard.id }}
+          {{- end }}
+          {{- if not $.Values.metrics.enabled }}
+          - --disable-metrics
+          {{- end }}
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          {{- if $.Values.leaderElection.leaseDuration }}
+            - name: CATTLE_ELECTION_LEASE_DURATION
+              value: {{$.Values.leaderElection.leaseDuration}}
+          {{- end }}
+          {{- if $.Values.leaderElection.retryPeriod }}
+            - name: CATTLE_ELECTION_RETRY_PERIOD
+              value: {{$.Values.leaderElection.retryPeriod}}
+          {{- end }}
+          {{- if $.Values.leaderElection.renewDeadline }}
+            - name: CATTLE_ELECTION_RENEW_DEADLINE
+              value: {{$.Values.leaderElection.renewDeadline}}
+          {{- end }}
+          {{- if $.Values.proxy }}
+            - name: HTTP_PROXY
+              value: {{ $.Values.proxy }}
+            - name: HTTPS_PROXY
+              value: {{ $.Values.proxy }}
+            - name: NO_PROXY
+              value: {{ $.Values.noProxy }}
+          {{- end }}
+          {{- if $.Values.controller.reconciler.workers.gitrepo }}
+            - name: HELMOPS_RECONCILER_WORKERS
+              value: {{ quote $.Values.controller.reconciler.workers.gitrepo }}
+          {{- end }}
+{{- if $.Values.extraEnv }}
+{{ toYaml $.Values.extraEnv | indent 12}}
+{{- end }}
+          {{- if $.Values.debug }}
+            - name: CATTLE_DEV_MODE
+              value: "true"
+          {{- end }}
+          {{- if not $.Values.disableSecurityContext }}
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            privileged: false
+            capabilities:
+                drop:
+                - ALL
+          {{- end }}
+          volumeMounts:
+            - mountPath: /tmp
+              name: tmp
+      nodeSelector: {{ include "linux-node-selector" $shard.id | nindent 8 }}
+{{- if $.Values.nodeSelector }}
+{{ toYaml $.Values.nodeSelector | indent 8 }}
+{{- end }}
+{{- if $shard.nodeSelector -}}
+{{- range $key, $value := $shard.nodeSelector }}
+{{ $key | indent 8}}: {{ $value }}
+{{- end }}
+{{- end }}
+      tolerations: {{ include "linux-node-tolerations" $shard.id | nindent 8 }}
+{{- if $.Values.tolerations }}
+{{ toYaml $.Values.tolerations | indent 8 }}
+{{- end }}
+      {{- if $.Values.priorityClassName }}
+      priorityClassName: "{{$.Values.priorityClassName}}"
+      {{- end }}
+
+{{- if not $.Values.disableSecurityContext }}
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+{{- end }}
+      volumes:
+        - name: tmp
+          emptyDir: {}
+{{- end }}
+---
+{{- end }}

charts/fleet/templates/rbac_helmops.yaml

@@ -0,0 +1,97 @@
+{{- if has (dict "name" "EXPERIMENTAL_HELM_OPS" "value" "true") .Values.extraEnv }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: helmops
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - 'secrets'
+    verbs:
+      - "create"
+      - "list"
+  - apiGroups:
+      - ""
+    resources:
+      - 'configmaps'
+    verbs:
+      - '*'
+  - apiGroups:
+      - "fleet.cattle.io"
+    resources:
+      - "helmapps"
+      - "helmapps/status"
+    verbs:
+      - "*"
+  - apiGroups:
+      - "fleet.cattle.io"
+    resources:
+      - "bundles"
+      - "bundledeployments"
+    verbs:
+      - list
+      - delete
+      - get
+      - watch
+      - update
+      - create
+  - apiGroups:
+      - ""
+    resources:
+      - 'events'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - serviceaccounts
+    verbs:
+      - "create"
+  - apiGroups:
+      - ""
+    resources:
+      - namespaces
+    verbs:
+      - "create"
+      - "delete"
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: helmops-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: helmops
+subjects:
+  - kind: ServiceAccount
+    name: helmops
+    namespace: {{ .Release.Namespace }}
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: helmops
+rules:
+  - apiGroups:
+      - "coordination.k8s.io"
+    resources:
+      - "leases"
+    verbs:
+      - "*"
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: helmops
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: helmops
+subjects:
+  - kind: ServiceAccount
+    name: helmops
+{{- end }}

charts/fleet/templates/serviceaccount_helmops.yaml

@@ -0,0 +1,6 @@
+{{- if has (dict "name" "EXPERIMENTAL_HELM_OPS" "value" "true") .Values.extraEnv }}
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: helmops
+{{- end }}

charts/fleet/values.yaml

@@ -109,6 +109,8 @@ controller:
 # extraEnv:
 # - name: EXPERIMENTAL_OCI_STORAGE
 #   value: "true"
+# - name: EXPERIMENTAL_HELM_OPS
+#   value: "true"
 
 # shards:
 #   - id: shard0

dev/setup-fleet

@@ -42,6 +42,8 @@ helm -n cattle-fleet-system upgrade --install --create-namespace --wait --reset-
   $shards_settings \
   --set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
   --set-string extraEnv[0].value=true \
+  --set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
+  --set-string extraEnv[1].value=true \
   --set garbageCollectionInterval=1s \
   --set debug=true --set debugLevel=1 fleet charts/fleet
 

e2e/assets/helmapp/helmapp.yaml

@@ -0,0 +1,13 @@
+apiVersion: fleet.cattle.io/v1alpha1
+kind: HelmApp
+metadata:
+  name: {{.Name}}
+  namespace: "fleet-local"
+spec:
+  helm:
+    releaseName: testhelm
+    repo: {{.Repo}}
+    chart: {{.Chart}}
+  namespace: {{.Namespace}}
+  helmSecretName: {{.HelmSecretName}}
+  insecureSkipTLSVerify: true