Skip to content

Commit

Permalink
Initial proposal of the new Helmops controller.
Browse files Browse the repository at this point in the history
It adds a new custom resource `HelmApp` (resource name open to debate) that describes a helm chart to be deployed.

The resource contains all the fealds from the classic `fleet.yaml` file plus a few new from the `GitRepo`
resource.

`HelmApp` yaml example:

```yaml
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmApp
metadata:
  name: sample1
  namespace: fleet-local
spec:
  helm:
    releaseName: testhelm
    repo: https://charts.bitnami.com/bitnami
    chart: postgresql
    version: 16.2.1
  insecureSkipTLSVerify: true
```

The implementation tries to share as much as possible from a `Bundle` spec inside the new resource, because it helps
to "transform" the `HelmApp` into a deployment (no coversion is needed for most of the spec).

The new controller was also implemented splitting the functionality into 2 controllers (similar to what we did for the `GitRepo` controller). This allows us to reuse most of the status handling code, as display fields in the status of the new resource are as similar as possible to have consistent user experience and to integrate with the UI in the same way the `GitRepo` does.

When a new `HelmApp` resource is applied it is transformed into a single `Bundle`, adding some extra fields to let the `Bundle` reconciler know that this is not a regular `Bundle` coming from a `GitRepo`.

Similar as we did for OCI storage, the `Bundle` created from a `HelmApp` does not contain resources. The helm chart to be deployed is downloaded by the agent.

Code for downloading the helm chart is reused from gitops, so the same formats are supported.
Insecure TLS skipping was added the the ChartURL and LoadDirectory functions in order to support this for gitops and helmops.

If we need a secret to access the helm repository we can use the `helmSecretName` field. This secret will be cloned to secrets under the `BundleDeployment` namespace (same as we did for the OCI storage secret handling).

The PR includes unit, integration (most of code is tested this way) and just one single e2e test so far just to test the whole feature together in a real cluster.

Note: This is an experimental feature. In order to activate the `HelmApp` reconciling and `Bundle` deployment you need to the the environment variable: `EXPERIMENTAL_HELM_OPS=true`

Refers to: rancher#2962
  • Loading branch information
0xavi0 committed Nov 22, 2024
1 parent ec03adc commit 58483d0
Show file tree
Hide file tree
Showing 63 changed files with 6,896 additions and 341 deletions.
2 changes: 2 additions & 0 deletions .github/scripts/deploy-fleet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ eventually helm upgrade --install fleet charts/fleet \
$shards_settings \
--set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
--set-string extraEnv[0].value=true \
--set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
--set-string extraEnv[1].value=true \
--set garbageCollectionInterval=1s \
--set debug=true --set debugLevel=1

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ For instance, using Homebrew:
brew install helm
```

Install the Fleet Helm charts (there's two because we separate out CRDs for ultimate flexibility.)
Install the Helm App charts (there's two because we separate out CRDs for ultimate flexibility.)

```shell
helm -n cattle-fleet-system install --create-namespace --wait \
Expand Down
4 changes: 4 additions & 0 deletions charts/fleet-agent/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ spec:
- ALL
{{- end }}
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /.kube
name: kube
- env:
Expand Down Expand Up @@ -89,6 +91,8 @@ spec:
- ALL
{{- end }}
volumes:
- name: tmp
emptyDir: {}
- name: kube
emptyDir: {}
serviceAccountName: fleet-agent
Expand Down
2 changes: 1 addition & 1 deletion charts/fleet-crd/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Fleet CRD Helm Chart

Fleet CustomResourceDefinitions Helm chart is a requirement for the Fleet Helm Chart.
Fleet CustomResourceDefinitions Helm chart is a requirement for the Helm App Chart.

The Fleet documentation is centralized in the [doc website](https://fleet.rancher.io/).
2,305 changes: 2,305 additions & 0 deletions charts/fleet-crd/templates/crds.yaml

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions charts/fleet/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Fleet Helm Chart
# Helm App Chart

Fleet is GitOps at scale. Fleet is designed to manage multiple clusters.

Expand All @@ -21,7 +21,7 @@ Get helm if you don't have it. Helm 3 is just a CLI.

## Install Fleet

Install the Fleet Helm charts (there are two because we separate out CRDs for ultimate flexibility.):
Install the Helm App charts (there are two because we separate out CRDs for ultimate flexibility.):

```
$ helm repo add fleet https://rancher.github.io/fleet-helm-charts/
Expand Down
2 changes: 2 additions & 0 deletions charts/fleet/ci/debug-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ controller:
extraEnv:
- name: EXPERIMENTAL_OCI_STORAGE
value: "true"
- name: EXPERIMENTAL_HELM_OPS
value: "true"

shards:
- id: shard0
Expand Down
2 changes: 2 additions & 0 deletions charts/fleet/ci/nobootstrap-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ controller:
extraEnv:
- name: EXPERIMENTAL_OCI_STORAGE
value: "true"
- name: EXPERIMENTAL_HELM_OPS
value: "true"

shards:
- id: shard0
Expand Down
2 changes: 2 additions & 0 deletions charts/fleet/ci/nodebug-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ controller:
extraEnv:
- name: EXPERIMENTAL_OCI_STORAGE
value: "true"
- name: EXPERIMENTAL_HELM_OPS
value: "true"

shards:
- id: shard0
Expand Down
2 changes: 2 additions & 0 deletions charts/fleet/ci/nogitops-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ controller:
extraEnv:
- name: EXPERIMENTAL_OCI_STORAGE
value: "true"
- name: EXPERIMENTAL_HELM_OPS
value: "true"

shards:
- id: shard0
Expand Down
131 changes: 131 additions & 0 deletions charts/fleet/templates/deployment_helmops.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
{{- $shards := list (dict "id" "" "nodeSelector" dict) -}}
{{- $uniqueShards := list -}}
{{- if .Values.shards -}}
{{- range .Values.shards -}}
{{- if not (has .id $uniqueShards) -}}
{{- $shards = append $shards . -}}
{{- $uniqueShards = append $uniqueShards .id -}}
{{- end -}}
{{- end -}}
{{- end -}}

{{ range $shard := $shards }}
{{- if $.Values.gitops.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: "helmops{{if $shard.id }}-shard-{{ $shard.id }}{{end}}"
spec:
selector:
matchLabels:
app: "helmops"
template:
metadata:
labels:
app: "helmops"
fleet.cattle.io/shard-id: "{{ $shard.id }}"
{{- if empty $shard.id }}
fleet.cattle.io/shard-default: "true"
{{- end }}
spec:
serviceAccountName: helmops
containers:
- image: "{{ template "system_default_registry" $ }}{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
name: helmops
{{- if $.Values.metrics.enabled }}
ports:
- containerPort: 8081
name: metrics
{{- end }}
args:
- fleetcontroller
- helmops
{{- if $.Values.debug }}
- --debug
- --debug-level
- {{ quote $.Values.debugLevel }}
{{- end }}
{{- if $shard.id }}
- --shard-id
- {{ quote $shard.id }}
{{- end }}
{{- if not $.Values.metrics.enabled }}
- --disable-metrics
{{- end }}
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
{{- if $.Values.leaderElection.leaseDuration }}
- name: CATTLE_ELECTION_LEASE_DURATION
value: {{$.Values.leaderElection.leaseDuration}}
{{- end }}
{{- if $.Values.leaderElection.retryPeriod }}
- name: CATTLE_ELECTION_RETRY_PERIOD
value: {{$.Values.leaderElection.retryPeriod}}
{{- end }}
{{- if $.Values.leaderElection.renewDeadline }}
- name: CATTLE_ELECTION_RENEW_DEADLINE
value: {{$.Values.leaderElection.renewDeadline}}
{{- end }}
{{- if $.Values.proxy }}
- name: HTTP_PROXY
value: {{ $.Values.proxy }}
- name: HTTPS_PROXY
value: {{ $.Values.proxy }}
- name: NO_PROXY
value: {{ $.Values.noProxy }}
{{- end }}
{{- if $.Values.controller.reconciler.workers.gitrepo }}
- name: HELMOPS_RECONCILER_WORKERS
value: {{ quote $.Values.controller.reconciler.workers.gitrepo }}
{{- end }}
{{- if $.Values.extraEnv }}
{{ toYaml $.Values.extraEnv | indent 12}}
{{- end }}
{{- if $.Values.debug }}
- name: CATTLE_DEV_MODE
value: "true"
{{- end }}
{{- if not $.Values.disableSecurityContext }}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
privileged: false
capabilities:
drop:
- ALL
{{- end }}
volumeMounts:
- mountPath: /tmp
name: tmp
nodeSelector: {{ include "linux-node-selector" $shard.id | nindent 8 }}
{{- if $.Values.nodeSelector }}
{{ toYaml $.Values.nodeSelector | indent 8 }}
{{- end }}
{{- if $shard.nodeSelector -}}
{{- range $key, $value := $shard.nodeSelector }}
{{ $key | indent 8}}: {{ $value }}
{{- end }}
{{- end }}
tolerations: {{ include "linux-node-tolerations" $shard.id | nindent 8 }}
{{- if $.Values.tolerations }}
{{ toYaml $.Values.tolerations | indent 8 }}
{{- end }}
{{- if $.Values.priorityClassName }}
priorityClassName: "{{$.Values.priorityClassName}}"
{{- end }}

{{- if not $.Values.disableSecurityContext }}
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
{{- end }}
volumes:
- name: tmp
emptyDir: {}
{{- end }}
---
{{- end }}
97 changes: 97 additions & 0 deletions charts/fleet/templates/rbac_helmops.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{{- if .Values.gitops.enabled }}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: helmops
rules:
- apiGroups:
- ""
resources:
- 'secrets'
verbs:
- "create"
- "list"
- apiGroups:
- ""
resources:
- 'configmaps'
verbs:
- '*'
- apiGroups:
- "fleet.cattle.io"
resources:
- "helmapps"
- "helmapps/status"
verbs:
- "*"
- apiGroups:
- "fleet.cattle.io"
resources:
- "bundles"
- "bundledeployments"
verbs:
- list
- delete
- get
- watch
- update
- create
- apiGroups:
- ""
resources:
- 'events'
verbs:
- '*'
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- "create"
- apiGroups:
- ""
resources:
- namespaces
verbs:
- "create"
- "delete"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: helmops-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: helmops
subjects:
- kind: ServiceAccount
name: helmops
namespace: {{ .Release.Namespace }}

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: helmops
rules:
- apiGroups:
- "coordination.k8s.io"
resources:
- "leases"
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: helmops
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: helmops
subjects:
- kind: ServiceAccount
name: helmops
{{- end }}
6 changes: 6 additions & 0 deletions charts/fleet/templates/serviceaccount_helmops.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{{- if .Values.gitops.enabled }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: helmops
{{- end }}
2 changes: 2 additions & 0 deletions dev/setup-fleet
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ helm -n cattle-fleet-system upgrade --install --create-namespace --wait --reset-
$shards_settings \
--set-string extraEnv[0].name=EXPERIMENTAL_OCI_STORAGE \
--set-string extraEnv[0].value=true \
--set-string extraEnv[1].name=EXPERIMENTAL_HELM_OPS \
--set-string extraEnv[1].value=true \
--set garbageCollectionInterval=1s \
--set debug=true --set debugLevel=1 fleet charts/fleet

Expand Down
13 changes: 13 additions & 0 deletions e2e/assets/helmapp/helmapp.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmApp
metadata:
name: {{.Name}}
namespace: "fleet-local"
spec:
helm:
releaseName: testhelm
repo: {{.Repo}}
chart: {{.Chart}}
namespace: {{.Namespace}}
helmSecretName: {{.HelmSecretName}}
insecureSkipTLSVerify: true
Loading

0 comments on commit 58483d0

Please sign in to comment.