Merge branch 'kapicorp:master' into master

docs/pages/blog/2023-08-27.md

@@ -0,0 +1,228 @@
+---
+author: Alessandro De Maria
+author_gh_user: ademariag
+read_time: 10m
+publish_date: 27/08/2023
+---
+
+# :kapitan-logo: **Deploying Keda with Kapitan**
+
+We have worked hard to bring out a brand new way of experience Kapitan, through something that we call [generators](https://generators.kapicorp.com)
+
+Although the concept is something we've introduced in 2020 with our blog post [Keep your ship together with Kapitan](https://medium.com/kapitan-blog/keep-your-ship-together-with-kapitan-d82d441cc3e7), the sheer amount of new capabilities (and frankly, the embarassing lack of documentation and examples) forces me to show you the new capabilities using a practicle example: deploying [Keda](https://keda.sh/docs/deploy/).
+
+## Objective of this tutorial
+
+We are going to deploy Keda using the helm chart approach. While Kapitan supports a native way to deploy helm charts using the helm input type, we are going instead to use a generator based approach using the "`charts`" generator.
+
+This tutorial will show you how to configure kapitan to:
+
+* download a helm chart
+* compile a helm chart
+* modify a helm chart using mutations
+
+The content of this tutorial is already available on the [`kapitan-reference`](https://github.com/kapicorp/kapitan-reference)
+
+## Deploying KEDA
+
+### Define parameters
+
+```yaml 
+## inventory/classes/components/keda.yml
+parameters:
+  keda:
+    params:
+      # Variables to reference from other places
+      application_version: 2.11.2
+      service_account_name: keda-operator
+      chart_name: keda
+      chart_version: 2.11.2
+      chart_dir: system/sources/charts/${keda:params:chart_name}/${keda:params:chart_name}/${keda:params:chart_version}/${keda:params:application_version}
+      namespace: keda
+      helm_values: {}
+...
+```
+
+!!! tip "Override Helm Values"
+    As an example we could be passing to helm an override to the default `values` parameters to make the operator deploy 2 replicas.
+
+    ```yaml
+      helm_values:
+        operator:
+          replicaCount: 2  
+    ```
+
+### Download the chart
+
+Kapitan supports downloading dependencies, including helm charts.
+
+When Kapitan is run with the `--fetch`, it will download the dependency if not already present.
+Use `--force-fetch` if you want to download it every time. Learn more about [External dependencies](https://kapitan.dev/pages/external_dependencies/#defining-dependencies)
+
+```yaml
+## inventory/classes/components/keda.yml
+...
+  kapitan:
+    dependencies:
+      # Tells kapitan to download the helm chart into the chart_dir directory
+      - type: helm
+        output_path: ${keda:params:chart_dir}
+        source: https://kedacore.github.io/charts
+        version: ${keda:params:chart_version}
+        chart_name: ${keda:params:chart_name}
+...
+```
+
+!!! tip "Parameter interpolation"
+    Notice how we are using parameter interpolation from the previously defined `keda.params` section. This will make it easier in the future to override some aspects of the configuration on a per-target base.
+
+### Generate the chart
+
+```yaml
+## inventory/classes/components/keda.yml
+...
+  charts:
+     # Configures a helm generator to compile files for the given chart
+    keda:
+      chart_dir: ${keda:params:chart_dir}
+      helm_params:
+        namespace: ${keda:params:namespace}
+        name: ${keda:params:chart_name}
+      helm_values: ${keda:params:helm_values}
+```
+
+### Compile
+
+Before we can see any effect, we need to attach the class to a target. We will create a simple target which looks like
+
+```yaml
+# inventory/targets/tutorials/keda.yml
+classes:
+- common
+- components.keda
+```
+
+Now when we run `kapitan compile` we will see the chart being donwloaded and the manifests being produced.
+
+
+```shell
+./kapitan compile -t keda --fetch
+Dependency keda: saved to system/sources/charts/keda/keda/2.11.2/2.11.2
+Rendered inventory (1.87s)
+Compiled keda (2.09s)
+```
+!!! note "`kapitan compile` breakdown"
+
+    * `--fetch` tells kapitan to fetch the chart if it is not found locally
+    * `-t keda` tells kapitan to compile only the previously defined `keda.yml` target
+
+```shell
+ls -l compiled/keda/manifests/
+total 660
+-rw-r--r-- 1 ademaria root 659081 Aug 29 10:25 keda-bundle.yml
+-rw-r--r-- 1 ademaria root     79 Aug 29 10:25 keda-namespace.yml
+-rw-r--r-- 1 ademaria root   7092 Aug 29 10:25 keda-rbac.yml
+-rw-r--r-- 1 ademaria root   1783 Aug 29 10:25 keda-service.yml
+```
+
+## Using mutations
+
+Now let's do a couple of things that would not be easy to do with helm natively.
+
+You can already notice that the content of the chart is being splitted into multiple files: this is because the Generator is configured to separate different resources types into different files for convenience and consistency. The mechanism behing it is the "Mutation" of type "bundle" which tells Kapitan which file to save a resource into.
+
+Here are some example "mutation" which separates different `kinds` into different files
+
+```yaml
+        mutations:
+          bundle:
+            - conditions:
+                kind: [Ingress]
+              filename: '{content.component_name}-ingress'
+              ...
+            - conditions:
+                kind: [HorizontalPodAutoscaler, PodDisruptionBudget, VerticalPodAutoscaler]
+              filename: '{content.component_name}-scaling'
+            - conditions:
+                kind: ['*']
+              filename: '{content.component_name}-bundle'
+```
+
+!!! tip "Catch-all rule"
+    Notice the catchall rule at the end that puts everything that has not matched into the `bundle.yml` file
+
+### `bundle` mutation`
+
+Currently most of the keda related resources are bundled into the `-bundle.yml` file
+Instead, we want to separate them into their own file.
+
+Let's add this configuration:
+
+```yaml
+
+  charts:
+     # Configures a helm generator to compile files for the given chart
+    keda:
+      chart_dir: ${keda:params:chart_dir}
+      ... 
+      mutations:
+        bundle:
+          - conditions:
+              # CRDs need to be setup separately
+              kind: [CustomResourceDefinition]
+            filename: '{content.component_name}-crds'
+```
+
+Upon compile, you can now see that the CRD are being moved to a different file:
+
+```shell
+ls -l compiled/keda/manifests/
+total 664
+-rw-r--r-- 1 ademaria root  11405 Aug 29 10:56 keda-bundle.yml
+-rw-r--r-- 1 ademaria root 647672 Aug 29 10:56 keda-crds.yml
+-rw-r--r-- 1 ademaria root     79 Aug 29 10:56 keda-namespace.yml
+-rw-r--r-- 1 ademaria root   7092 Aug 29 10:56 keda-rbac.yml
+-rw-r--r-- 1 ademaria root   1783 Aug 29 10:56 keda-service.yml
+```
+
+### `patch` mutation
+
+As we are using Argo, we want to pass a special `argocd.argoproj.io/sync-options` annotation to the CRD only so that ArgoCD can handle them properly.
+
+For this we are going to use the `patch` mutation:
+
+```yaml
+
+...
+      mutations:
+...
+        patch:
+          - conditions:
+              kind: [CustomResourceDefinition]
+            patch:
+              metadata:
+                annotations:
+                  argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true
+```
+
+Upon compile, you can now see that the CRDs have been modified as required:
+
+```shell
+
+diff --git a/compiled/keda/manifests/keda-crds.yml b/compiled/keda/manifests/keda-crds.yml
+index 2662bf3..9306c3a 100644
+--- a/compiled/keda/manifests/keda-crds.yml
++++ b/compiled/keda/manifests/keda-crds.yml
+@@ -2,6 +2,7 @@ apiVersion: apiextensions.k8s.io/v1
+ kind: CustomResourceDefinition
+ metadata:
+   annotations:
++    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true
+     controller-gen.kubebuilder.io/version: v0.12.0
+
+```
+
+## Summary
+
+With this tutorial have explored some capabilities of Kapitan to manage and perform changes to helm charts.
+Next tutorial will show how to make use of Keda and deploy a generator for Keda resources

kapitan/resources.py

@@ -210,12 +210,14 @@ def search_imports(cwd, import_str, search_paths):
     - search_paths is the location where to look for import_str if not in cwd
     The only supported parameters are cwd and import_str, so search_paths
     needs to be closured.
+    This function returns a tuple[str, bytes] since jsonnet 0.19.0 require the
+    content of the file to be provided as a bytes type instead of a str.
     """
     basename = os.path.basename(import_str)
     full_import_path = os.path.normpath(os.path.join(cwd, import_str))
 
     if full_import_path in JSONNET_CACHE:
-        return full_import_path, JSONNET_CACHE[full_import_path]
+        return full_import_path, JSONNET_CACHE[full_import_path].encode()
 
     if not os.path.exists(full_import_path):
         # if import_str not found, search in install_path
@@ -246,7 +248,7 @@ def search_imports(cwd, import_str, search_paths):
         normalised_path_content = f.read()
         JSONNET_CACHE[normalised_path] = normalised_path_content
 
-    return normalised_path, normalised_path_content
+    return normalised_path, normalised_path_content.encode()
 
 
 def inventory(search_paths, target, inventory_path=None):

mkdocs.yml

@@ -119,6 +119,7 @@ nav:
       - Terraform: pages/examples/terraform.md
   - Blog:
     - 2023:
+      - Kapitan Generators: pages/blog/2023-08-27.md
       - New Kapitan release: pages/blog/2023-01-16.md
     - 2022:
       - 5 years of Kapitan: pages/blog/2022-12-04.md

pyproject.toml

@@ -47,7 +47,7 @@ gitpython = "^3.1.30"
 google-api-python-client = "^2.15.0"
 hvac = "^0.11.0"
 jinja2 = "^3.0.1"
-jsonnet = "^0.18.0"
+jsonnet = "^0.20.0"
 jsonschema = "^4.17.3"
 kadet = "^0.2.2"
 python-gnupg = "^0.4.7"
@@ -62,7 +62,7 @@ certifi = "*"
 gitdb = "^4.0.10"
 packaging = "^23.0"
 typing-extensions = "^4.0.0"
-gojsonnet = { version = "^0.17.0", optional = true }
+gojsonnet = { version = "^0.20.0", optional = true }
 docker = { version = "^5.0.0", optional = true }
 
 [tool.poetry.extras]