diff --git a/charts/airflow/CHANGELOG.md b/charts/airflow/CHANGELOG.md index e11d1f59..392fdfde 100644 --- a/charts/airflow/CHANGELOG.md +++ b/charts/airflow/CHANGELOG.md @@ -8,16 +8,63 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html) ### Added - _see [Unreleased]_ -### Fixed +### Changed - _see [Unreleased]_ ### Removed - _see [Unreleased]_ -### Docs +### Fixed - _see [Unreleased]_ +## [8.4.0] - 2021-07-09 + +### Added +- allow referencing Secrets/ConfigMaps in `airflow.{users,connections,pools,variables}` ([#281](https://github.com/airflow-helm/charts/pull/281)) + - __WARNING:__ the meaning of `airflow.{usersUpdate,connectionsUpdate,poolsUpdate,variablesUpdate}` have changed, + they now configure if a Deployment is created to perpetually sync `airflow.{users,connections,pools,variables}`, + or if a single Job is created on each `helm upgrade ...` + - __Docs:__ + - [How to create airflow users?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-users) + - [How to create airflow connections?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-connections) + - [How to create airflow variables?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-variables) + - [How to create airflow pools?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-pools) + - __New Values:__ + - `airflow.usersTemplates` + - `airflow.connectionsTemplates` + - `airflow.variablesTemplates` + - `airflow.sync`: + - `airflow.sync.resources` + - `airflow.sync.nodeSelector` + - `airflow.sync.affinity` + - `airflow.sync.tolerations` + - `airflow.sync.securityContext` + - `airflow.sync.podLabels` + - `airflow.sync.annotations` + - `airflow.sync.podAnnotations` + - `airflow.sync.safeToEvict` +- removed the need for `helmWait` value ([#266](https://github.com/airflow-helm/charts/pull/266)) + - __NOTE__: this is great for users of the `--wait` flag with `helm install ...` (which is common in tools like [argo-cd](https://github.com/argoproj/argo-cd/)) + +### Changed +- the default `airflow.image` is now `apache/airflow:2.1.1-python3.8` ([#286](https://github.com/airflow-helm/charts/issues/286)) + - __NOTE:__ this does not affect support for older versions, see the [airflow version support matrix](https://github.com/airflow-helm/charts/tree/main/charts/airflow#airflow-version-support) +- the `Chart.yaml` now explicitly specifies `apiVersion=v2` (requiring helm 3) ([#278](https://github.com/airflow-helm/charts/issues/278)) +- the `requirements.yaml` file was removed in preference of the `v2` dependencies method (specifying in `Chart.yaml`) ([#278](https://github.com/airflow-helm/charts/issues/278)) +- git-sync containers are now deployed in webserver, regardless of `airflow.legacyCommands` ([#288](https://github.com/airflow-helm/charts/pull/288)) + - __NOTE:__ this allows users who store airflow plugins in their git repo to see them in the webserver +- `wait-for-db-migrations` init-containers now work properly when `airflow.legacyCommands=true` ([#271](https://github.com/airflow-helm/charts/pull/271)) + - __NOTE:__ this is important for airflow `1.10.X`, as previously Pods would not actually wait, and so would CrashLoopBackoff until the `upgrade-db` Job had finished +- improve validation of `{logs,dags}.persistence.accessMode` ([#269](https://github.com/airflow-helm/charts/pull/269)) + +### Fixed +- include volumeMounts in init-containers ([#255](https://github.com/airflow-helm/charts/pull/255)) +- add `release` to worker Service selector ([#267](https://github.com/airflow-helm/charts/pull/267)) +- mount `dags-data` with `readOnly=true` if `accessMode=ReadOnlyMany` ([#268](https://github.com/airflow-helm/charts/pull/268)) +- only validate `ingress.{web,flower}.path` if `ingress.enabled=true` ([#270](https://github.com/airflow-helm/charts/pull/270)) +- multiple Schedulers could run if `legacyCommands=true` (due to rollingUpdate) ([#272](https://github.com/airflow-helm/charts/pull/272)) + ## [8.3.2] - 2021-06-30 ### Docs - added this changelog ([#231](https://github.com/airflow-helm/charts/issues/231)) @@ -513,6 +560,7 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html) > https://github.com/helm/charts/tree/master/stable/airflow [Unreleased]: https://github.com/airflow-helm/charts/compare/airflow-8.3.2...HEAD +[8.4.0]: https://github.com/airflow-helm/charts/compare/airflow-8.3.2...airflow-8.4.0 [8.3.2]: https://github.com/airflow-helm/charts/compare/airflow-8.3.1...airflow-8.3.2 [8.3.1]: https://github.com/airflow-helm/charts/compare/airflow-8.3.0...airflow-8.3.1 [8.3.0]: https://github.com/airflow-helm/charts/compare/airflow-8.2.0...airflow-8.3.0 diff --git a/charts/airflow/requirements.lock b/charts/airflow/Chart.lock similarity index 84% rename from charts/airflow/requirements.lock rename to charts/airflow/Chart.lock index 0dc30b98..5dde93b3 100644 --- a/charts/airflow/requirements.lock +++ b/charts/airflow/Chart.lock @@ -6,4 +6,4 @@ dependencies: repository: https://charts.helm.sh/stable version: 10.5.7 digest: sha256:bfe30fcaf72a0609856bf1a5dab2b941e8ef89b30ba935aa820743a6758f4fc4 -generated: "2020-11-05T16:33:27.586085+11:00" +generated: "2021-07-09T14:11:48.265069+10:00" diff --git a/charts/airflow/Chart.yaml b/charts/airflow/Chart.yaml index 4820a639..57b4928a 100644 --- a/charts/airflow/Chart.yaml +++ b/charts/airflow/Chart.yaml @@ -1,8 +1,8 @@ -apiVersion: v1 +apiVersion: v2 description: the community Apache Airflow Helm Chart - used to deploy Apache Airflow on Kubernetes name: airflow -version: 8.3.2 -appVersion: 2.0.1 +version: 8.4.0 +appVersion: 2.1.1 icon: https://airflow.apache.org/_images/pin_large.png home: https://airflow.apache.org/ maintainers: @@ -16,3 +16,12 @@ keywords: - airflow - dag - workflow +dependencies: + - name: postgresql + version: 8.6.4 + repository: https://charts.helm.sh/stable + condition: postgresql.enabled + - name: redis + version: 10.5.7 + repository: https://charts.helm.sh/stable + condition: redis.enabled diff --git a/charts/airflow/README.md b/charts/airflow/README.md index 7fc6fcd6..5281fef7 100644 --- a/charts/airflow/README.md +++ b/charts/airflow/README.md @@ -1,17 +1,22 @@ -# Airflow Helm Chart (Community) [![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/airflow-helm)](https://artifacthub.io/packages/search?repo=airflow-helm) +# Airflow Helm Chart (User Community) -This chart provides a standard way to deploy [Apache Airflow](https://airflow.apache.org/) on your Kubernetes cluster, and is used by many thousands of companies for their production deployments of Airflow. +[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/airflow-helm)](https://artifacthub.io/packages/search?repo=airflow-helm) + +This chart provides a standard way to deploy [Apache Airflow](https://airflow.apache.org/) on your Kubernetes cluster, +and is used by thousands of companies for their production deployments of Airflow. > 🟦 __Discussion__ 🟦 -> -> This `community` chart is an alternative to the `official` chart found in the `apache/airflow` repo.
+> +> The `user community` chart is an alternative to the `official` chart found in the `apache/airflow` repo.
> There are differences between the charts, so you should evaluate which is better for your organisation. -> -> The goals of the `community` chart are:
-> (1) be easy to use & configure
-> (2) provide long-term support for older airflow versions
-> (3) automatically detect bad configs - and fail the install
-> (4) have useful documentation - written by people who actually use airflow in production +> +> The `user community` chart has existed since 2018 and was previously called `stable/airflow` on the official [helm/charts](https://github.com/helm/charts/tree/master/stable/airflow) repo. +> +> The goals of the `user community` chart are:
+> (1) be easy to configure
+> (2) support older airflow versions
+> (3) provide great documentation
+> (4) automatically detect bad configs
## Quickstart Guide @@ -19,8 +24,8 @@ These steps will allow you to quickly install Apache Airflow on your Kubernetes ### 1. Install the Chart -> 🟨 __Note__ 🟨 -> +> 🟨 __Note__ 🟨 +> > In production, we encourage using a tool like [ArgoCD](https://argoproj.github.io/argo-cd/), rather than running `helm install` manually ```sh @@ -65,23 +70,25 @@ The [CHANGELOG.md](CHANGELOG.md) is found at the root of this chart folder. ### Airflow Version Support +See [the guide here](#how-to-use-a-specific-version-of-airflow) on how to explicitly set your airflow version. + . | `1.10.X` | `2.0.X` | `2.1.X` --- | --- | --- | --- chart - `7.X.X` | ✅ | ❌ | ❌ chart - `8.X.X` | ✅ [1] | ✅ | ✅ -[0] [see here](#how-to-use-a-specific-version-of-airflow) for a guide on how to set your airflow version
 [1] you must set `airflow.legacyCommands = true` to use airflow `1.10.X` with chart `8.X.X` ### Airflow Executor Support +Set your airflow executor-type using the `airflow.executor` value. + . | `CeleryExecutor` | `KubernetesExecutor` | `CeleryKubernetesExecutor` --- | --- | --- | --- -chart - `7.X.X` | ✅ | ✅ | ❌ -chart - `8.X.X` | ✅ | ✅ [1] | ✅ [1] +chart - `7.X.X` | ✅ | ✅ [1] | ❌ +chart - `8.X.X` | ✅ | ✅ | ✅ -[0] you can set the executor using the `airflow.executor` value
 -[1] we encourage you to use airflow `2.X.X` with chart `8.X.X`, and make use of the new `airflow.kubernetesPodTemplate.*` values +[1] we encourage you to upgrade the chart to `8.X.X`, so you can use the `airflow.kubernetesPodTemplate` values (which require airflow `2.X.X`) ### Examples diff --git a/charts/airflow/requirements.yaml b/charts/airflow/requirements.yaml deleted file mode 100644 index b31bfae6..00000000 --- a/charts/airflow/requirements.yaml +++ /dev/null @@ -1,9 +0,0 @@ -dependencies: -- name: postgresql - version: 8.6.4 - repository: https://charts.helm.sh/stable - condition: postgresql.enabled -- name: redis - version: 10.5.7 - repository: https://charts.helm.sh/stable - condition: redis.enabled diff --git a/charts/airflow/values.yaml b/charts/airflow/values.yaml index e8ba9790..5826440e 100644 --- a/charts/airflow/values.yaml +++ b/charts/airflow/values.yaml @@ -10,7 +10,7 @@ airflow: ## image: repository: apache/airflow - tag: 2.0.1-python3.8 + tag: 2.1.1-python3.8 ## values: Always or IfNotPresent pullPolicy: IfNotPresent pullSecret: ""