diff --git a/docs/cert-manager.md b/docs/cert-manager.md index 3d5041c8862..c6d8f6583d5 100644 --- a/docs/cert-manager.md +++ b/docs/cert-manager.md @@ -1,77 +1,466 @@ -# MinIO tenant with cert-manager [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) +# TLS Certificate management with Cert Manager [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) -This document explains how to deploy a MinIO tenant using certificates generated -by [cert-manager](https://cert-manager.io/). +This document explains how to deploy MinIO Operator and a MinIO Tenant using certificates generated by [cert-manager](https://cert-manager.io/). + +On a Kubernetes cluster, Certificate Manager requires a global level `Cluster Issuer` to generate intermediary Issuers and certificates. +At the namespace level, Cert Manager issues certificates derived from an `Issuer`. +The main difference between a `Cluster Issuer` and an `Issuer` is that `Cluster Issuer` can issue certificates for several +namespaces and an `Issuer` can only issue them in a single namespace. + +To learn more about Cert Manager Issuers refer to their documentation https://cert-manager.io/docs/concepts/issuer/. + +Below is a logical view of a Kubernetes cluster: + +![Cert-manager-namespaces.png](images%2FCert-manager-namespaces.png) + +This cluster contains 4 namespaces: + +- minio-operator +- tenant-1 +- tenant-2 +- other-namespace + +This guide shows you how to set up different Certificate Authorities (CA) in each namespace, all of them referencing a global `Cluster Issuer`. + +![Cert-manager Issuers.png](images%2FCert-manager%20Issuers.png) + +Cert Manager is installed in the `cert-manager` namespace. + +The global `Cluster Issuer` is created in the default namespace. + +A local `Issuer` is created in each tenant namespace. + +An `Issuer` is also created in the `minio-operator` namespace. More about services that require TLS certificates +in the `minio-operator` namespace are covered below in [MinIO Operator services with cert-manager](# MinIO Operator services with cert-manager). + +> [!NOTE] +> This guide uses a self-signed `Cluster Issuer`. You can also use [other Issuers supported by Cert Manager](https://cert-manager.io/docs/configuration/issuers/). +> The main difference is that you must provide the `Issuer` CA certificate to minio, instead of the CA's mentioned in this guide. ## Getting Started ### Prerequisites - Kubernetes version `+v1.21`. While cert-manager - supports [earlier K8s versions](https://cert-manager.io/docs/installation/supported-releases/), the MinIO Operator - requires 1.21 or later. -- MinIO Operator installed + supports [earlier K8s versions](https://cert-manager.io/docs/installation/supported-releases/), MinIO Operator requires 1.21 or later. +- [kustomize](https://kustomize.io/) installed - `kubectl` access to your `k8s` cluster -- [cert-manager](https://cert-manager.io/docs/installation/) 1.7.X or later installed + +### Setup Cert-manager + +Install [cert-manager](https://cert-manager.io/docs/releases/release-notes/release-notes-1.12/) 1.12.X LTS is preferred, +or install latest, for example using kubectl. ```bash -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.2/cert-manager.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.9/cert-manager.yaml ``` -- [kustomize](https://kustomize.io/) installed +For more details on the Cert Manager refer to https://cert-manager.io/docs/installation/. -### Deploy tenant +### Create Cluster Self-signed root Issuer -Use the example to deploy a `MinIO` tenant using tls certificates generated by `cert-manager`, go into base folder of -the operator project and run the following command. +The `Cluster Issuer` is the cluster level Issuer all other certificates are derived from. Request Cert Manager to +generate this by creating a `ClusterIssuer` resource: -```bash -kustomize build examples/kustomization/tenant-certmanager | kubectl apply -f - +```yaml +# selfsigned-root-clusterissuer.yaml +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: selfsigned-root +spec: + selfSigned: {} ``` -This file request `cert-manager` to issue a new certificate based on the following internal domains. +```shell +kubectl apply -f selfsigned-root-clusterissuer.yaml +``` + +# MinIO Operator services with cert-manager + +MinIO Operator manages the TLS certificate issuing for the services hosted in the `minio-operator` namespace. These include Secure Token Service `sts` and the Console `console` service. + +This section describes how to generate the `sts` and `console` TLS certificates with Cert Manager. +These certificates must be issued before installing Operator. +Be sure to follow step [Create Cluster Self-signed root Issuer ](#Create Cluster Self-signed root Issuer) mentioned above. + +## Secure Token Service (STS) + +MinIO STS is a service included with MinIO Operator that provides Native IAM Authentication for Kubernetes. In essence +this service allows you to control access to your MinIO tenant from your kubernetes applications without having to explicitly create credentials +for each application. For more information on the Service see the MinIO docs at https://min.io/docs/minio/kubernetes/upstream/developers/sts-for-operator.html. +There is also an [STS](https://github.com/minio/operator/blob/master/docs/STS.md) guide in the docs and example client code in +https://github.com/minio/operator/tree/master/examples/kustomization/sts-example. + +For the purpose of this guide, STS Service can be considered a webserver presented with a TLS certificate for https traffic. +This guide covers how to **disable** the automatic generation of the certificate in MinIO Operator and issue the certificate using +Cert Manager instead. + +## Console service + +The console service is the well-known UI provided by Operator, it looks like the image below: + +![console-dashboard.png](images%2Fconsole-dashboard.png) + +When you install MinIO Operator, by default the Console service comes with TLS disabled. This guide explains +how to issue the TLS certificate for this service with Cert Manager and enable TLS for https traffic. + +### Create minio-operator namespace CA Issuer + +To create the Certificate Authority (CA) certificate used to issue certificates for services in the `minio-operator` +namespace, first create the minio-operator namespace + +```shell +kubectl create ns minio-operator +``` + +Request a Certificate with `spec.isCA: true` specified. This is our CA for the `minio-operator` namespace. ```yaml +# operator-ca-tls-secret.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: minio-operator-ca-certificate + namespace: minio-operator +spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f operator-ca-tls-secret.yaml +``` +A new secret with the name `operator-ca-tls` is created in the `minio-operator` namespace, this is the CA issuing TLS certificates for the services in the `minio-operator` namespace. + +> [!IMPORTANT] +> Make sure to trust this certificate in your applications that need to interact with either the `console` or the `sts` service. + + +Now create the `Issuer`: + +```yaml +# operator-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: minio-operator-ca-issuer + namespace: minio-operator +spec: + ca: + secretName: operator-ca-tls +``` + +```shell +kubectl apply -f operator-ca-issuer.yaml +``` + +### Create TLS certificate for STS service + +Request Cert Manager to issue a new certificate containing following DNS domains: + +```shell +sts +sts.minio-operator.svc. +sts.minio-operator.svc. +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> `cluster domain` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o yaml | yq ".data"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) + +Create a `Certificate` for the domains mentioned above: + +```yaml +# sts-tls-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: sts-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer +``` + +```shell +kubectl apply -f sts-tls-certificate.yaml +``` +This creates a secret called `sts-tls` in the `minio-operator` namespace. + +> [!IMPORTANT] +> The secret name is not optional. Make sure the secret name is `sts-tls` by setting `spec.secretName: sts-tls` as in the example above. + +### Create TLS certificate for Console service + +Request Cert Manager to issue a new certificate including the following DNS domains for console: + +```shell +console +console.minio-operator.svc. +console.minio-operator.svc. +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> `` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o yaml | yq ".data"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) + +Create a `Certificate` for the domains mentioned above: + +```yaml +# console-tls-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: console-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - console + - console.minio-operator.svc + - console.minio-operator.svc.cluster.local + secretName: console-tls + issuerRef: + name: minio-operator-ca-issuer +``` + +```shell +kubectl apply -f console-tls-certificate.yaml +``` + +This creates a secret called `console-tls` in the `minio-operator` namespace. + +> [!IMPORTANT] +> The secret name is not optional. Make sure the secret name is `console-tls` by setting `spec.secretName: console-tls` as in the example above. + +### Install Operator with Auto TLS disabled for STS + +When installing the Operator deployment, make sure to set `OPERATOR_STS_AUTO_TLS_ENABLED: off` env variable in the `minio-operator` container. This prevents +MinIO Operator from issuing the certificate for STS and instead wait for you to provide the TLS certificate issued by Cert Manager. + +Also make sure to set `MINIO_CONSOLE_TLS_ENABLE: off` env variable, this prevents MinIO Operator from issuing the certificate for Console +and instead will expect you to provide the TLS certificate issued by Cert Manager. + +> [!WARNING] +> Missing to provide the secret `sts-tls` containing the TLS certificate or providing an invalid key-pair in the secret will +> prevent the STS service from start. + +A way to make sure that the env variables are properly set is using kustomization to patch the `minio-operator` deployment: + +```yaml +# minio-operator/kustomization.yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: +- github.com/minio/operator/resources + +patches: +- patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: minio-operator + namespace: minio-operator + spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: MINIO_CONSOLE_TLS_ENABLE + value: "off" + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" +``` + +```shell +kubectl apply -k minio-operator +``` + +# MinIO tenant TLS certificates with cert-manager + +## Create tenant-1 namespace CA Issuer + +To create the Certificate Authority (CA) certificate in the namespace `tenant-1`, first create the namespace + +```shell +kubectl create ns tenant-1 +``` + +Next, request a Certificate with `spec.isCA: true` specified: + +```yaml +# tenant-1-ca-certificate.yaml +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-1-ca-certificate + namespace: tenant-1 +spec: + isCA: true + commonName: tenant-1-ca + secretName: tenant-1-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io +``` + +```shell +kubectl apply -f tenant-1-ca-certificate.yaml +``` + + +Then create the `Issuer`: + +```yaml +# tenant-1-ca-issuer.yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: tenant-1-ca-issuer + namespace: tenant-1 +spec: + ca: + secretName: tenant-1-ca-tls +``` + +```shell +kubectl apply -f tenant-1-ca-issuer.yaml +``` + +## Deploy the tenant + +### Create a certificate for Tenant + +Request Cert Manager issue a new TLS server certificate for MinIO that includes the following DNS domains: + +```shell +minio. +minio..svc +minio..svc. +*.-hl..svc. +*..svc. +*..minio..svc.' +``` + +> [!IMPORTANT] +> Replace `` with the actual values for your MinIO tenant. +> * `` is the internal root DNS domain assigned in your Kubernetes cluster. Typically this is `cluster.local`, check on your coredns +> configuration for the correct value for your Kubernetes cluster. For example, using `kubectl get configmap coredns -n kube-system -o yaml | yq ".data"`. +> The way the root DNS domain is managed can vary depending on the Kubernetes distribution (Openshift, Rancher, EKS, etc.) +> * `tenant-name` is the name provided to your tenant in the `metadata.name` of the Tenant YAML. For this example it is `myminio`. +> * `namespace` is the namespace where the tenant is created, the `metadata.namespace` notes that in the Tenant YAML. For this example it is `tenant-1`. + +Create a `Certificate` for the domains mentioned above: + +```yaml +# tenant-1-minio-certificate.yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: tenant-certmanager-cert - namespace: minio-tenant + namespace: tenant-1 spec: dnsNames: - - "*.tenant-certmanager.svc.cluster.local" - - "*.myminio.tenant-certmanager.svc.cluster.local" - - "*.myminio-hl.tenant-certmanager.svc.cluster.local" - secretName: tenant-certmanager-tls + - "minio.tenant-1" + - "minio.tenant-1.svc" + - 'minio.tenant-1.svc.cluster.local' + - '*.minio.tenant-1.svc.cluster.local' + - '*.myminio-hl.tenant-1.svc.cluster.local' + - '*.myminio.minio.tenant-1.svc.cluster.local' + secretName: myminio-tls issuerRef: - name: tenant-certmanager-issuer + name: tenant-1-ca-issuer +``` + +> [!TIP] +> For this example, the Tenant name is `myminio`. We recommend naming the secret in the field `spec.secretName` as `-tls`, following the naming +> convention MinIO Operator uses when creates certificates with Autocert enabled (`spec.requestAutoCert: true`). + +```shell +kubectl apply -f tenant-1-minio-certificate.yaml ``` -Then it creates a new tenant including the new `tenant-certmanager-tls` secret in the `externalCertSecret` field. +### Configure the tenant to use the certificate created by cert-manager + +In the tenant spec, do the following: + +* Disable Autocert `spec.requestAutoCert: false`. This instructs Operator to not attempt to issue certificates and instead rely on Cert Manager to provide them in a secret. +* Reference the Secret containing the TLS certificate from the previous step in `spec.externalCertSecret`. ```yaml +apiVersion: minio.min.io/v2 +kind: Tenant +metadata: + name: myminio + namespace: tenant-1 +spec: +... + ## Disable default tls certificates. + requestAutoCert: false ## Use certificates generated by cert-manager. externalCertSecret: - - name: tenant-certmanager-tls + - name: myminio-tls type: cert-manager.io/v1 +... ``` -### Create `operator-ca-tls` secret +## Trust tenant-1 CA in MinIO Operator -Copy the cert-manager CA from the tenant certificate, this will allow Operator to trust the cert-manager CA and allow Operator to trust the Tenant certificate +MinIO Operator can trust as many CA certificates as provided. To do this, create a secret with the prefix `operator-ca-tls-` +followed by a unique identifier in the `minio-operator` namespace. -```sh -kubectl get secrets -n tenant-certmanager tenant-certmanager-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt +MinIO Operator mounts and trusts all certificates issued by the provided CA's. This is required because +MinIO Operator performs health checks using the */minio/health/cluster* endpoint. + +If Operator is not correctly configured to trust the MinIO Certificate (or its CA), you will see an error message like the following in the Operator Pod logs: + +```error +Failed to get cluster health: Get "https://minio.tenant-1.svc.cluster.local/minio/health/cluster": +x509: certificate signed by unknown authority ``` +For more details about health checks, see https://min.io/docs/minio/Kubernetes/upstream/operations/monitoring/healthcheck-probe.html#cluster-write-quorum. -Create the secret +### Create `operator-ca-tls-tenant-1` secret +Copy the Cert Manager generated CA public key (ca.crt) into the `minio-operator` namespace. This allows Operator to trust +the cert-manager issued CA and the derived certificates. + +Create a `ca.crt` file containing the CA: ```sh -kubectl create secret generic operator-ca-tls --from-file=ca.crt -n minio-operator +kubectl get secrets -n tenant-1 tenant-1-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt ``` -Restart the minio-operator - +Create the secret: ```sh -kubectl rollout restart deployment.apps/minio-operator -n minio-operator +kubectl create secret generic operator-ca-tls-tenant-1 --from-file=ca.crt -n minio-operator ``` +> [!TIP] +> In this example we choose a secret name of `operator-ca-tls-tenant-1`. Note the tenant namespace +> `tenant-1` is used as suffix for easy identification of which namespace the CA is coming from. diff --git a/docs/env-variables.md b/docs/env-variables.md index 2f529af3a1a..5fd691b928f 100644 --- a/docs/env-variables.md +++ b/docs/env-variables.md @@ -2,16 +2,17 @@ Operator behavior can be customized using environment variables in the `minio-operator` deployment. Here is a list of the available environment variables: -| Variable Name | Description | Possible values | default | -| --- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|---------------------------------| -|MINIO_OPERATOR_CERTIFICATES_VERSION| This forces which certificate api version to use. | `v1`,`v1beta1` | whichever api k8s provides | -|MINIO_OPERATOR_RUNTIME | This tells us which runtime we have i.e. (EKS, Rancher, OpenShift, etc.). For example, if `EKS` is set, the CSR signerName to be used will be `"beta.eks.amazonaws.com/app-serving"` | EKS, Rancher, OpenShift | | -|MINIO_OPERATOR_CSR_SIGNER_NAME| The name to use for the CSR Signer. It will override the default | | `kubernetes.io/kubelet-serving` | -|SUBNET_BASE_URL| Subnet base URL | | https://subnet.min.io | -|OPERATOR_CERT_PASSWD| This is used to decrypt the private key in the TLS certificate for operator, if needed | | | -|MINIO_OPERATOR_DEPLOYMENT_NAME| This specifies a custom deployment name for Operator | | `minio-operator` | -|OPERATOR_STS_ENABLED| This toggles the STS Service on or off | `on`, `off` | `on` | -|MINIO_CONSOLE_DEPLOYMENT_NAME| This is the default name of the console deployment | | `console` | -|MINIO_CONSOLE_TLS_ENABLE| This toggles the Console TLS on or off | `on`, `off` | `off` | | | | -|WATCHED_NAMESPACE| The namespaces which the operator watches for MinIO tenants. Defaults to `""` for all namespaces. | | | -|MINIO_OPERATOR_IMAGE| This variable controls the image of the minio instance's sidecar and validate-arguments. if not set, the mirrors of the minio instance's sidecar and validate-arguments will use the operator's image. | "" | "" | \ No newline at end of file +| Variable Name | Description | Possible values | default | +| --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|---------------------------------| +|MINIO_OPERATOR_CERTIFICATES_VERSION| This forces which certificate api version to use. | `v1`,`v1beta1` | whichever api k8s provides | +|MINIO_OPERATOR_RUNTIME | This tells us which runtime we have i.e. (EKS, Rancher, OpenShift, etc.). For example, if `EKS` is set, the CSR signerName to be used will be `"beta.eks.amazonaws.com/app-serving"` | EKS, Rancher, OpenShift | | +|MINIO_OPERATOR_CSR_SIGNER_NAME| The name to use for the CSR Signer. It will override the default | | `kubernetes.io/kubelet-serving` | +|SUBNET_BASE_URL| Subnet base URL | | https://subnet.min.io | +|OPERATOR_CERT_PASSWD| This is used to decrypt the private key in the TLS certificate for operator, if needed | | | +|MINIO_OPERATOR_DEPLOYMENT_NAME| This specifies a custom deployment name for Operator | | `minio-operator` | +|OPERATOR_STS_ENABLED| This toggles the STS Service on or off | `on`, `off` | `on` | +|OPERATOR_STS_AUTO_TLS_ENABLED| Env variable name to turn on and off generating the STS TLS certificate automatically using CSR. If it is disabled, you must provide a certificate issued externally | `on`, `off` | `on` | +|MINIO_CONSOLE_DEPLOYMENT_NAME| This is the default name of the console deployment | | `console` | +|MINIO_CONSOLE_TLS_ENABLE| This toggles the Console TLS on or off | `on`, `off` | `off` | | | | +|WATCHED_NAMESPACE| The namespaces which the operator watches for MinIO tenants. Defaults to `""` for all namespaces. | | | +|MINIO_OPERATOR_IMAGE| This variable controls the image of the minio instance's sidecar and validate-arguments. If not set, the mirrors of the minio instance's sidecar and validate-arguments use the operator's image. | "" | "" | \ No newline at end of file diff --git a/docs/images/Cert-manager Issuers.png b/docs/images/Cert-manager Issuers.png new file mode 100644 index 00000000000..c34e5ae7584 Binary files /dev/null and b/docs/images/Cert-manager Issuers.png differ diff --git a/docs/images/Cert-manager-namespaces.png b/docs/images/Cert-manager-namespaces.png new file mode 100644 index 00000000000..583c4c4d548 Binary files /dev/null and b/docs/images/Cert-manager-namespaces.png differ diff --git a/docs/tls.md b/docs/tls.md index e6543074303..b6654bcd141 100644 --- a/docs/tls.md +++ b/docs/tls.md @@ -4,7 +4,7 @@ This document explains how to enable TLS on MinIO pods. ## Automatic TLS -This approach creates TLS certificates automatically using the Kubernetes cluster root Certificate Authority (CA) to establish trust. In this approach, MinIO Operator creates a private key, and a certificate signing request (CSR) which is submitted via the `certificates.k8s.io` API for signing. Automatic TLS approach creates other certificates required for KES as well as explained in [KES document](./kes.md). +This approach creates TLS certificates automatically using the Kubernetes cluster root Certificate Authority (CA) to establish trust. In this approach, MinIO Operator creates a private key and a certificate signing request (CSR) and submits them via the `certificates.k8s.io` API for signing. Automatic TLS approach creates other certificates required for KES as well as explained in [KES document](./kes.md). To enable automatic CSR generation on Tenant, set `requestAutoCert` field in the config file to `true`. Optionally you can also pass additional configuration parameters to be used under `certConfig` section. The `certConfig` section currently supports below fields: @@ -12,15 +12,15 @@ To enable automatic CSR generation on Tenant, set `requestAutoCert` field in the - organizationName: By default set to list `["system:nodes"]`. Change it to a list with the name of your organization, e.g., `organizationName: [my-org]`. -- dnsNames: By default set to list of all pod DNS names that are part of current Tenant. Any value added under this section will be appended to the list of existing pod DNS names. +- dnsNames: By default set to a list of all pod DNS names that are part of current Tenant. Any value added under this section will be appended to the list of existing pod DNS names. -Once you enable `requestAutoCert` field and create the Tenant, MinIO Operator creates a CSR for this instance and sends to the Kubernetes API server. MinIO Operator will then approve the CSR. After the CSR is approved and Certificate available, MinIO operator downloads the certificate and then mounts the Private Key and Certificate within the Tenant pod. +Once you enable the `requestAutoCert` field and create the Tenant, MinIO Operator creates a CSR for this instance and sends to the Kubernetes API server. MinIO Operator will then approve the CSR. After the CSR is approved and Certificate available, MinIO operator downloads the certificate and then mounts the Private Key and Certificate within the Tenant pod. --- ## Pass Certificate Secret to Tenant -This approach involves acquiring a CA signed or self-signed certificate and use a Kubernetes Secret resource to store this information. Once you have the key and certificate file available, create a Kubernetes Secret using +This approach involves acquiring a CA signed or self-signed certificate and using a Kubernetes Secret resource to store this information. Once you have the key and certificate file available, create a Kubernetes Secret with: ```bash kubectl create secret generic tls-ssl-minio --from-file=path/to/private.key --from-file=path/to/public.crt @@ -48,49 +48,5 @@ Once created, set the name of the Secret (in this example `tls-ssl-minio`) under ## Using cert-manager -[Certificate Manager](https://cert-manager.io) is a Kubernetes Operator capable of automatically issuing certificates from multiple Issuers. Integration with MinIO is simple. First, create a new certificate issuer; for this demonstration the issuer certificate will be self-signed: - -```yaml -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: selfsigning-issuer -spec: - selfSigned: {} -``` - -Now it's possible to issue the MinIO certificate using the above issuer: - -```yaml ---- -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: tls-minio -spec: - commonName: minio.example.com - secretName: tls-minio - dnsNames: - # Ingress domain - - minio.example.com - # Internal domain - - minio-hl-svc - - minio-hl-svc.default.svc.cluster.local - issuerRef: - name: selfsigning-issuer -``` - -Finally, configure MinIO to use the newly created TLS certificate: - -```yaml - externalCertSecret: - - name: tls-minio - type: kubernetes.io/tls -``` - -## Using your own CA Certificate for MinIO Operator - -If your `MinIO` tenants are using `custom certificates` or certificates generated by your own internal `certificate authority` (ie: `cert-manager`). -`MinIO Operator` needs to trust the `TLS` connections in order to talk to the `MinIO tenants`, for that you need to create a new secret in the -`minio-operator` namespace named `operator-ca-tls`, inside this secret create a new key `ca.crt` that will include the public certificate -for your internal certificate authority. +[Certificate Manager](https://cert-manager.io) is a Kubernetes Operator capable of automatically issuing certificates from multiple Issuers. +For instructions on using Cert Manager with MinIO please follow the guide in the [cert-manager.md](cert-manager.md) document. \ No newline at end of file diff --git a/examples/kustomization/cert-manager/kustomization.yaml b/examples/kustomization/cert-manager/kustomization.yaml new file mode 100644 index 00000000000..8264beadc02 --- /dev/null +++ b/examples/kustomization/cert-manager/kustomization.yaml @@ -0,0 +1,5 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - selfsigned-root-clusterissuer.yaml diff --git a/examples/kustomization/cert-manager/selfsigned-root-clusterissuer.yaml b/examples/kustomization/cert-manager/selfsigned-root-clusterissuer.yaml new file mode 100644 index 00000000000..479d3a4e359 --- /dev/null +++ b/examples/kustomization/cert-manager/selfsigned-root-clusterissuer.yaml @@ -0,0 +1,6 @@ +apiVersion: cert-manager.io/v1 +kind: ClusterIssuer +metadata: + name: selfsigned-root +spec: + selfSigned: {} diff --git a/examples/kustomization/operator-certmanager/console-tls-certificate.yaml b/examples/kustomization/operator-certmanager/console-tls-certificate.yaml new file mode 100644 index 00000000000..7c14904810d --- /dev/null +++ b/examples/kustomization/operator-certmanager/console-tls-certificate.yaml @@ -0,0 +1,13 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: console-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - console + - console.minio-operator.svc + - console.minio-operator.svc.cluster.local + secretName: console-tls + issuerRef: + name: minio-operator-ca-issuer diff --git a/examples/kustomization/operator-certmanager/kustomization.yaml b/examples/kustomization/operator-certmanager/kustomization.yaml new file mode 100644 index 00000000000..aeaf5c11393 --- /dev/null +++ b/examples/kustomization/operator-certmanager/kustomization.yaml @@ -0,0 +1,13 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../cert-manager + - ../../../resources + - operator-ca-tls-secret.yaml + - sts-tls-certificate.yaml + - console-tls-certificate.yaml + - minio-operator-ca-issuer.yaml + +patches: + - path: operator-deployment.yaml diff --git a/examples/kustomization/operator-certmanager/minio-operator-ca-issuer.yaml b/examples/kustomization/operator-certmanager/minio-operator-ca-issuer.yaml new file mode 100644 index 00000000000..9affeceb409 --- /dev/null +++ b/examples/kustomization/operator-certmanager/minio-operator-ca-issuer.yaml @@ -0,0 +1,8 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: minio-operator-ca-issuer + namespace: minio-operator +spec: + ca: + secretName: operator-ca-tls \ No newline at end of file diff --git a/examples/kustomization/operator-certmanager/operator-ca-tls-secret.yaml b/examples/kustomization/operator-certmanager/operator-ca-tls-secret.yaml new file mode 100644 index 00000000000..719adf29795 --- /dev/null +++ b/examples/kustomization/operator-certmanager/operator-ca-tls-secret.yaml @@ -0,0 +1,17 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: minio-operator-ca-certificate + namespace: minio-operator +spec: + isCA: true + commonName: operator + secretName: operator-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io diff --git a/examples/kustomization/operator-certmanager/operator-deployment.yaml b/examples/kustomization/operator-certmanager/operator-deployment.yaml new file mode 100644 index 00000000000..311d42d0694 --- /dev/null +++ b/examples/kustomization/operator-certmanager/operator-deployment.yaml @@ -0,0 +1,17 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: minio-operator + namespace: minio-operator +spec: + template: + spec: + containers: + - name: minio-operator + env: + - name: MINIO_CONSOLE_TLS_ENABLE + value: "off" + - name: OPERATOR_STS_AUTO_TLS_ENABLED + value: "off" + - name: OPERATOR_STS_ENABLED + value: "on" diff --git a/examples/kustomization/operator-certmanager/sts-tls-certificate.yaml b/examples/kustomization/operator-certmanager/sts-tls-certificate.yaml new file mode 100644 index 00000000000..41f686b36db --- /dev/null +++ b/examples/kustomization/operator-certmanager/sts-tls-certificate.yaml @@ -0,0 +1,13 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: sts-certmanager-cert + namespace: minio-operator +spec: + dnsNames: + - sts + - sts.minio-operator.svc + - sts.minio-operator.svc.cluster.local + secretName: sts-tls + issuerRef: + name: minio-operator-ca-issuer diff --git a/examples/kustomization/sts-example/sample-data/mc-job-setup-bucket.yaml b/examples/kustomization/sts-example/sample-data/mc-job-setup-bucket.yaml index 057fb55acdc..d7c96dea2c8 100644 --- a/examples/kustomization/sts-example/sample-data/mc-job-setup-bucket.yaml +++ b/examples/kustomization/sts-example/sample-data/mc-job-setup-bucket.yaml @@ -32,7 +32,7 @@ metadata: name: setup-bucket namespace: minio-tenant-1 spec: - backoffLimit: 5 + backoffLimit: 10 template: spec: serviceAccountName: mc-job-sa diff --git a/examples/kustomization/tenant-certmanager-kes/certificates.yaml b/examples/kustomization/tenant-certmanager-kes/certificates.yaml deleted file mode 100644 index 111e4216f44..00000000000 --- a/examples/kustomization/tenant-certmanager-kes/certificates.yaml +++ /dev/null @@ -1,35 +0,0 @@ -apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: tenant-certmanager-issuer - namespace: tenant-certmanager -spec: - selfSigned: { } ---- -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: tenant-certmanager-cert - namespace: tenant-certmanager -spec: - dnsNames: - - "*.tenant-certmanager.svc.cluster.local" - - "*.myminio.tenant-certmanager.svc.cluster.local" - - "*.myminio-hl.tenant-certmanager.svc.cluster.local" - secretName: tenant-certmanager-tls - issuerRef: - name: tenant-certmanager-issuer ---- -apiVersion: cert-manager.io/v1 -kind: Certificate -metadata: - name: tenant-certmanager-2-cert - namespace: tenant-certmanager -spec: - dnsNames: - - "*.tenant-certmanager.svc.cluster.local" - - "*.myminio.tenant-certmanager.svc.cluster.local" - - "*.myminio-hl.tenant-certmanager.svc.cluster.local" - secretName: tenant-certmanager-2-tls - issuerRef: - name: tenant-certmanager-issuer diff --git a/examples/kustomization/tenant-certmanager-kes/kustomization.yaml b/examples/kustomization/tenant-certmanager-kes/kustomization.yaml index 9593f2e0234..89a5a547ed8 100644 --- a/examples/kustomization/tenant-certmanager-kes/kustomization.yaml +++ b/examples/kustomization/tenant-certmanager-kes/kustomization.yaml @@ -4,7 +4,8 @@ kind: Kustomization namespace: tenant-certmanager-kes resources: - - certificates.yaml + - ../cert-manager + - myminio-kes-certificate.yaml - vault.yaml - ../tenant-kes-encryption diff --git a/examples/kustomization/tenant-certmanager-kes/myminio-kes-certificate.yaml b/examples/kustomization/tenant-certmanager-kes/myminio-kes-certificate.yaml new file mode 100644 index 00000000000..b7d5a7c2ff6 --- /dev/null +++ b/examples/kustomization/tenant-certmanager-kes/myminio-kes-certificate.yaml @@ -0,0 +1,15 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-certmanager-2-cert + namespace: tenant-certmanager +spec: + dnsNames: + - "minio.tenant-certmanager" + - "minio.tenant-certmanager.svc" + - "*.tenant-certmanager.svc.cluster.local" + - "*.myminio.tenant-certmanager.svc.cluster.local" + - "*.myminio-hl.tenant-certmanager.svc.cluster.local" + secretName: tenant-certmanager-2-tls + issuerRef: + name: tenant-certmanager-issuer diff --git a/examples/kustomization/tenant-certmanager/kustomization.yaml b/examples/kustomization/tenant-certmanager/kustomization.yaml index 106ddb022c1..802404daaf8 100644 --- a/examples/kustomization/tenant-certmanager/kustomization.yaml +++ b/examples/kustomization/tenant-certmanager/kustomization.yaml @@ -1,11 +1,12 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization - namespace: tenant-certmanager resources: - - certificates.yaml + - tenant-certmanager-issuer.yaml + - tenant-certmanager-ca-certificate.yaml + - tenant-certmanager-myminio-certificate.yaml - ../base -patchesStrategicMerge: - - tenant.yaml +patches: + - path: tenant.yaml diff --git a/examples/kustomization/tenant-certmanager/tenant-certmanager-ca-certificate.yaml b/examples/kustomization/tenant-certmanager/tenant-certmanager-ca-certificate.yaml new file mode 100644 index 00000000000..00568ea57b7 --- /dev/null +++ b/examples/kustomization/tenant-certmanager/tenant-certmanager-ca-certificate.yaml @@ -0,0 +1,17 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: tenant-certmanager-ca-certificate + namespace: tenant-certmanager +spec: + isCA: true + commonName: tenant-certmanager-ca + secretName: tenant-certmanager-ca-tls + duration: 70128h # 8y + privateKey: + algorithm: ECDSA + size: 256 + issuerRef: + name: selfsigned-root + kind: ClusterIssuer + group: cert-manager.io diff --git a/examples/kustomization/tenant-certmanager/tenant-certmanager-issuer.yaml b/examples/kustomization/tenant-certmanager/tenant-certmanager-issuer.yaml new file mode 100644 index 00000000000..facee06a396 --- /dev/null +++ b/examples/kustomization/tenant-certmanager/tenant-certmanager-issuer.yaml @@ -0,0 +1,7 @@ +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: tenant-certmanager-issuer + namespace: tenant-certmanager +spec: + selfSigned: { } diff --git a/examples/kustomization/tenant-certmanager/certificates.yaml b/examples/kustomization/tenant-certmanager/tenant-certmanager-myminio-certificate.yaml similarity index 73% rename from examples/kustomization/tenant-certmanager/certificates.yaml rename to examples/kustomization/tenant-certmanager/tenant-certmanager-myminio-certificate.yaml index c2c84f279d8..cb39810ff0b 100644 --- a/examples/kustomization/tenant-certmanager/certificates.yaml +++ b/examples/kustomization/tenant-certmanager/tenant-certmanager-myminio-certificate.yaml @@ -1,16 +1,8 @@ apiVersion: cert-manager.io/v1 -kind: Issuer -metadata: - name: tenant-certmanager-issuer - namespace: minio-tenant -spec: - selfSigned: { } ---- -apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: tenant-certmanager-cert - namespace: minio-tenant + namespace: tenant-certmanager spec: # If you see "Empty issuer DN not allowed in X509Certificates" # You will need to add spec.subject.organizations or spec.commonName @@ -23,9 +15,12 @@ spec: # - jetstack # commonName: example.com dnsNames: + - "minio.tenant-certmanager" + - "minio.tenant-certmanager.svc" + - "minio.tenant-certmanager.svc.cluster.local" + - "*.myminio-hl.tenant-certmanager.svc.cluster.local" - "*.tenant-certmanager.svc.cluster.local" - "*.myminio.tenant-certmanager.svc.cluster.local" - - "*.myminio-hl.tenant-certmanager.svc.cluster.local" - secretName: tenant-certmanager-tls + secretName: myminio-tls issuerRef: - name: tenant-certmanager-issuer \ No newline at end of file + name: tenant-certmanager-issuer diff --git a/examples/kustomization/tenant-certmanager/tenant.yaml b/examples/kustomization/tenant-certmanager/tenant.yaml index e8155b4453a..0ec92d16841 100644 --- a/examples/kustomization/tenant-certmanager/tenant.yaml +++ b/examples/kustomization/tenant-certmanager/tenant.yaml @@ -8,5 +8,5 @@ spec: requestAutoCert: false ## Use certificates generated by cert-manager. externalCertSecret: - - name: tenant-certmanager-tls + - name: myminio-tls type: cert-manager.io/v1 diff --git a/kustomization.yaml b/kustomization.yaml index dbaaf8f8a94..d480503034e 100644 --- a/kustomization.yaml +++ b/kustomization.yaml @@ -2,18 +2,16 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: minio-operator commonAnnotations: - operator.min.io/authors: "MinIO, Inc." - operator.min.io/license: "AGPLv3" - operator.min.io/support: "https://subnet.min.io" + operator.min.io/authors: MinIO, Inc. + operator.min.io/license: AGPLv3 + operator.min.io/support: https://subnet.min.io resources: - - resources/base/namespace.yaml - - resources/base/service-account.yaml - - resources/base/cluster-role.yaml - - resources/base/cluster-role-binding.yaml - - resources/base/crds/ - - resources/base/service.yaml - - resources/base/deployment.yaml - - resources/base/console-ui.yaml - - +- resources/base/namespace.yaml +- resources/base/service-account.yaml +- resources/base/cluster-role.yaml +- resources/base/cluster-role-binding.yaml +- resources/base/crds/ +- resources/base/service.yaml +- resources/base/deployment.yaml +- resources/base/console-ui.yaml diff --git a/pkg/controller/main-controller.go b/pkg/controller/main-controller.go index 91f7a5e0a1d..8e34682d458 100644 --- a/pkg/controller/main-controller.go +++ b/pkg/controller/main-controller.go @@ -417,7 +417,7 @@ func (c *Controller) startSTSAPIServer(ctx context.Context, notificationChannel } certsManager, err := xcerts.NewManager(ctx, publicCertPath, privateKeyPath, LoadX509KeyPair) if err != nil { - klog.Errorf("HTTPS STS API server failed to load CA certificate: %v", err) + klog.Errorf("HTTPS STS API server failed to load certificate: %v", err) notificationChannel <- &EventNotification{ Type: STSServerNotification, Err: err, @@ -501,9 +501,11 @@ func leaderRun(ctx context.Context, c *Controller, threadiness int, stopCh <-cha go func() { if utils.GetOperatorRuntime() == common.OperatorRuntimeOpenshift { klog.Infof("STS is enabled, skipping TLS certificate generation on Openshift deployment") - } else { - klog.Infof("STS is enabled, starting API certificate setup") + } else if IsSTSAutocertEnabled() { + klog.Infof("STS Autocert is enabled, starting API certificate setup.") c.generateSTSTLSCert() + } else { + klog.Infof("STS Autocert is disabled, skipping certificate generation.") } }() } diff --git a/pkg/controller/sts.go b/pkg/controller/sts.go index 1c9f1ac4472..6d64b6257a0 100644 --- a/pkg/controller/sts.go +++ b/pkg/controller/sts.go @@ -43,6 +43,10 @@ const ( // STSEnabled Env variable name to turn on and off the STS Service is enabled, disabled by default STSEnabled = "OPERATOR_STS_ENABLED" + // STSAutoTLSEnabled Env variable name to turn on and off generation of the STS TLS automatically using CSR, if disabled + // a certificate issued externally needs to be provided + STSAutoTLSEnabled = "OPERATOR_STS_AUTO_TLS_ENABLED" + // STSTLSSecretName is the name of secret created for the Operator STS TLS certs STSTLSSecretName = "sts-tls" ) @@ -398,6 +402,15 @@ func IsSTSEnabled() bool { return true } +// IsSTSAutocertEnabled Validates if STS Autocert is turned on, is enabled by default. +func IsSTSAutocertEnabled() bool { + value, set := os.LookupEnv(STSAutoTLSEnabled) + if set { + return value == "on" + } + return true +} + // generateConsoleTLSCert Issues the Operator Console TLS Certificate func (c *Controller) generateSTSTLSCert() (*string, *string) { return c.generateTLSCertificateForService("sts", STSTLSSecretName, getOperatorDeploymentName()) diff --git a/testing/certmanager/mc-admin-info-pod.yaml b/testing/certmanager/mc-admin-info-pod.yaml new file mode 100644 index 00000000000..0b2e6868277 --- /dev/null +++ b/testing/certmanager/mc-admin-info-pod.yaml @@ -0,0 +1,32 @@ +apiVersion: v1 +kind: Pod +metadata: + labels: + run: admin-mc + name: admin-mc + namespace: tenant-certmanager +spec: + containers: + - command: + - bash + - -c + - until (mc admin info myminio/ ); do echo 'waiting... for 5secs' && sleep 5; done + env: + - name: MC_HOST_myminio + value: https://minio:minio123@minio.tenant-certmanager.svc.cluster.local + image: quay.io/minio/mc + name: admin-mc + volumeMounts: + - name: certmanager-ca + mountPath: /root/.mc/certs/CAs/myminio.crt + subPath: tls.crt + volumes: + - name: certmanager-ca + secret: + secretName: myminio-tls + items: + - key: tls.crt + path: tls.crt + mode: 422 + dnsPolicy: ClusterFirst + restartPolicy: Never \ No newline at end of file diff --git a/testing/common.sh b/testing/common.sh index b2798de7547..c5a3300fd28 100644 --- a/testing/common.sh +++ b/testing/common.sh @@ -590,6 +590,12 @@ function install_operator() { echo "key, value for pod selector in kustomize test" key=name value=minio-operator + elif [ "$1" = "certmanager" ]; then + echo "Installing Current Operator with certmanager" + try kubectl apply -k "${SCRIPT_DIR}/../examples/kustomization/operator-certmanager" + echo "key, value for pod selector in kustomize test" + key=name + value=minio-operator else echo "Installing Current Operator" echo "When coming from the upgrade test, the operator is already installed." @@ -743,10 +749,25 @@ function check_tenant_status() { # File: operator/helm/tenant/values.yaml # Content: s3.bucketDNS: false echo "In helm values by default bucketDNS.s3 is disabled, skipping mc validation on helm test" + elif [ "$4" = "certmanager" ]; then + echo "Running mc admin info pod for tenant-certmanager-ca-tls secret..." + try kubectl apply -f "${SCRIPT_DIR}/certmanager/mc-admin-info-pod.yaml" + sleep 10 + echo "Wait to mc admin info minio/" + try kubectl wait --for=jsonpath="{.status.phase}=Succeeded" --timeout=10m pod/admin-mc --namespace tenant-certmanager + + # Retrieve the logs + kubectl logs admin-mc -n tenant-certmanager else - kubectl run admin-mc -i --tty --image quay.io/minio/mc \ + try kubectl run --restart=Never admin-mc --image quay.io/minio/mc \ --env="MC_HOST_minio=https://${USER}:${PASSWORD}@minio.${1}.svc.cluster.local" \ - --command -- bash -c "until (mc admin info minio/); do echo 'waiting... for 5secs' && sleep 5; done" + --command -- bash -c "until (mc admin info minio/ ); do echo 'waiting... for 5secs' && sleep 5; done" + sleep 10 + echo "Wait to mc admin info minio/" + try kubectl wait --for=jsonpath="{.status.phase}=Succeeded" --timeout=10m pod/admin-mc --namespace default + + # Retrieve the logs + kubectl logs admin-mc fi echo "Done." @@ -756,10 +777,10 @@ function check_tenant_status() { function install_cert_manager_tenant() { echo "Install cert-manager tenant from our example:" - try kubectl apply -k github.com/minio/operator/examples/kustomization/tenant-certmanager + try kubectl apply -k "${SCRIPT_DIR}/../examples/kustomization/tenant-certmanager" echo "Wait until tenant-certmanager-tls secret is generated by cert-manager..." - while ! kubectl get secret tenant-certmanager-tls --namespace tenant-certmanager + while ! kubectl get secret tenant-certmanager-ca-tls --namespace tenant-certmanager do echo "Waiting for my secret. Current secrets are:" kubectl get secrets -n tenant-certmanager @@ -769,9 +790,9 @@ function install_cert_manager_tenant() { # https://github.com/minio/operator/blob/master/docs/cert-manager.md echo "# Pass the CA cert to our Operator to trust the tenant:" echo "## First get the CA from cert-manager secret..." - try kubectl get secrets -n tenant-certmanager tenant-certmanager-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > public.crt + try kubectl get secrets -n tenant-certmanager tenant-certmanager-ca-tls -o=jsonpath='{.data.ca\.crt}' | base64 -d > public.crt echo "## Then create the secret in operator's namespace..." - try kubectl create secret generic operator-ca-tls --from-file=public.crt -n minio-operator + try kubectl create secret generic operator-ca-tls-tenant-certmanager --from-file=public.crt -n minio-operator echo "## Finally restart minio operator pods to catch up and trust tenant..." try kubectl rollout restart deployment.apps/minio-operator -n minio-operator @@ -852,7 +873,7 @@ function setup_sts_bucket() { namespace="minio-tenant-1" condition="condition=Complete" selector="metadata.name=setup-bucket" - try wait_for_resource_field_selector $namespace job $condition $selector + try wait_for_resource_field_selector $namespace job $condition $selector "600s" echo "Installing setup bucket job: DONE" } diff --git a/testing/deploy-cert-manager-tenant.sh b/testing/deploy-cert-manager-tenant.sh index 88326608e6f..1bd1a091f79 100755 --- a/testing/deploy-cert-manager-tenant.sh +++ b/testing/deploy-cert-manager-tenant.sh @@ -29,11 +29,11 @@ function main() { install_cert_manager - install_operator + install_operator certmanager install_cert_manager_tenant - check_tenant_status tenant-certmanager myminio + check_tenant_status tenant-certmanager myminio minio "certmanager" destroy_kind } diff --git a/testing/sts/operator/kustomization.yaml b/testing/sts/operator/kustomization.yaml index d41c4edaa38..9c9a55d9fdd 100644 --- a/testing/sts/operator/kustomization.yaml +++ b/testing/sts/operator/kustomization.yaml @@ -4,6 +4,6 @@ kind: Kustomization bases: - ../../../resources -patchesStrategicMerge: - - deployment.yaml - - console-deployment.yaml +patches: + - path: deployment.yaml + - path: console-deployment.yaml