Trust custom Certificate Authority

If your AWX has to trust custom Certificate Authority, you can pass the CA certificates to AWX. This is helpful in cases:

Use private Git repository via SSL, without ignoring SSL verification.
Use LDAPS to authenticate users.

Refer to the official documentation for more information.

Note that if you want to trust custom CA for jobs (running playbooks or accessing inventory sources), a different approach is required. Refer to Appendix: Trust custom CA for jobs.

Overview

Trusting custom Certificate Authority can be achieved by following steps:

Creating new Secret which includes your certificates
Passing it to your AWX by specifying the name of the Secret as your AWX's specification

There are two kinds of certificate, one is used to trust LDAP server, and the other is used as the CA bundle.

Fields in the specification for AWX	Keys in Secret	Containers that the certificate will be mounted	Paths that the certificate will be mounted as
`ldap_cacert_secret`	`ldap-ca.crt`	`awx-web`	`/etc/openldap/certs/ldap-ca.crt`
`bundle_cacert_secret`	`bundle-ca.crt`	`awx-web`, `awx-task`, and `awx-ee`	`/etc/pki/ca-trust/source/anchors/bundle-ca.crt`

Note that the awx-ee container is used to run management jobs only, this is not EE which runs your playbooks. If the EE running your playbook needs certificates, refer Appendix: Trust custom CA for jobs on the bottom.

Prepare required CA certificates

Place your certificates under base directory.

$ ls -l base
total 32
-rw-rw-r--. 1 kuro kuro  801 Feb 27 00:23 awx.yaml
-rw-rw-r--. 1 kuro kuro 1339 Feb 27 00:44 cacert.pem   👈👈👈
-rw-rw-r--. 1 kuro kuro  610 Feb 27 00:23 kustomization.yaml
...

Note that your certificates have to have PEM format. You can check the format of the certificates depending on which of the following commands succeeds.

# Works for PEM format
openssl x509 -in cacert.crt -text

# Works for DER format
openssl x509 -in cacert.crt -inform DER -text

# Works for PKCS #7 format
openssl pkcs7 -in cacert.crt -text

# Works for PKCS #12 format
openssl pkcs12 -in cacert.crt -info

If your certificate doesn't have PEM format, you can convert it by followings:

# Convert DER to PEM
openssl x509 -in cacert.crt -inform DER -out cacert.pem -outform PEM

# Convert PKCS #7 to PEM
openssl pkcs7 -print_certs -in cacert.crt -out cacert.pem -outform PEM

# Convert PKCS #12 to PEM
openssl pkcs12 -in cacert.crt -out cacert.pem -nokeys -nodes

Modify `base/kustomization.yaml`

Add following lines under secretGenerator in base/kustomization.yaml.

Note that this example provides both ldap-ca.crt and bundle-ca.crt, but you can remove unnecessary line if you don't need both of them. ldap-ca.crt will be used as the CA certificate for LDAP server, and bundle-ca.crt will be used as the CA bundle.

...
secretGenerator:
  ...
  - name: awx-custom-certs                              👈👈👈
    type: Opaque                                        👈👈👈
    files:                                              👈👈👈
      - ldap-ca.crt=<Name Of Your Certificate File>     👈👈👈
      - bundle-ca.crt=<Name Of Your Certificate File>   👈👈👈
  ...

Modify `base/awx.yaml`

Add following lines under secretGenerator in base/kustomization.yaml.

Note that this example provides both ldap_cacert_secret (should have ldap-ca.crt) and bundle_cacert_secret (should have bundle-ca.crt), but you can remove unnecessary line if you don't need both of them.

...
spec:
  ...
  ldap_cacert_secret: awx-custom-certs     👈👈👈
  bundle_cacert_secret: awx-custom-certs   👈👈👈
  ...

Apply configuration

Invoke apply command. This will start re-deployment of your AWX.

kubectl apply -k base

You can monitor the progress of the re-deployment by following command:

kubectl -n awx logs -f deployments/awx-operator-controller-manager

Troubleshooting

If you have problem with SSL connection such as LDAPS, you can verify your certificates inside the pod.

# Open Bash shell of the "awx-web" container
$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$

First of all, you should ensure your CA certificate is mounted and has PEM format. The certificate should be be dumped as readable plain text by following command, without any error.

# The secret ldap_cacert_secret is mounted as /etc/openldap/certs/ldap-ca.crt
bash-5.1$ openssl x509 -in /etc/openldap/certs/ldap-ca.crt -text

# The secret bundle_cacert_secret is mounted as /etc/pki/ca-trust/source/anchors/bundle-ca.crt
bash-5.1$ openssl x509 -in /etc/pki/ca-trust/source/anchors/bundle-ca.crt -text

Note that your certificate file should contain both intermediate CA and root CA, if your server certificate is signed by intermediate CA.

# Example output of concatenated CA cert; one for intermediate CA, one for root CA
bash-5.1$ cat /etc/openldap/certs/ldap-ca.crt
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlswDQYJKoZIhvcNAQEL
...
3Eyhv0l7mJw/86twDMFFax+cKOCRFV6NoPOpzK1mzAXmxth6vk8DeRm0ipVpQVQ=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIUftINZYmeHvcovY0qBHp+SqZWrlwwDQYJKoZIhvcNAQEL
...
lVsDxZfbZVpRGkDr8odNurNmz0Xcttr+ZVRkoTy5KUxqIZhQuS6ySJj7yoLawWY=
-----END CERTIFICATE-----

Now you can test SSL connection.

# This is an example to test connection to LDAP server over SSL using /etc/openldap/certs/ldap-ca.crt
bash-5.1$ echo | openssl s_client -connect ldap.example.com:636 -no-CAfile -CAfile /etc/openldap/certs/ldap-ca.crt
CONNECTED(00000003)
depth=2 C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
verify return:1
depth=1 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
verify return:1
depth=0 C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
verify return:1
---
Certificate chain                👈👈👈 Ensure that the full certificate chain is recognized
 0 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ldap.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
   ...
 1 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = ica.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   ...
 2 s:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   i:C = JP, ST = Example State, O = EXAMPLE.COM, CN = rca.example.com
   ...
---
...
---
SSL handshake has read 3210 bytes and written 413 bytes
Verification: OK                 👈👈👈 Ensure there is no verification error
---
...
SSL-Session:
    ...
    Verify return code: 0 (ok)   👈👈👈 Ensure there is no verification error
    ...

Appendix: Trust custom CA for jobs

The bundle_cacert_secret stated above does not add any certificates to automation job pods, so it can't be used to trust custom CA to run jobs e.g. running playbooks, accessing inventory sources.

To trust custom CA for your jobs, there are several solutions depend on your modules or plugins that used in your jobs.

Update system certificate store with custom CA certificate on automation job pods.
Place custom CA certificates on automation job pods and pass its path through environment variable or parameters for modules or plugins.

This guide provides some ways as examples to achieve above solutions by invoking update-ca-trust and adding environment variable REQUESTS_CA_BUNDLE for requests python module (since requests does not refer system certificate store, custom CA certificates should be passed through this environment variable).

Method 1: Use Container Group
- Add required certificates and environment variable on the automation job pods by mounting it using Container Group with custom pod specification.
Method 2: Mount host filesystem
- Add required certificates and environment variable on the automation job pods by mounting host filesystem.
Method 3: Add certificate file to EE image
- Add required certificates and environment variable on the automation job pods by adding it to the EE image.

In this example, the value of the environment variable REQUESTS_CA_BUNDLE is specified as /etc/pki/tls/cert.pem. This file is a symbolic link to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem that updated by update-ca-trust by combining and includes all CA certificates in the system certificate store into one file.

Method 1: Use Container Group

This method can be used for standard jobs, such as the case that some modules in your playbook require custom CA certificates. In addition, this method can also be used to access inventory sources.

Create Secret

Prepare CA certificate file with PEM format (refer to Prepare required CA certificates) first, then create Secret with the certificate.

kubectl -n awx create secret generic awx-custom-job-certs \
    --from-file=job-ca.crt=cacert.pem

Create Container Group on AWX

Create new Container Group by Administration > Instance Group > Add > Add container group on AWX, then enable Customize pod specification and put following specification.

This specification supports both jobs that refers system certificate store and the store for requests.

apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: default
  automountServiceAccountToken: false
  initContainers:
    - name: init
      image: quay.io/ansible/awx-ee:latest
      command:
        - /bin/sh
        - -c
        - |
          mkdir -p /etc/pki/ca-trust/extracted/{java,pem,openssl,edk2}
          update-ca-trust
      volumeMounts:
        - name: ca-trust-extracted
          mountPath: /etc/pki/ca-trust/extracted
        - name: awx-custom-job-certs
          mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
          subPath: job-ca.crt
          readOnly: true
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      env:
        - name: REQUESTS_CA_BUNDLE
          value: /etc/pki/tls/cert.pem
      resources:
        requests:
          cpu: 250m
          memory: 100Mi
      volumeMounts:
        - name: ca-trust-extracted
          mountPath: /etc/pki/ca-trust/extracted
        - name: awx-custom-job-certs
          mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
          subPath: job-ca.crt
          readOnly: true
  volumes:
    - name: ca-trust-extracted
      emptyDir: {}
    - name: awx-custom-job-certs
      secret:
        secretName: awx-custom-job-certs
        items:
          - key: job-ca.crt
            path: job-ca.crt

This specification has following changes from the default one:

Init container (initContainers) is added to invoke update-ca-trust to update certificate store using CA certificate that mounted through Secret resource.
Worker container mounts certificate store that updated by init container (volumeMounts).
Worker container has environment variable REQUESTS_CA_BUNDLE (env) that has path for updated certificate file.
Volumes (volumes) are added to achieve above changes.

If you just want to place CA certificates somewhere on automation job pods and update-ca-trust is not required for you, you can ignore initContainers and ca-trust-extracted (under volumes and volumeMounts), and change mountPath for awx-custom-job-certs.

Specify Container Group

The Container Group can be specified as Instance Groups for the Job Template or Organization.

If you want to use this Container Group to access inventory sources, specify it as Instance Groups for the Inventory or Organization.

Method 2: Mount host filesystem

This method can be used for both standard jobs and inventory sync.

Note that this method changes the global configuration of your K3s host and AWX, so whole system and other applications on your K3s host, and all automation job pods running on the AWX are also affected.

Update certificate store

Prepare CA certificate file with PEM format (refer to Prepare required CA certificates) first, then place the certificate file under /etc/pki/ca-trust/source/anchors on your K3s host, and invoke update-ca-trust.

$ ls -l /etc/pki/ca-trust/source/anchors
total 4
-rw-r--r--. 1 root root 1489 Apr 27 22:49 cacert.pem

$ sudo update-ca-trust

If your Kubernetes cluster has multiple nodes, this step have to be invoked on all nodes where your automation job pods will be launched on.

If you just want to place CA certificates somewhere on automation job pods and update-ca-trust is not required for you, you can ignore update-ca-trust.

Modify options on AWX

On Settings > Jobs settings > Edit page, modify following two options.

Enable Expose host paths for Container Groups.
Ensure "/etc/pki/ca-trust:/etc/pki/ca-trust:O" is specified in Paths to expose to isolated jobs.

Add environment variables to jobs

To add environment variables to your jobs, on Settings > Jobs settings > Edit page, modify Extra Environment Variables with following value. If the environment variable is not required for you, this step can be ignored.

{
  "REQUESTS_CA_BUNDLE": "/etc/pki/tls/cert.pem"
}

Method 3: Add certificate file to EE image

This method can be used for both standard jobs and inventory sync, and can be achieved by following two ways.

Add files to the existing image
Build new EE using Ansible Builder

Add files to the existing image

Prepare CA certificate file with PEM format (refer to Prepare required CA certificates), then you can add the certificate to existing image using following Dockerfile.

This example shows the way to add local certificate file (cacert.pem) and environment variable to quay.io/ansible/awx-ee:latest.

FROM quay.io/ansible/awx-ee:latest

USER root
COPY cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
RUN update-ca-trust
ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem

USER 1000

Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.

Build new EE using Ansible Builder

Prepare CA certificate file with PEM format (refer to Prepare required CA certificates), then you can add additional_build_files and additional_build_steps.append_final to your execution-environment.yml to add the certificate and environment variable to your EE.

...
additional_build_files:
  - src: certs/cacert.pem
    dest: certs

additional_build_steps:
  append_final:
    - COPY _build/certs/cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
    - RUN update-ca-trust
    - ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem

Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trust-custom-ca.md

trust-custom-ca.md

Trust custom Certificate Authority

Table of Contents

Overview

Prepare required CA certificates

Modify `base/kustomization.yaml`

Modify `base/awx.yaml`

Apply configuration

Troubleshooting

Appendix: Trust custom CA for jobs

Method 1: Use Container Group

Create Secret

Create Container Group on AWX

Specify Container Group

Method 2: Mount host filesystem

Update certificate store

Modify options on AWX

Add environment variables to jobs

Method 3: Add certificate file to EE image

Add files to the existing image

Build new EE using Ansible Builder

Files

trust-custom-ca.md

Latest commit

History

trust-custom-ca.md

File metadata and controls

Trust custom Certificate Authority

Table of Contents

Overview

Prepare required CA certificates

Modify base/kustomization.yaml

Modify base/awx.yaml

Apply configuration

Troubleshooting

Appendix: Trust custom CA for jobs

Method 1: Use Container Group

Create Secret

Create Container Group on AWX

Specify Container Group

Method 2: Mount host filesystem

Update certificate store

Modify options on AWX

Add environment variables to jobs

Method 3: Add certificate file to EE image

Add files to the existing image

Build new EE using Ansible Builder

Modify `base/kustomization.yaml`

Modify `base/awx.yaml`