feat: implement extra_settings_files #125

kurokobo · 2024-05-24T13:33:58Z

SUMMARY

This PR implements extra_settings_files, which is almost the same as my PR for AWX Operator ansible/awx-operator#1836

Required module (split_settings) to load additional *.py files is already installed in galaxy images: https://github.com/ansible/galaxy_ng/blob/c52e9b4e5548ca38946d7cc1bd8624459def1ec1/requirements/requirements.common.txt#L127-L128

Design:

Users can add extra settings by mounting *.py files from ConfigMaps or Secrets
This feature mounts specified *.py files under /etc/pulp/conf.d in following pods which already have settngs.py
- galaxy-api
- galaxy-content
- galaxy-worker
Users can specify multiple ConfigMaps and Secrets
Users can include multiple files in a single ConfigMap or Secret

Restrictions:

Completely the same as described in ansible/awx-operator#1836

ADDITIONAL INFORMATION

Tested with and without extra_settings_files:

Without extra_settings_files

---
apiVersion: galaxy.ansible.com/v1beta1
kind: Galaxy
metadata:
  namespace: galaxy
  name: galaxy
spec:
  ingress_type: nodeport
  nodeport_port: "30180"
  storage_type: File
  file_storage_access_mode: ReadWriteOnce
  file_storage_size: 8Gi
  no_log: false

  image: quay.io/ansible/galaxy-ng
  image_version: 9d2f8ce1
  image_web: quay.io/ansible/galaxy-ui
  image_web_version: 6116e760

Ensure there is no /etc/pulp/conf.d

$ kubectl -n galaxy exec -it deployment/galaxy-api -- ls -l /etc/pulp
total 4
drwxrwxr-x. 2 galaxy root    6 May  1 13:01 certs
drwxrwxr-x. 1 galaxy root  118 May 24 12:53 keys
-rw-r--r--. 1 root   root 1531 May 24 12:52 settings.py

Even if there is no actual directory and files, include() does not cause error since optional() is used to load files.

$ kubectl -n galaxy exec -it deployment/galaxy-api -- cat /etc/pulp/settings.py
import os
import traceback

from split_settings.tools import optional, include

GALAXY_API_PATH_PREFIX = "api/galaxy"
...
CSRF_TRUSTED_ORIGINS = ["http://192.168.0.221:30180", "https://192.168.0.221:30180"]

# Attempt to load settings from /etc/pulp/conf.d/*.py.
settings_dir = os.environ.get('PULP_SETTINGS_DIR', '/etc/pulp/conf.d/')
settings_files = os.path.join(settings_dir, '*.py')
try:
    include(optional(settings_files), scope=locals())
except ImportError:
    traceback.print_exc()
    sys.exit(1)

With extra_settings_files

---
apiVersion: galaxy.ansible.com/v1beta1
kind: Galaxy
metadata:
  namespace: galaxy
  name: galaxy
spec:
  ingress_type: nodeport
  nodeport_port: "30180"
  storage_type: File
  file_storage_access_mode: ReadWriteOnce
  file_storage_size: 8Gi
  no_log: false

  image: quay.io/ansible/galaxy-ng
  image_version: 9d2f8ce1
  image_web: quay.io/ansible/galaxy-ui
  image_web_version: 6116e760

  extra_settings_files:
    configmaps:
      - name: demo-configmap-01
        key: democonfigmap01_01.py
    secrets:
      - name: demo-secret-01
        key: demosecret01_01.py
      - name: demo-secret-01
        key: demosecret01_02.py
      - name: demo-secret-02
        key: demosecret02_01.py

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: galaxy
  name: demo-configmap-01
data:
  democonfigmap01_01.py: |
    GALAXY_FEATURE_FLAGS = {
      "display_repositories": True,
      # True by default so set False for testing
      "execution_environments": False,
      "legacy_roles": False,
      "ai_deny_index": False,
      "dab_resource_registry": False,
    }
---
apiVersion: v1
kind: Secret
metadata:
  namespace: galaxy
  name: demo-secret-01
stringData:
  demosecret01_01.py: |
    # False by default
    GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_ACCESS = True
  demosecret01_02.py: |
    # False by default
    GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_DOWNLOAD = True
---
apiVersion: v1
kind: Secret
metadata:
  namespace: galaxy
  name: demo-secret-02
stringData:
  demosecret02_01.py: |
    GALAXY_COLLECTION_SIGNING_SERVICE = None
    GALAXY_CONTAINER_SIGNING_SERVICE = None
    TOKEN_AUTH_DISABLED = True

Result:

$ kubectl -n galaxy exec -it deployment/galaxy-api -- ls -l /etc/pulp/conf.d
total 16
-rw-r--r--. 1 root root 224 May 24 12:59 democonfigmap01_01.py
-rw-r--r--. 1 root root  74 May 24 12:59 demosecret01_01.py
-rw-r--r--. 1 root root  76 May 24 12:59 demosecret01_02.py
-rw-r--r--. 1 root root 108 May 24 13:06 demosecret02_01.py

$ kubectl -n galaxy exec -it deployment/galaxy-content -- ls -l /etc/pulp/conf.d
total 16
-rw-r--r--. 1 root 1000 224 May 24 12:59 democonfigmap01_01.py
-rw-r--r--. 1 root 1000  74 May 24 12:59 demosecret01_01.py
-rw-r--r--. 1 root 1000  76 May 24 12:59 demosecret01_02.py
-rw-r--r--. 1 root 1000 108 May 24 13:05 demosecret02_01.py

$ kubectl -n galaxy exec -it deployment/galaxy-worker -- ls -l /etc/pulp/conf.d
total 16
-rw-r--r--. 1 root root 224 May 24 12:59 democonfigmap01_01.py
-rw-r--r--. 1 root root  74 May 24 12:59 demosecret01_01.py
-rw-r--r--. 1 root root  76 May 24 12:59 demosecret01_02.py
-rw-r--r--. 1 root root 108 May 24 13:06 demosecret02_01.py

$ kubectl -n galaxy exec -it deployment/galaxy-api -- grep -r "" /etc/pulp/conf.d 
/etc/pulp/conf.d/demosecret02_01.py:GALAXY_COLLECTION_SIGNING_SERVICE = None
/etc/pulp/conf.d/demosecret02_01.py:GALAXY_CONTAINER_SIGNING_SERVICE = None
/etc/pulp/conf.d/demosecret02_01.py:TOKEN_AUTH_DISABLED = True
/etc/pulp/conf.d/demosecret01_02.py:# False by default
/etc/pulp/conf.d/demosecret01_02.py:GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_DOWNLOAD = True
/etc/pulp/conf.d/demosecret01_01.py:# False by default
/etc/pulp/conf.d/demosecret01_01.py:GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_ACCESS = True
/etc/pulp/conf.d/democonfigmap01_01.py:GALAXY_FEATURE_FLAGS = {
/etc/pulp/conf.d/democonfigmap01_01.py:  "display_repositories": True,
/etc/pulp/conf.d/democonfigmap01_01.py:  # True by default so set False for testing
/etc/pulp/conf.d/democonfigmap01_01.py:  "execution_environments": False,
/etc/pulp/conf.d/democonfigmap01_01.py:  "legacy_roles": False,
/etc/pulp/conf.d/democonfigmap01_01.py:  "ai_deny_index": False,
/etc/pulp/conf.d/democonfigmap01_01.py:  "dab_resource_registry": False,
/etc/pulp/conf.d/democonfigmap01_01.py:}

$ kubectl -n galaxy exec -it deployment/galaxy-api -- cat /etc/pulp/settings.py
import os
import traceback

from split_settings.tools import optional, include

GALAXY_API_PATH_PREFIX = "api/galaxy"
CACHE_ENABLED = True
DB_ENCRYPTION_KEY = "/etc/pulp/keys/database_fields.symmetric.key"
GALAXY_COLLECTION_SIGNING_SERVICE = "ansible-default"
GALAXY_CONTAINER_SIGNING_SERVICE = "container-default"
ANSIBLE_CERTS_DIR = "/etc/pulp/keys/"
DATABASES = {"default": {"HOST": "galaxy-postgres-15", "ENGINE": "django.db.backends.postgresql_psycopg2", "NAME": "galaxy", "USER": "galaxy", "PASSWORD": "ZhMCgM3XJfyTLyjhMaf7hrkGjmi2NHWw", "PORT": "5432", "CONN_MAX_AGE": 0, "OPTIONS": {"sslmode": "prefer"}}}
STATIC_ROOT = "/app/galaxy_ng/app/static/"
REDIS_HOST = "galaxy-redis-svc"
REDIS_PORT = 6379
REDIS_PASSWORD = ""
CONTENT_ORIGIN = "http://192.168.0.221:30180"
ANSIBLE_API_HOSTNAME = "http://192.168.0.221:30180"
X_PULP_CONTENT_HOST = "galaxy-content-svc"
X_PULP_CONTENT_PORT = "24816"
TOKEN_SERVER = "http://192.168.0.221:30180/token/"
TOKEN_AUTH_DISABLED = False
TOKEN_SIGNATURE_ALGORITHM = "ES256"
PUBLIC_KEY_PATH = "/etc/pulp/keys/container_auth_public_key.pem"
PRIVATE_KEY_PATH = "/etc/pulp/keys/container_auth_private_key.pem"
CSRF_TRUSTED_ORIGINS = ["http://192.168.0.221:30180", "https://192.168.0.221:30180"]

# Attempt to load settings from /etc/pulp/conf.d/*.py.
settings_dir = os.environ.get('PULP_SETTINGS_DIR', '/etc/pulp/conf.d/')
settings_files = os.path.join(settings_dir, '*.py')
try:
    include(optional(settings_files), scope=locals())
except ImportError:
    traceback.print_exc()
    sys.exit(1)

I can confirm that additional settings in *.py take effects, e.g. browsing existing collections without logging in to the Galaxy by GALAXY_ENABLE_UNAUTHENTICATED_COLLECTION_ACCESS = True.

kurokobo · 2024-05-24T13:34:10Z

TODO:

Documentation

Docs previews:

kurokobo · 2024-05-24T13:37:14Z

@rooftopcellist @Denney-tech
F.Y.I.

kurokobo · 2024-05-24T14:58:31Z

@rooftopcellist @Denney-tech
Marked as ready for review. I always welcome any feedback you may have, thanks!

Denney-tech

Looks good to me, I don't know if I can "approve" but I will try.

Edit: Apparently "Approving" in a review is not the same as approving a workflow lol. TIL

docs/user-guide/advanced-configuration/extra-settings.md

kurokobo · 2024-05-27T12:52:03Z

Updated, ready for review again😃

rooftopcellist · 2024-06-06T01:08:45Z

o/ @kurokobo @Denney-tech Thank you for the PR and for testing it. This is a great addition!

Sorry for the delay here, it has been a busy couple weeks.. I have this on my list to review. CI has been triggered.

Denney-tech · 2024-06-14T20:00:58Z

LGTM

sonarcloud · 2024-07-25T16:18:42Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Denney-tech · 2024-10-02T20:17:03Z

Any chance we can see this get merged soon? Not sure if it's too stale to merge as-is, but I am disappointed it's been sitting here so long.

feat: implement extra_settings_files

857463e

kurokobo added 2 commits May 24, 2024 23:52

docs: update docs for extra_settings_files

e3b2df6

docs: publish docs for db configuration

5a0d68b

kurokobo marked this pull request as ready for review May 24, 2024 14:56

Denney-tech approved these changes May 24, 2024

View reviewed changes

kurokobo commented May 26, 2024

View reviewed changes

docs/user-guide/advanced-configuration/extra-settings.md Outdated Show resolved Hide resolved

docs: remove incorrect restrictions

6bd9d66

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement extra_settings_files #125

feat: implement extra_settings_files #125

kurokobo commented May 24, 2024

kurokobo commented May 24, 2024 •

edited

Loading

kurokobo commented May 24, 2024

kurokobo commented May 24, 2024

Denney-tech left a comment •

edited

Loading

kurokobo commented May 27, 2024

rooftopcellist commented Jun 6, 2024

Denney-tech commented Jun 14, 2024

sonarcloud bot commented Jul 25, 2024

Denney-tech commented Oct 2, 2024

feat: implement extra_settings_files #125

Are you sure you want to change the base?

feat: implement extra_settings_files #125

Conversation

kurokobo commented May 24, 2024

SUMMARY

ADDITIONAL INFORMATION

kurokobo commented May 24, 2024 • edited Loading

kurokobo commented May 24, 2024

kurokobo commented May 24, 2024

Denney-tech left a comment • edited Loading

Choose a reason for hiding this comment

kurokobo commented May 27, 2024

rooftopcellist commented Jun 6, 2024

Denney-tech commented Jun 14, 2024

sonarcloud bot commented Jul 25, 2024

Quality Gate passed

Denney-tech commented Oct 2, 2024

kurokobo commented May 24, 2024 •

edited

Loading

Denney-tech left a comment •

edited

Loading