fix CI builds and rework testing harness (#419)

* lock transitive python dependencies with pip-tools

After yet another breaking package update
(docker/docker-py#3257),
its high time we permanently pin all development packages
*and their dependencies* to a known.-good version.
pip-tools lets us do that without losing the simple requirements.txt
file that allows easy installation, so lets use that.

* rework testing harness for py12, remove pytest-virtualenv dep

a little while ago, we were starting to get "imp" import errors
in CI builds, which I initially were assumed to be from pytest.
In fact, it was the pytest-virtualenv package that we used
to generate a virtualenv with a specific ansible version for tests.

As there is no easy replacement for this package, and the package
itself is stale (no release in 5 years), the testing harness now just
installs the desired version of ansible-core straight into the main
devel virtualenv. This is a bit ugly as it could interfere with other
user tasks, but it leaves the interface the same and helps simplify the test fixture code.
Said code also now reliably isolates tes environments from another and the user-wide collections.

Combined with package pinning introduced in the previous commit,
we should no longer have CI failures due to packages or their deps
updating and breaking things.

.circleci/config.yml

@@ -13,7 +13,7 @@ filters: &semver-tagged
 executors:
   pytest:
     machine:
-      # use an older ubuntu image with python 3.11 until pytest 8.0 is out
+      # python 3.11, needed for ansible-test 2.15
       image: ubuntu-2204:2023.10.1
     resource_class: large
 
@@ -27,37 +27,21 @@ jobs:
         description: Version of Ansible to use for testing
         type: string
       step-version:
-        type: enum
-        enum:
-          - latest
-          - compat
+        type: string
         description: Version of smallstep to test
       node-python-version:
         description: Version of python to use for module tests
         type: string
     executor: pytest
     parallelism: << parameters.parallelism >>
     steps:
-      - when:
-          condition:
-            equal: ["<< parameters.step-version >>", "latest"]
-          steps:
-            - collection-testing/pytest:
-                pytest-args: >
-                  --ansible-version << parameters.ansible-version >>
-                  --step-cli-version "latest"
-                  --step-ca-version "latest"
-                  --node-python-version << parameters.node-python-version >>
-      - when:
-          condition:
-            equal: ["<< parameters.step-version >>", "compat"]
-          steps:
-            - collection-testing/pytest:
-                pytest-args: >
-                  --ansible-version << parameters.ansible-version >>
-                  --step-cli-version "0.24.0"
-                  --step-ca-version "0.24.0"
-                  --node-python-version << parameters.node-python-version >>
+      - collection-testing/pytest:
+          pytest-args: >
+            --ci
+            --ansible-version "<< parameters.ansible-version >>"
+            --step-cli-version "<< parameters.step-version >>"
+            --step-ca-version "<< parameters.step-version >>"
+            --node-python-version "<< parameters.node-python-version >>"
 
 workflows:
   ci:
@@ -68,7 +52,7 @@ workflows:
           matrix:
             parameters:
               ansible-version: ["2.15", "2.16"]
-              step-version: ["latest", "compat"]
+              step-version: ["latest", "0.24.0"]
               node-python-version: ["3.7"]
       - collection-testing/pre-commit-lint:
           name: Lint

.config/molecule/config.yml

@@ -1,6 +1,8 @@
 ---
 dependency:
   name: galaxy
+  options:
+    requirements-file: requirements.molecule.yml
 
 driver:
   name: docker
@@ -18,14 +20,14 @@ provisioner:
 
 scenario:
   test_sequence:
-    - destroy
-    - dependency
     - syntax
+    - dependency
+    - destroy
     - create
     - prepare
     - converge
     - idempotence
-    - check  # also run check mode in regular tests
+    - check # also run check mode in regular tests
     - side_effect
     - verify
     - destroy

.github/renovate.json5

@@ -1,5 +1,14 @@
 {
     "extends": [
         "github>maxhoesel-ansible/.github:renovate-config"
-    ]
+    ],
+    "pip-compile": {
+        "fileMatch": ["^requirements\\.txt$"]
+    },
+    "pip_requirements": {
+        "enabled": false
+    },
+    "pip_setup": {
+        "enabled": false
+    }
 }

CONTRIBUTING.md

@@ -170,6 +170,16 @@ The CI also builds the docs to ensure they don't break silently.
 
 ## Maintainer information
 
+### Updating Dependencies
+
+While the *Ansible* collection itself doesn't have any dependencies outside of ansible itself, the tooling used to build and test the collection does.
+We use [`pip-tools`](https://github.com/jazzband/pip-tools/) to lock these dependencies to a specific version for testing.
+This prevents random CI failures because of [`requests` updates et. al.](https://github.com/docker/docker-py/pull/3257), but still gives us a simple `requirements.txt` that anyone can install.
+
+The direct dependencies are stored in `requirements.in`, use `scripts/udate_requirements.sh` to generate a new `requirements.txt`.
+Do **not** generate `requirements.txt` in another way or remove the header, else renovate [won't be able to resolve and update dependencies in CI!](https://docs.renovatebot.com/modules/manager/pip-compile/#assumption-of-header-with-a-command)
+
+
 ### Raising minimum supported step versions
 
 1. Change the versions in [`plugins/module_utils/constants.py`](./plugins/module_utils/constants.py)

galaxy.yml

@@ -1,27 +1,26 @@
 authors:
-- Max Hösel <[email protected]>
+  - Max Hösel <[email protected]>
 build_ignore:
-- .circleci
-- .github
-- .config
-- .vscode
-- dist
-- CODE_OF_CONDUCT.md
-- CONTRIBUTING.md
-- .yamllint
-- '*.tar.gz'
-- .venv
-- .pytest_cache
-- scripts
-- .pre-commit-config.yaml
-- .readthedocs.yaml
-- pyproject.toml
-- requirements.txt
-- '**/requirements.txt'
-- docs/tmp
-- docs/build
+  - .circleci
+  - .github
+  - .vscode
+  - dist
+  - CODE_OF_CONDUCT.md
+  - CONTRIBUTING.md
+  - .yamllint
+  - "*.tar.gz"
+  - .venv
+  - .pytest_cache
+  - scripts
+  - .pre-commit-config.yaml
+  - .readthedocs.yaml
+  - pyproject.toml
+  - requirements.txt
+  - "**/requirements.txt"
+  - docs/tmp
+  - docs/build
 dependencies:
-  community.general: '>=1.0.0'
+  community.general: ">=1.0.0"
 description: Install, configure and use the Smallstep CA server and CLI tool
 issues: https://github.com/maxhoesel-ansible/ansible-collection-smallstep/issues
 license_file: LICENSE
@@ -30,8 +29,8 @@ namespace: maxhoesel
 readme: README.md
 repository: https://github.com/maxhoesel-ansible/ansible-collection-smallstep
 tags:
-- smallstep
-- ca
-- application
-- certificates
+  - smallstep
+  - ca
+  - application
+  - certificates
 version: 0.24.5

requirements.in

@@ -0,0 +1,37 @@
+# This file is only a template!
+# Use requirements.txt generated by pip-compile to install dependencies
+
+# Requirements for developing this collection
+# Includes utilities, CLI helpers and so on
+
+# Include a version of ansible-core for IDE hints and the default pytest version
+# It is also needed for docs generation.
+#
+# This installed version can be overriden by running pytest with the --ansible-version command.
+# Ideally pytest would install another venv just for the test, but the pytest-virtualenv package
+# is ancient and incompatible with python 3.12.
+ansible-core==2.16.6
+
+# Linting & Formatting
+ansible-lint==24.5.0
+pylint==3.2.1
+autopep8==2.1.0
+pre-commit==3.7.1
+
+# Utility packages used in test fixtures and scripts
+pytest==8.2.0
+pyyaml==6.0.1
+packaging==24.0
+docker==7.1.0
+# Dependencies for executing the role scenarios.
+molecule==6.0.2
+molecule-plugins[docker]==23.4.1
+
+# Generating requirements and syncing venv
+pip-tools==7.4.1
+
+# Docs
+antsibull-docs==2.11.0
+ansible-pygments==0.1.1
+sphinx==7.3.7
+sphinx-ansible-theme==0.10.3