Refactor Ansible repository to use collection

[dupremathieu]

Abstract

Currently, the Ansible is a monolithic repository containing Ansible's playbooks to perform various action such as the first initialization, maintenance and VM installation. It also contains some CI-specific files.

Each SEAPATH variant (Yocto or Debian) has different git branches.

There is currently no plan to provide users or hardware manufacturer specific Ansible files.

The goal of this changes proposal is to solve those lacks using Ansible collections.

Ansible collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. Collection can be placed in their all repository and be fetched by Ansible-galaxy only if needed. It can be used in SEAPATH to separate all Ansible files by theme (e.g. setup, vm_management, Debian specific) and offer a way for hardware manufacturers and users to add specific features.

Current status

• All Ansible files are inside the Ansible repository (including CI files).
• Ansible Yocto files and Ansible Debian files are stored in different branches.
• To extend Ansible we have to fork the Ansible repository.
• We use git submodules to fetch external dependencies.
• The script prepare.sh is used to setup the environment.

Detailed Description

Ansible collection mechanism is described here: https://docs.ansible.com/ansible/latest/collections_guide/index.html.

Ansible collection brings the following features:

• group playbooks, variables, files, roles and plugins
• documentation generator
• fetch from https://galaxy.ansible.com/ or git repository using ansible-galaxy command
• dependencies between collection
• Execution Environments for system dependencies (libvirt, docker, ceph-ansible specific ansible version)

We propose to create the following collections:

• seapath.setup.debian: Debian setup specific Ansible files
• seapath.setup.centos: CentOS setup specific Ansible files (base on the PR Support for CentOS stream 9 #373)
• seapath.setup.debian_hardening: Debian hardening specific Ansible files
• seapath.setup.common: Common setup Ansible files
• seapath.setup.common_hypervisor: Common hypervisor specific Ansible files
• seapath.setup.cluster_setup: Ansible files to create and configure a SEAPATH cluster
• seapath.vm.management: Ansible files to manage VMs on SEAPATH
• seapath.ci.system_tests: Ansible files to perform system
• seapath.ci.iec61850_tests: IEC 61850 tests

The Ansible repository will only contain examples and documentation.
We also suggest creating an example of ansible-requirements.yaml which make the glue between collections.

With these modifications, the setup flow changes to respect this order:

• distro specific
• distro specific hardening
• machine specific
• user specific
• common part
• hypervisor specific part
• cluster setup common part
• cluster setup user-specific part

Benefit to SEAPATH

• The Ansible repository will become clearer to use
• Improve the modularity
• Easy support to extend SEAPATH outside the SEAPATH repositories
• Provide to users and hardware manufacturers a standard way to extend the Ansible parts of SEAPATH
• Replace the git submodule system which is painful to use
• Separate CI files from Regular files
• Facilitate the integration of multiple SEAPATH variant
• Ansible Execution Environment can be used to manage ceph-ansible specific dependencies (Ansible version)

Works to be done

• The Ansible repository (main and debian-main) branches have to be split into Ansible collection repositories.
• Merge all shareable Ansible files from main and debian-main branches and only variant specific files in separate collections.
• The documentation has to be updated.
• An ansible-requirements.yaml and main setup playbook example has be created.

[BastienDESBOS]

Several topics are aimed to be treated by the proposition (Ansible collection) such as :

• Managing multi distribution in SEAPATH (Yocto, Debian and CentOs today)
• Defining a framework to collaborate with different actors (hardware manufacturers, VIED suppliers…)
• Reorganizing playbooks to improve readability.
  These three topics correspond to specific individual needs that could be addressed as such.

To be able to give a thoughtful opinion on the subject, some questions need to be answered :

• What’s the common part (in seapath.setup.common and seapath.setup.common_hypervisor) ? Examples
• What ratio is common compare to what is not ? Examples
• What would be the reference for the common part ? Yocto or Debian approach ?
• What organization would be established ? What happens in case of evolution on the common/yocto/debian/centos part ? How independent the different distributions would be ? number of CI, review, validation…
• What are the risks and gain of such change ? Examples compare to current state
• What would be the one shot effort to reach this new organization ? What would be the long term effort to maintain it ?
• Is there other example of ansible collections use which could be transposable ?
• What would be the conditions of the success of this major change ?
  To answer to an important part of those questions, I think necessary to code a proof of concept that would illustrate the feasibility / difficulty of this solution.

[dupremathieu]

I will try to answer the questions:

What’s the common part (in seapath.setup.common and seapath.setup.common_hypervisor) ? Examples

The common part is:

• all playbooks inside the playbook directory except playbooks prefixed by ci_ and cluster_setup_prerequisdebian.yaml, cluster_setup_hardened_debian.yaml and cluster_setup_unhardened_debian.yaml.
• seapath_import_vm_disk, seapath_manage_disks, seapath_vm_manage and snmp roles

What ratio is common compare to what is not ? Examples

Actually, in the debian-main branch it have the following distribution of playbooks codes:

• 318 Kio common (64%)
• 156 Kio Debian (32%)
• 20 Kio CI (4%)

What would be the reference for the common part ? Yocto or Debian approach ?

SEAPATH maintainer's would need to ensure that common branch would work for reference design. In case of conflict, they could decide to move some common part into specific playbooks. The adding of a new reference approach would be decided by TSC voting members after demonstrating a tested and maintained solution in a context of digital substation.

What organization would be established ? What happens in case of evolution on the common/yocto/debian/centos part ? How independent the different distributions would be ? number of CI, review, validation…

It's up for discussion.

What are the risks and gain of such change ? Examples compare to current state

The gains are already described in "Benefit to SEAPATH" section.

About the risk, the main change is that it will no longer possible for a SEAPATH variant to implement only a part of SEAPATH features. As a result, if a SEAPATH variant becomes no more compliant with the common part, it will remain locked to an earlier version of SEAPATH.

What would be the one shot effort to reach this new organization ? What would be the long term effort to maintain it ?

The one shot effort is difficult to evaluate. The challenge of the long-term effort is to keep each SEAPATH variant compliant with the changes made in common. The effort required will depend on the modification, bearing in mind that Ansible already provides an abstraction layer. As far as I'm concerned, given SEAPATH's progress, I don't see any big changes coming that would require a great deal of effort on these parts.

Is there other example of ansible collections use which could be transposable ?

You can find all the public Ansible collections here: https://galaxy.ansible.com/ui/collections/

What would be the conditions of the success of this major change ?

I think we'll be able to answer this question once the proof of concept has been completed.