changed all yml to yaml (as preferred by yaml)

.github/workflows/linting.yml

@@ -15,6 +15,6 @@ jobs:
           pip install -r requirements.txt
           pip install -r requirements-dev.txt
       - name: ansible_lint
-        run: ansible-lint resources/playbook/roles/bibigrid/tasks/main.yml
+        run: ansible-lint resources/playbook/roles/bibigrid/tasks/main.yaml
       - name: pylint_lint
         run: pylint bibigrid

.gitignore

@@ -5,12 +5,12 @@
 # variable resources
 resources/playbook/ansible.cfg
 resources/playbook/roles/bibigrid/templates/slurm/slurm.j2
-resources/playbook/site.yml
+resources/playbook/site.yaml
 resources/playbook/ansible_hosts
 resources/playbook/vars/
 resources/playbook/host_vars/
 resources/playbook/group_vars/
-resources/tests/bibigrid_test.yml
+resources/tests/bibigrid_test.yaml
 
 # Roles
 resources/playbook/roles_galaxy/*

README.md

@@ -1,6 +1,6 @@
 # BiBiGrid
-BiBiGrid is a cloud cluster creation and management framework for OpenStack
-(and more providers in the future).
+BiBiGrid is a framework for creating and managing cloud clusters, currently supporting OpenStack. 
+Future versions will support additional cloud providers.
 
 BiBiGrid uses Ansible to configure standard Ubuntu 20.04/22.04 LTS as 
 well as Debian 11 cloud images. Depending on your configuration BiBiGrid
@@ -26,22 +26,23 @@ might be just what you need.
 <summary> Brief, technical BiBiGrid overview </summary>
 
 ### How to configure a cluster?
-#### Configuration File: bibigrid.yml
-A [template](bibigrid.yml) file is included in the repository ([bibigrid.yml](bibigrid.yml)). 
+#### Configuration File: bibigrid.yaml
+A [template](bibigrid.yaml) file is included in the repository ([bibigrid.yaml](bibigrid.yaml)). 
 
-The cluster configuration file consists of a list of configurations. Every configuration describes the provider specific configuration.
-The first configuration additionally contains all the keys that apply to the entire cluster (roles for example).
-Currently only clusters with one provider are possible, so focus only on the first configuration in the list.
 
-The configuration template [bibigrid.yml](bibigrid.yml) contains many helpful comments, making completing it easier for you.
+The cluster configuration file, `bibigrid.yaml`, consists of a list of configurations. 
+Each configuration describes provider-specific settings. 
+The first configuration in the list also contains keys that apply to the entire cluster (e.g., roles).
+
+The configuration template [bibigrid.yaml](bibigrid.yaml) contains many helpful comments, making completing it easier for you.
 
 [You need more details?](documentation/markdown/features/configuration.md)
 
-#### Cloud Specification Data: clouds.yml
+#### Cloud Specification Data: clouds.yaml
 To access the cloud, authentication information is required.
 You can download your `clouds.yaml` from OpenStack.
 
-Your `clouds.yaml` is to be placed in `~/.config/bibigrid/` and will be loaded by BiBiGrid on execution.
+Place the `clouds.yaml` file in the `~/.config/bibigrid/` directory. BiBiGrid will load this file during execution.
 
 [You need more details?](documentation/markdown/features/cloud_specification_data.md)
 
@@ -50,28 +51,27 @@ If you haven't used BiBiGrid1 in the past or are unfamiliar with OpenStack, we h
 [tutorial](https://github.com/deNBI/bibigrid_clum2022) instead.
 
 #### Preparation
-1. Download (or create) the `clouds.yaml` (and optionally `clouds-public.yaml`) file as described [above](#cloud-specification-data-cloudsyml). 
+1. Download (or create) your `clouds.yaml` file (and optionally `clouds-public.yaml`) as described [above](#cloud-specification-data-cloudsyaml). 
 2. Place the `clouds.yaml` into `~/.config/bibigrid`
-3. Fill the configuration, `bibigrid.yml`, with your specifics. At least you need: A master instance with valid type and image, 
-a region, an availability zone, an sshUser (most likely ubuntu) and a subnet. 
-You probably also want at least one worker with a valid type, image and count.
+3. Fill in the `bibigrid.yaml` configuration file with your specifics. At a minimum you need to specify: a master instance with valid type and image, 
+an sshUser (most likely ubuntu) and a subnet. 
+You will likely also want to specify at least one worker instance with a valid type, image, and count.
 4. If your cloud provider runs post-launch services, you need to set the `waitForServices` 
 key appropriately which expects a list of services to wait for.
 5. Create a virtual environment from `bibigrid/requirements.txt`. 
 See [here](https://www.akamai.com/blog/developers/how-building-virtual-python-environment) for more detailed info. 
 6. Take a look at [First execution](#first-execution)
 
 #### First execution
-Before follow the steps described at [Preparation](#preparation).
+Before proceeding, ensure you have completed the steps described in the [Preparation section](#preparation).
 
-After cloning the repository navigate to `bibigrid`.
-In order to execute BiBiGrid source the virtual environment created during [preparation](#preparation).
-Take a look at BiBiGrid's [Command Line Interface](documentation/markdown/features/CLI.md) 
-if you want to explore for yourself.
+After cloning the repository, navigate to the bibigrid directory. 
+Source the virtual environment created during [preparation](#preparation) to execute BiBiGrid.
+Refer to BiBiGrid's [Command Line Interface documentation](documentation/markdown/features/CLI.md) if you want to explore additional options.
 
 A first execution run through could be:
-1. `./bibigrid.sh -i [path-to-bibigrid.yml] -ch`: checks the configuration
-2. `./bibigrid.sh -i 'bibigrid.yml -i [path-to-bibigrid.yml] -c'`: creates the cluster (execute only if check was successful)
+1. `./bibigrid.sh -i [path-to-bibigrid.yaml] -ch`: checks the configuration
+2. `./bibigrid.sh -i 'bibigrid.yaml -i [path-to-bibigrid.yaml] -c'`: creates the cluster (execute only if check was successful)
 3. Use **BiBiGrid's create output** to investigate the created cluster further. Especially connecting to the ide might be helpful. 
 Otherwise, connect using ssh.
 4. While in ssh try `sinfo` to printing node info
@@ -85,17 +85,17 @@ Great! You've just started and terminated your first cluster using BiBiGrid!
 </details>
 
 ### Troubleshooting
-If your cluster doesn't start up, please first make sure your configurations file is valid (`-ch`). 
-If it is not, try to modify the configurations file to make it valid. Use `-v` or `-vv` to get a more verbose output, 
-so you can find the issue faster. Also double check if you have sufficient permissions to access the project. 
-If you can't make your configurations file valid, please contact a developer.
-If that's the case, please contact a developer and/or manually check if your quotas are exceeded. 
-Some quotas can currently not be checked by bibigrid.
+If your cluster doesn't start up, first ensure your configuration file is valid using the `-ch` option.
+If the configuration is invalid, modify the file as needed.
+Use the `-v` or `-vv` options for more verbose output to help identify the issue faster.
+Also, double-check that you have sufficient permissions to access the project.
+If you cannot make your configuration file valid, please contact a developer.
+Additionally, manually check if your quotas are exceeded, as some quotas cannot currently be checked by BiBiGrid.
 
 **Whenever you contact a developer, please send your logfile along.**
 
 # Documentation
-If you would like to learn more about BiBiGrid please follow a fitting link:
+For more information about BiBiGrid, please visit the following links:
 - [BiBiGrid Features](documentation/markdown/bibigrid_feature_list.md)
 - [Software used by BiBiGrid](documentation/markdown/bibigrid_software_list.md)
 
@@ -118,7 +118,9 @@ Workers are powered down once they are not used for a longer period.
 [https://github.com/BiBiServ/Development-Guidelines](https://github.com/BiBiServ/Development-Guidelines)
 
 ## On implementing concrete providers
-New concrete providers can be implemented very easily. Just copy the `provider.py` file and implement all methods for
-your cloud-provider. Also inherit from the `provider` class. After that add your provider to the providerHandler lists; giving it a associated name for the
-configuration files. By that, your provider is automatically added to BiBiGrid's tests and regular execution. By testing
-your provider first, you will see whether all provider methods are implemented as expected.
+Implementing new cloud providers is straightforward. 
+Copy the `provider.py` file and implement all necessary methods for your cloud provider.
+Inherit from the `provider` class.
+Add your provider to the `providerHandler` lists and assign it an associated name for the configuration files.
+This will automatically include your provider in BiBiGrid's tests and regular execution.
+Test your provider to ensure all methods are implemented correctly.

bibigrid.yml → bibigrid.yaml

@@ -102,6 +102,6 @@
 
   # elastic_scheduling: # for large or slow clusters increasing these timeouts might be necessary to avoid failures
   #   SuspendTimeout: 30 # after SuspendTimeout seconds, slurm allows to power up the node again
-  #   ResumeTimeout: 900 # if a node doesn't start in ResumeTimeout seconds, the start is considered failed.
+  #   ResumeTimeout: 1200 # if a node doesn't start in ResumeTimeout seconds, the start is considered failed.
 
   #- [next configurations]

bibigrid/core/startup_rest.py

@@ -332,7 +332,7 @@ async def ready(cluster_id: str):
 
 
 def test_validate():
-    with open('test.yml', 'rb') as file:
+    with open('test.yaml', 'rb') as file:
         response = client.post("/bibigrid/validate", files={"config_file": file})
     assert response.status_code == 200
     response_data = response.json()
@@ -341,15 +341,15 @@ def test_validate():
 
 
 def test_create():
-    with open('test.yml', 'rb') as file:
+    with open('test.yaml', 'rb') as file:
         response = client.post("/bibigrid/create", files={"config_file": file})
     assert response.status_code == 200
     response_data = response.json()
     assert "cluster_id" in response_data
 
 
 def test_terminate_cluster():
-    with open('test.yml', 'rb') as file:
+    with open('test.yaml', 'rb') as file:
         response = client.post("/bibigrid/terminate", params={"cluster_id": "2uiy5ka2c5y1k8o"},
                                files={"config_file": file})
     assert response.status_code == 200

bibigrid/core/utility/ansible_commands.py

@@ -30,7 +30,7 @@
 #     'else echo"Ansible hosts not reachable. There seems to be a misconfiguration."; fi',"Check for ")
 
 # Run ansible-galaxy to install ansible-galaxy roles from galaxy, git or url (.tar.gz)
-# GALAXY = f"ansible-galaxy install --roles-path {aRP.ADDITIONAL_ROLES_ROOT_PATH_REMOTE} -r {aRP.REQUIREMENTS_YML}"
+# GALAXY = f"ansible-galaxy install --roles-path {aRP.ADDITIONAL_ROLES_ROOT_PATH_REMOTE} -r {aRP.REQUIREMENTS_YAML}"
 
 # Extract ansible roles from files (.tar.gz, .tgz)
 # EXTRACT = f"for f in $(find /tmp/roles -type f -regex '.*\\.t\\(ar\\.\\)?gz'); " \
@@ -49,7 +49,7 @@
                         "Adjust playbook home permission.")
 MV_ANSIBLE_CONFIG = (
     "sudo install -D /opt/playbook/ansible.cfg /etc/ansible/ansible.cfg", "Move ansible configuration.")
-EXECUTE = (f"ansible-playbook {os.path.join(a_rp.PLAYBOOK_PATH_REMOTE, a_rp.SITE_YML)} -i "
+EXECUTE = (f"ansible-playbook {os.path.join(a_rp.PLAYBOOK_PATH_REMOTE, a_rp.SITE_YAML)} -i "
            f"{os.path.join(a_rp.PLAYBOOK_PATH_REMOTE, a_rp.ANSIBLE_HOSTS)} -l {{}}",
            "Execute ansible playbook. Be patient.")
 

bibigrid/core/utility/ansible_configurator.py

@@ -21,7 +21,7 @@
 VPNGTW_ROLES = [{"role": "bibigrid", "tags": ["bibigrid", "bibigrid-vpngtw"]}]
 MASTER_ROLES = [{"role": "bibigrid", "tags": ["bibigrid", "bibigrid-master"]}]
 WORKER_ROLES = [{"role": "bibigrid", "tags": ["bibigrid", "bibigrid-worker"]}]
-VARS_FILES = [aRP.CONFIG_YML, aRP.HOSTS_YML]
+VARS_FILES = [aRP.CONFIG_YAML, aRP.HOSTS_YAML]
 IDE_CONF = {"ide": False, "workspace": ide.DEFAULT_IDE_WORKSPACE, "port_start": ide.REMOTE_BIND_ADDRESS,
             "port_end": ide.DEFAULT_IDE_PORT_END, "build": False}
 ZABBIX_CONF = {"db": "zabbix", "db_user": "zabbix", "db_password": "zabbix", "timezone": "Europe/Berlin",

bibigrid/core/utility/handler/configuration_handler.py

@@ -16,7 +16,7 @@
 CLOUD_CONFIGURATION_KEY = "cloud"
 
 
-def read_configuration(log, path="bibigrid.yml", configuration_list=True):
+def read_configuration(log, path, configuration_list=True):
     """
     Reads yaml from file and returns configuration
     @param log:
@@ -143,7 +143,7 @@ def get_cloud_specifications(configurations, log):
     @return: list of dicts: cloud_specifications of every configuration
     """
     clouds, clouds_public = get_clouds_files(log)
-    log.debug("Loaded clouds.yml and clouds_public.yml")
+    log.debug("Loaded clouds.yaml and clouds_public.yaml")
     cloud_specifications = []
     if isinstance(clouds, dict):
         for configuration in configurations:

bibigrid/core/utility/paths/ansible_resources_path.py

@@ -8,20 +8,20 @@
 
 # UNIVERSAL
 ANSIBLE_HOSTS: str = "ansible_hosts"
-COMMON_YML: str = "common.yml"
-SITE_YML: str = "site.yml"
-REQUIREMENTS_YML: str = "requirements.yml"
+COMMON_YAML: str = "common.yaml"
+SITE_YAML: str = "site.yaml"
+REQUIREMENTS_YAML: str = "requirements.yaml"
 UPLOAD_PATH: str = "/tmp/roles/"
 VARS_PATH: str = "vars/"
 GROUP_VARS_PATH: str = "group_vars/"
 HOST_VARS_PATH: str = "host_vars/"
 ROLES_PATH: str = "roles/"
-LOGIN_YML: str = VARS_PATH + "login.yml"
-CONFIG_YML: str = VARS_PATH + "common_configuration.yml"
-HOSTS_YML: str = VARS_PATH + "hosts.yml"
-WORKER_SPECIFICATION_YML: str = VARS_PATH + "worker_specification.yml"
+LOGIN_YAML: str = VARS_PATH + "login.yaml"
+CONFIG_YAML: str = VARS_PATH + "common_configuration.yaml"
+HOSTS_YAML: str = VARS_PATH + "hosts.yaml"
+WORKER_SPECIFICATION_YAML: str = VARS_PATH + "worker_specification.yaml"
 ADDITIONAL_ROLES_PATH: str = ROLES_PATH + "additional/"
-DEFAULT_IP_FILE = VARS_PATH + "{{ ansible_default_ipv4.address }}.yml"
+DEFAULT_IP_FILE = VARS_PATH + "{{ ansible_default_ipv4.address }}.yaml"
 ANSIBLE_CFG = "ansible.cfg"
 SLURM_CONF = "slurm.conf"
 SLURM_J2 = "slurm.j2"
@@ -30,14 +30,14 @@
 PLAYBOOK = "playbook/"
 PLAYBOOK_PATH: str = os.path.join(b_p.RESOURCES_PATH, PLAYBOOK)
 ANSIBLE_CFG_PATH = os.path.join(PLAYBOOK_PATH, ANSIBLE_CFG)
-HOSTS_FILE = os.path.join(PLAYBOOK_PATH, HOSTS_YML)
+HOSTS_FILE = os.path.join(PLAYBOOK_PATH, HOSTS_YAML)
 HOSTS_CONFIG_FILE: str = os.path.join(PLAYBOOK_PATH, ANSIBLE_HOSTS)
 CONFIG_ROOT_PATH: str = os.path.join(PLAYBOOK_PATH, VARS_PATH)
 ROLES_ROOT_PATH: str = os.path.join(PLAYBOOK_PATH, ROLES_PATH)
-COMMONS_LOGIN_FILE: str = os.path.join(PLAYBOOK_PATH, LOGIN_YML)
-COMMONS_CONFIG_FILE: str = os.path.join(PLAYBOOK_PATH, CONFIG_YML)
-SITE_CONFIG_FILE: str = os.path.join(PLAYBOOK_PATH, SITE_YML)
-WORKER_SPECIFICATION_FILE: str = os.path.join(PLAYBOOK_PATH, WORKER_SPECIFICATION_YML)
+COMMONS_LOGIN_FILE: str = os.path.join(PLAYBOOK_PATH, LOGIN_YAML)
+COMMONS_CONFIG_FILE: str = os.path.join(PLAYBOOK_PATH, CONFIG_YAML)
+SITE_CONFIG_FILE: str = os.path.join(PLAYBOOK_PATH, SITE_YAML)
+WORKER_SPECIFICATION_FILE: str = os.path.join(PLAYBOOK_PATH, WORKER_SPECIFICATION_YAML)
 ADDITIONAL_ROLES_ROOT_PATH: str = ROLES_ROOT_PATH + ADDITIONAL_ROLES_PATH
 VARS_FOLDER = os.path.join(PLAYBOOK_PATH, VARS_PATH)
 GROUP_VARS_FOLDER = os.path.join(PLAYBOOK_PATH, GROUP_VARS_PATH)
@@ -58,9 +58,9 @@
 HOSTS_CONFIG_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + ANSIBLE_HOSTS
 CONFIG_ROOT_PATH_REMOTE: str = PLAYBOOK_PATH_REMOTE + VARS_PATH
 ROLES_ROOT_PATH_REMOTE: str = PLAYBOOK_PATH_REMOTE + ROLES_PATH
-COMMONS_LOGIN_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + LOGIN_YML
-COMMONS_CONFIG_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + CONFIG_YML
-SITE_CONFIG_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + SITE_YML
-WORKER_SPECIFICATION_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + WORKER_SPECIFICATION_YML
+COMMONS_LOGIN_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + LOGIN_YAML
+COMMONS_CONFIG_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + CONFIG_YAML
+SITE_CONFIG_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + SITE_YAML
+WORKER_SPECIFICATION_FILE_REMOTE: str = PLAYBOOK_PATH_REMOTE + WORKER_SPECIFICATION_YAML
 ADDITIONAL_ROLES_ROOT_PATH_REMOTE: str = ROLES_ROOT_PATH + ADDITIONAL_ROLES_PATH
-REQUIREMENTS_CONFIG_FILE_REMOTE: str = ADDITIONAL_ROLES_ROOT_PATH_REMOTE + REQUIREMENTS_YML
+REQUIREMENTS_CONFIG_FILE_REMOTE: str = ADDITIONAL_ROLES_ROOT_PATH_REMOTE + REQUIREMENTS_YAML

bibigrid/core/utility/validate_schema.py

@@ -1,5 +1,5 @@
 """
-Handles the schema validation for BiBiGrid's configuration yml.
+Handles the schema validation for BiBiGrid's configuration yaml.
 """
 
 from schema import Schema, Optional, Or, SchemaError

documentation/markdown/features/CLI.md

@@ -5,7 +5,7 @@ Available command line parameters:
 - `-h, --help`            show help message and exit
 - `-v, --verbose`         Increases output verbosity (can be of great use when cluster fails to start). `-v` adds more
   detailed info to the logfile, `-vv` adds debug information to the logfile.
-- `-d, --debug`           Keeps cluster active in case of an error. Offers termination after successful create.
+- `-d, --debug`           Keeps cluster active in case of an error. Offers termination after successful create. Prints full stack trace on errors.
 - `-i <path>, --config_input <path> (required)` Path to YAML configurations file. Relative paths can be used and start
   at `~/.config/bibigrid`
 - `-cid <cluster-id>, --cluster_id <cluster-id>` Cluster id is needed for ide and termination. If no cluster id is set,

documentation/markdown/features/check.md

@@ -1,5 +1,7 @@
 # Check
 
+Besides the in depth check which is explained below there is a schema check in [validate_schema.py](../../../bibigrid/core/utility/validate_schema.py).
+
 ## Exactly one master or vpn instance per configuration
 
 There can only be a single master or a single vpn-gateway (vpngtw) per configuration.
@@ -49,7 +51,7 @@ are whitelisted. This check will only print a warning, but will not fail the che
 If this check doesn't succeed, downloading the `clouds.yaml` again might be the fastest solution.
 You can read more about these files here [Cloud-specification](cloud_specification_data.md)
 
-### Valid
+### Clouds.yaml validity
 
 A `clouds.yaml` entry is considered valid if - combined with any `clouds-public.yaml` entries it refers to using the
 profile

documentation/markdown/features/cluster_commands.md

@@ -8,7 +8,7 @@ Similar to `sinfo` but shows detailed information regarding node features.
 Thereby, it helps you with understanding any worker startup issues.
 
 ## [bibiplay](../../../resources/bin/bibiplay)
-`bibiplay` is mainly a shortcut for `ansible-playbook /opt/playbook/site.yml -i /opt/playbook/ansible_hosts`
+`bibiplay` is mainly a shortcut for `ansible-playbook /opt/playbook/site.yaml -i /opt/playbook/ansible_hosts`
 which allows you to execute the ansible playbook more easily.
 
 ### Examples

documentation/markdown/features/configuration.md

@@ -3,10 +3,10 @@
 > **Note**
 > 
 > First take a look at our [Hands-On BiBiGrid Tutorial](https://github.com/deNBI/bibigrid_clum2022) and our 
-> example [bibigrid.yml](../../../bibigrid.yml).
+> example [bibigrid.yaml](../../../bibigrid.yaml).
 > This documentation is no replacement for those, but provides greater detail - too much detail for first time users.
 
-The configuration file (often called `bibigrid.yml`) contains important information about cluster creation.
+The configuration file (often called `bibigrid.yaml`) contains important information about cluster creation.
 The cluster configuration holds a list of configurations where each configuration has a specific 
 cloud (location) and infrastructure (e.g. OpenStack). For single-cloud use cases you just need a single configuration.
 However, you can use additional configurations to set up a multi-cloud.
@@ -167,7 +167,7 @@ If some nodes need to be active during the entire cluster lifetime, [onDemand](#
 ###### Defaults
 ```yaml
 slurmConf:
-    db: slurm # see task 042-slurm-server.yml
+    db: slurm # see task 042-slurm-server.yaml
     db_user: slurm
     db_password: changeme
     munge_key: # automatically generated via id_generation.generate_munge_key