Merge pull request #28 from pescobar/inventory_folder

added inventory/ folder

README.md

@@ -37,45 +37,61 @@ ansible-galaxy role install -r requirements.yml -p ./roles
 
 ### Configuration
 
-The EESSI specific settings can be found in `group_vars/all.yml`, and in `templates` we added our own templates
+The EESSI specific settings can be found in `inventory/group_vars/all.yml`, and in `templates` we added our own templates
 of Squid configurations for the Stratum 1 and local proxy servers.
-For all playbooks you will also need to have an appropriate Ansible `hosts` file;
-see the supplied `hosts.example` for the structure and host groups that you need for these playbooks.
+For all playbooks you will also need to have an appropriate Ansible `hosts` file in the `inventory` folder;
+see the supplied `inventory/hosts.example` for the structure and host groups that you need for these playbooks.
+
+Ansible offers several ways to override any configuration parameters. Of course you can edit a playbook or the `all.yml` file,
+but it is best to keep these files unmodified.
+
+#### Machine-specific configuration
+If the setting is for one specific machine (e.g. your Stratum 1 machine), it is recommended to make a file in the `inventory/host_vars` directory and use the machine name as name of the file.
+This file can contain any settings that should be overridden for this particular machine. See `stratum0host.example` in that directory for an example.
+Any other files that you will create in this directory will be ignored by git.
+
+
+#### Site-specific configuration
+Any other site-specific configuration items can go into a file `inventory/local_site_specific_vars.yml` (which will be ignored by git).
+We provided an example file that shows the kind of configuration that you should minimally provide.
+You can also add more items that you would like to override to this file. See the next section for instructions about passing
+your configuration file to the playbook.
+
 
 ## Running the playbooks
 
 In general, all the playbooks can be run like this:
 ```
-ansible-playbook -i hosts -b <name of playbook>.yml
+ansible-playbook -b -e @inventory/local_site_specific_vars.yml <name of playbook>.yml
 ```
-where `-i` allows you to specify the path to your hosts file, and `-b` means "become", i.e. run with `sudo`.
+Here the option `-e @/path/to/your/config.yml` is used to include your site-specific configuration file.
+The `-b` option means "become", i.e. run with `sudo`.
 If this requires a password, include `-K`, which will ask for the `sudo` password when running the playbook:
 ```
-ansible-playbook -i hosts -b -K <name of playbook>.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml <name of playbook>.yml
 ```
 
-Before you run any of the commands below, make sure that you updated the file `group_vars/all.yml`
-and include the new/extra URLs of any server you want to change/add (e.g. add your Stratum 1).
+Before you run any of the commands below, make sure that you created a `inventory/hosts` file, a site-specific configuration file,
+and, if necessary, created machine-specific configuration files in `inventory/host_vars`.
 
 ### Firewalls
 To make all communication between the CVMFS services possible, some ports have to be opened on the Stratum 0 (default: port 80), 
 Stratum 1 (default: port 80 and 8000), and local proxy (default: port 3128).
-These default port numbers are listed in `roles/cvmfs/defaults/main.yml`, but can be overridden elsewhere.
+These default port numbers are listed in `roles/galaxyproject.cvmfs/defaults/main.yml`, but can be overridden in your local
+configuration file (`local_site_specific_vars.yml`).
 
 The Ansible playbook can update your firewall rules automatically (`firewalld` on Redhat systems, `ufw` on Debian systems), 
 but by default it will not do this. If you want to enable this functionality, set `cvmfs_manage_firewall` to `true`.
-This can be done in either `group_vars/all.yml`, or in a vars section in your hosts or playbook file, or by passing 
-`-e cvmfs_manage_firewall=true` to the `ansible-playbook` command.
 
 ### Stratum 0
 First install the Stratum 0 server:
 ```
-ansible-playbook -i hosts -b -K stratum0.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml stratum0.yml
 ```
 
 Then install the files for the configuration repository:
 ```
-ansible-playbook -i hosts -b -K stratum0-deploy-cvmfs-config.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml stratum0-deploy-cvmfs-config.yml
 ```
 
 Note that there can be only one Stratum 0, so you should only run this playbook
@@ -87,55 +103,47 @@ the (geographically) closest Stratum 1 server for your client and proxies.
 More information on how to (freely) obtain this key is available in the CVMFS documentation: 
 https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup .
 
-You can put your license key in `group_vars/all.yml`, or add a section in your `hosts` file:
-```yaml
-[cvmfsstratum1servers:vars]
-cvmfs_geo_license_key=XXXXX
-```
+You can put your license key in the local configuration file `inventory/local_site_specific_vars.yml`. 
 
 Furthermore, the Stratum 1 runs a Squid server. The template configuration file can be found at 
 `templates/eessi_stratum1_squid.conf.j2`.
 If you want to customize it, for instance for limiting the access to the Stratum 1,
-you can make your own version of this template file and point to it by editing the playbook or
-adding the following to `group_vars/all.yml` or the section in your `hosts` file:
-```yaml
-cvmfs_squid_conf_src=/path/to/your_stratum1_squid.conf.j2
-```
+you can make your own version of this template file and point to it by overriding the following setting in `inventory/local_site_specific_vars.yml`.
+See the comments in the example file for more details.
+
 Install the Stratum 1 using:
 ```
-ansible-playbook -i hosts -b -K stratum1.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml stratum1.yml
 ```
 This will automatically make replicas of all the repositories defined in `group_vars/all.yml`.
 
 ### Local proxies
 The local proxies also need a Squid configuration file; the default can be found in 
 `templates/localproxy_squid.conf.j2`.
+If you want to customize the Squid configuration more, you can also make your own file, and point to in `inventory/local_site_specific_vars.yml`.
+See the comments in the example file for more details.
 
-You have to define the lists of IP addresses / ranges (using CIDR notation) that are allowed to use the proxy using the variable `cvmfs_localproxy_allowed_clients`.
-You can put this, for instance, in your hosts file. See `hosts.example` for more details.
-
-If you want to customize the Squid configuration more, you can also make your own file, and point to it using `cvmfs_squid_conf_src` (see the Stratum 1 section).
+Furthermore, you have to define the lists of IP addresses / ranges (using CIDR notation) that are allowed to use the proxy using the variable `local_cvmfs_http_proxies_allowed_clients`.
+Again, see `inventory/local_site_specific_vars.yml.example` for more details.
 
 Do keep in mind that you should never accept proxy request from everywhere to everywhere!
 Besides having a Squid configuration with the right ACLs, it is recommended to also have a firewall that limits access to your proxy.
 
 Deploy your proxies using:
 ```
-ansible-playbook -i hosts -b -K localproxy.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml localproxy.yml
 ```
 
 ### Clients
 Make sure that your hosts file contains the list of hosts where the CVMFS client should be installed.
-Furthermore, you can add a vars section for the clients that contains the list of (local) proxy servers
-that your clients should use:
-```yaml
-[cvmfsclients:vars]
-cvmfs_http_proxies=["your-local.proxy:3128"]
-```
+Furthermore, you can define a list of (local) proxy servers
+that your clients should use in `inventory/local_site_specific_vars.yml` using the parameter `local_cvmfs_http_proxies`.
+See `inventory/local_site_specific_vars.yml.example` for more details.
 If you just want to roll out one client without a proxy, you can leave this out.
+
 Finally, run the playbook:
 ```
-ansible-playbook -i hosts -b -K client.yml
+ansible-playbook -b -K -e @inventory/local_site_specific_vars.yml client.yml
 ```
 
 ## Verification and usage

ansible.cfg

@@ -0,0 +1,2 @@
+[defaults]
+inventory = ./inventory/hosts

client.yml

@@ -2,7 +2,5 @@
 ---
 - name: CVMFS clients
   hosts: cvmfsclients
-  vars:
-    eessi_cvmfs_repos_enabled: config-repo
   roles:
     - galaxyproject.cvmfs

inventory/.gitignore

@@ -0,0 +1 @@
+local_site_specific_vars.yml

group_vars/all.yml → inventory/group_vars/all.yml

@@ -1,12 +1,6 @@
 ---
 # vars file for eessi project
 
-# The license key for the Geo API:
-# https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup
-# For some unclear reason, the Stratum 1 installation fails when this is not set:
-# https://github.com/EESSI/filesystem-layer/issues/2
-#cvmfs_geo_license_key: 
-
 # Automatically configure EESSI CVMFS repos.
 eessi_cvmfs_repos_enabled: config-repo
 

inventory/group_vars/cvmfsclients

@@ -0,0 +1,8 @@
+---
+
+# List of proxies to be used for the clients.
+# Override this setting in your local_site_specific_vars.yml file.
+cvmfs_http_proxies: "{{ local_cvmfs_http_proxies }}"
+
+# Use the CVMFS configuration repository for the clients.
+eessi_cvmfs_repos_enabled: config-repo

inventory/group_vars/cvmfslocalproxies

@@ -0,0 +1,7 @@
+---
+
+cvmfs_squid_conf_src: "{{ local_proxies_cvmfs_squid_conf_src | default('eessi_localproxy_squid.conf.j2') }}"
+
+# List of clients allowed to access your proxies.
+# Override this setting in local_site_specific_vars.yml.
+cvmfs_localproxy_allowed_clients: "{{ local_cvmfs_http_proxies_allowed_clients }}"

inventory/group_vars/cvmfsstratum0servers

@@ -0,0 +1,3 @@
+---
+
+cvmfs_repositories: "{{ eessi_cvmfs_repositories + [eessi_cvmfs_config_repo.repository] }}"

inventory/group_vars/cvmfsstratum1servers

@@ -0,0 +1,5 @@
+---
+
+cvmfs_repositories: "{{ eessi_cvmfs_repositories + [eessi_cvmfs_config_repo.repository] }}"
+cvmfs_keys: "{{ eessi_cvmfs_keys + [eessi_cvmfs_config_repo.key] }}"
+cvmfs_squid_conf_src: "{{ local_stratum1_cvmfs_squid_conf_src | default('eessi_stratum1_squid.conf.j2') }}"

inventory/host_vars/.gitignore

@@ -0,0 +1,4 @@
+*
+!.gitignore
+!stratum0host.example
+!stratum1host.example

inventory/host_vars/stratum0host.example

@@ -0,0 +1 @@
+cvmfs_srv_device: /dev/sdb

inventory/hosts.example

@@ -0,0 +1,13 @@
+[cvmfsstratum0servers]
+your-stratum0.org
+
+[cvmfsstratum1servers]
+your-stratum1.org
+
+[cvmfslocalproxies]
+your-proxy-1
+your-proxy-2
+
+[cvmfsclients]
+your-client-1
+your-client-2

inventory/local_site_specific_vars.yml.example

@@ -0,0 +1,31 @@
+# The license key for the Geo API:
+# https://cvmfs.readthedocs.io/en/stable/cpt-replica.html#geo-api-setup
+# For some unclear reason, the Stratum 1 installation fails when this is not set:
+# https://github.com/EESSI/filesystem-layer/issues/2
+cvmfs_geo_license_key: INSERT_YOUR_KEY
+
+
+# List of clients allowed to access your local proxies.
+# Add individual IPs and/or use CIDR notation.
+local_cvmfs_http_proxies_allowed_clients:
+  - 192.168.0.0/12
+  - 10.0.0.15
+
+
+# List of all http proxies that should be configured for the clients.
+# Remove or comment the line if you do not want to use a proxy; in this
+# case it will be set to "DIRECT" in the client configuration.
+local_cvmfs_http_proxies:
+  - your-proxy-1:3128
+  - your-proxy-2:3128
+# The following one-liner can be used to automatically add all the hosts
+# defined in the cvmfslocalproxies group in your hosts file
+# to local_cvmfs_http_proxies, using port number 3128.
+# local_cvmfs_http_proxies: "{{ groups.cvmfslocalproxies | map('regex_replace', '^(.*)$', '\\1:3128') | list }}"
+
+
+# Uncomment if you want to use your own Squid configuration template for the local proxies
+# local_proxies_cvmfs_squid_conf_src: "/path/to/your/template"
+
+# Uncomment if you want to use your own Squid configuration template for the Stratum 1
+# local_stratum1_cvmfs_squid_conf_src: : "/path/to/your/template"

localproxy.yml

@@ -2,7 +2,5 @@
 ---
 - name: CVMFS local proxies.
   hosts: cvmfslocalproxies
-  vars:
-    cvmfs_squid_conf_src: eessi_localproxy_squid.conf.j2
   roles:
     - galaxyproject.cvmfs

stratum0.yml

@@ -2,8 +2,6 @@
 ---
 - name: CVMFS Stratum 0
   hosts: cvmfsstratum0servers
-  vars:
-    cvmfs_repositories: "{{ eessi_cvmfs_repositories + [eessi_cvmfs_config_repo.repository] }}"
   roles:
     - role: geerlingguy.repo-epel
       when: ansible_facts['os_family'] == 'RedHat'

stratum1.yml

@@ -2,11 +2,6 @@
 ---
 - name: CVMFS Stratum 1
   hosts: cvmfsstratum1servers
-  vars:
-    #cvmfs_srv_device: /dev/sdb
-    cvmfs_repositories: "{{ eessi_cvmfs_repositories + [eessi_cvmfs_config_repo.repository] }}"
-    cvmfs_keys: "{{ eessi_cvmfs_keys + [eessi_cvmfs_config_repo.key] }}"
-    cvmfs_squid_conf_src: eessi_stratum1_squid.conf.j2
   roles:
     - role: geerlingguy.repo-epel
       when: ansible_facts['os_family'] == 'RedHat'