Merge Dev into Master (#347)

* fix : print possible configuration errors to stdout to be more specific that "an error occurs" * fix #222: print possible configuration errors to stdout * enhancement #281: add support for cloud site region (Openstack) * Fix/#275 - Integrated 'useHostname' config parameter (#280) * Adjusted ideConf Readme * Extended configuration by useHostname parameter, adjusted AnsibleHostsConfig writing if useHostname is enabled * Added useHostname config parameter to config schema, adjusted default parameters marked with "" * Fixed 'useHostname' instead of 'useHostnames' typo * Added description to 'useHostnames' parameter * enhancement #283: add option to overwrite service CIDR mask * enhancement #283: adjust changelog * enhancement #283: adjust documentation * Moved serviceCIDR validation from config to validator, added method description for validateServiceCIDR(string), Added string pattern description to CONFIGURATION_SCHEMA.md * bump version to 2.2.2 (main bugfix release) - readd missing 'break' in intentMode switch/case directive * fix/#217 : - adjust theia task - remove unused (unlinked) cloud9 task * #285: move CIDR mask validation into Configuration setter method * security/#291: change default ip range for opened ports from 0.0.0.0/0 to current * enhancement/#289 : - bump node version manager to 0.37.0 - bump prebuild theia-version to 1.8 * fix/#297: disable auto-upgrade as early as possible (#298) * fix/#297: disable auto-upgrade as early as possible * #297 add check for /var/lib/dpkg/lock * Move client2 module/#299 (#300) * Adjusted ideConf Readme * Moved client to providerModule * update ChangeLog.md * bump version from 2.2.2 to 2.3 * Determine quotas/#257 (#306) * Adjusted ideConf Readme * Moved Intents for Openstack to separate directory * Changed interface OSClient to correct implementation via OSClientV3 * ProviderTypes have not been tested and remained unused, check config independently only in -ch and -c case * Moved config validation to validator * Removed unnecessary duplicate block in checking instance and image resources, added documentation to ValidateIntent * Renamed ValidatorOpenstack method validate in validateCredentials to be more precise * Added OpenStack Implementation of ValidateIntent * Check quotas exceeded before launching worker instances in scaling up, Changed scale up method createWorkerInstances into createAdditionalWorkerInstances, Assign clusterId in runIntent() in StartUp in forehand, Removed help intentMode from runIntent(), since it is handled before, Added ValidateIntentOpenstack, Added todos * Created configureClusterInstances() including master and worker instance configuration to check quotas in forehand, changed list of map entries for instanceTypes to map, removed unnecessary 'prepare' value permanently, Fixed some issues in google cloud, aws and azure and other warnings * Fixed ValidateIntent checkQuotas() wrong parameters issue * Fixed issue with master and worker same providerType, enhanced logging * adjust ChangeLog.md bump jackson databind to 2.9.10.7 * adjust ChangeLog.md * Bump guava from 28.0-jre to 29.0-jre in /bibigrid-openstack Bumps [guava](https://github.com/google/guava) from 28.0-jre to 29.0-jre. - [Release notes](https://github.com/google/guava/releases) - [Commits](https://github.com/google/guava/commits) Signed-off-by: dependabot[bot] <[email protected]> * Bump snakeyaml from 1.25 to 1.26 in /bibigrid-core Bumps [snakeyaml](https://bitbucket.org/asomov/snakeyaml) from 1.25 to 1.26. - [Commits](https://bitbucket.org/asomov/snakeyaml/branches/compare/snakeyaml-1.26..snakeyaml-1.25) --- updated-dependencies: - dependency-name: org.yaml:snakeyaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Fix ide config check/#293 (#313) * Adjusted ideConf Readme * Remove ConfigurationFile.java which has been fully replaced by Configuration * Moved CommandLine interpretation to separate class. * Moved config validation into validate and create only, minor fixes * Removed deprecated slave instances configuration * Added missing loadConfiguration() method doc * Added opportunity to install IDE subsequently * Added cluster initialization in CreateCluster * Fixed rest api createCluster components * Minor enhancements * Moved loaded cluster to IdeIntent instead of loading in intent * Moved Yaml Interpretation into separate model, minor bug fixes * Enhanced YamlInterpreter logging * Moved isIpAdressFile() to YamlInterpreter * Added LoadIdeConfiguration Method in LoadClusterConfigurationIntent, minor code enhancements * Fix rebase merging errors * error handling when ide not installed, separate installSubsequently() method to install afterwards (not yet implemented) * Abort IDE start when configuration not loaded successfully * fix(BibigridInfoIdGetHandler):just added a space to avoid confusion * Fixes missing clusterMap loading in BibigridTerminateIdDeleteHandler (#316) * Added ansible ping check in shell script, Removed unnecessary warn logging (#319) * add workaround for buggy openstack4j identity api v3 handling (#321) * bump version to 2.3.1 * adjust slurm task (slurm 20.11, configuration) * fixes security vulnerabilities (#323, #324, #326) - bump jackson-databind to version 2.9.10.8 - bump undertow to version 2.1.6.Final * adjust configuration (#327) * adjust common and master configuration for slurm (#327) * return on empty line to dismiss empty warnings, Replace 'XXXX' with '****' in credentials output (#320) * -move all slurm task in an extra role (slurm, #327) - add ansible.cfg to speed up configuration (#332) - add support for slurmdbd (#331) and slurmrestd * remove ganglia task from common,master and worker role (#249) * remove ganglia, oge and other deprecated parts from documentation (#249) * make AnsibleConfig aware of removed OGE/Ganglia options. * cleanup * add ipRange values to example config * Add accounting and priority settings/#335 (#338) * Added AccountSettings and PrioritySettings and scheduler config in var/ * Transferred AccountingSettings and PrioritySettings directly into slurm.conf * fix(Ansible): wait till bielefeld_environment service is done if it exists (#345) * fix race condition * fix(cluster-scaling): added cluster initialization when adding worker instances (#346) * Probably the last release of the Java-Based Version of BiBiGrid * Update README.md Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: tdilger <[email protected]> Co-authored-by: tdilger <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dweinholz <[email protected]> Co-authored-by: Christian Henke <[email protected]> Co-authored-by: cnexcale <[email protected]>
BiBiServ · Dec 13, 2022 · b1c61af · b1c61af
1 parent 9c459ab
commit b1c61af
Show file tree

Hide file tree

Showing 35 changed files with 414 additions and 626 deletions.
diff --git a/ChangeLog.md b/ChangeLog.md
@@ -1,3 +1,14 @@
+## Version 2.3.1 (3/8/2021)
+
+This will probably the latest version of BiBigrid depended on Java.   We are currently working on a complete 
+reimplementation: BiBiGrid2.
+
+## Fixes
+- minor ansible configuration cleanup
+
+## Features
+- (#327)
+
 ## Version 2.3 (3/2/2021)
 
 ## Fixes

diff --git a/README.md b/README.md
@@ -1,8 +1,14 @@
 # BiBiGrid
+
+> **Note**
+> The Java-based version of BiBiGrid was discontinued. The latest release is version 2.3.1.
+> Since this version isn't maintained any longer you should consider using its successor.
+
+
 BiBiGrid is a tool for an easy cluster setup inside a cloud environment.
 It is written in Java and run on any OS a Java runtime is provided - any 
 Java 8 is supported. BiBiGrid and its Cmdline UI based on a general cloud 
-provider api. Currently the implementation is based on OpenStack ([Openstack4j](https://github.com/openstack4j/openstack4j)).  
+provider api. Currently, the implementation is based on OpenStack ([Openstack4j](https://github.com/openstack4j/openstack4j)).  
 There also exists implementations for Google (Compute Engine, using the official Google Cloud SDK), 
 Amazon (AWS EC2 using the official AWS SDK) and Microsoft (Azure using the official Azure SDK) (WIP)
 which are currently not provided tested.
@@ -16,7 +22,7 @@ a shared filesystem (on local discs and attached volumes), a cloud IDE for writi
 ([Theia Web IDE](https://github.com/theia-ide/theia)) and many more.
 
 During resource instantiation BiBiGrid configures the network, local and network volumes, (network) file systems and 
-also the software for an immediately usage of the started cluster. 
+also the software for an immediate usage of the started cluster. 
 
 When using preinstalled images a full configured and ready to use cluster is available within a few minutes.
 

diff --git a/bibigrid-aws/pom.xml b/bibigrid-aws/pom.xml
@@ -16,7 +16,7 @@
         <dependency>
             <groupId>de.unibi.cebitec.bibigrid</groupId>
             <artifactId>bibigrid-core</artifactId>
-            <version>2.3</version>
+            <version>2.3.1</version>
         </dependency>
         <dependency>
             <groupId>com.amazonaws</groupId>

diff --git a/bibigrid-azure/pom.xml b/bibigrid-azure/pom.xml
@@ -6,7 +6,7 @@
         <artifactId>bibigrid</artifactId>
         <groupId>de.unibi.cebitec.bibigrid</groupId>
 
-        <version>2.3</version>
+        <version>2.3.1</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 
@@ -17,7 +17,7 @@
         <dependency>
             <groupId>de.unibi.cebitec.bibigrid</groupId>
             <artifactId>bibigrid-core</artifactId>
-            <version>2.3</version>
+            <version>2.3.1</version>
         </dependency>
         <dependency>
             <groupId>com.microsoft.azure</groupId>

diff --git a/bibigrid-core/pom.xml b/bibigrid-core/pom.xml
@@ -67,7 +67,7 @@
         <dependency>
             <groupId>com.fasterxml.jackson.core</groupId>
             <artifactId>jackson-databind</artifactId>
-            <version>2.9.10.7</version>
+            <version>2.9.10.8</version>
             <scope>compile</scope>
         </dependency>
     </dependencies>

diff --git a/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/intents/CreateCluster.java b/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/intents/CreateCluster.java
@@ -232,6 +232,13 @@ public boolean createWorkerInstances(int batchIndex, int count) {
             io.printStackTrace();
             return false;
         }
+
+        if (cluster.getMasterInstance() == null) {
+            LoadClusterConfigurationIntent loadIntent = providerModule.getLoadClusterConfigurationIntent(config);
+            loadIntent.loadClusterConfiguration(cluster.getClusterId());
+            cluster = loadIntent.getCluster(cluster.getClusterId());
+        }
+
         Session sshSession = null;
         boolean success = true;
         try {

diff --git a/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/model/AnsibleConfig.java b/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/model/AnsibleConfig.java
@@ -58,6 +58,19 @@ public static void writeHostsFile(ChannelSftp channel, String sshUser, List<Inst
         }
     }
 
+    /**
+     * Helper methods that create a role data structure
+     * @param name role name
+     * @param tags a list of tags bound to role
+     * @return role hashmap
+     */
+    private static HashMap createRole(String name, List tags) {
+        HashMap role = new LinkedHashMap<String,Object>();
+        role.put("role",name);
+        role.put("tags",tags);
+        return role;
+    }
+
     /**
      * Generates site.yml automatically including custom ansible roles.
      *
@@ -68,6 +81,7 @@ public static void writeHostsFile(ChannelSftp channel, String sshUser, List<Inst
     public static void writeSiteFile(OutputStream stream,
                               Map<String, String> customMasterRoles,
                               Map<String, String> customWorkerRoles) {
+
         List<String> common_vars =
                 Arrays.asList(AnsibleResources.LOGIN_YML, AnsibleResources.INSTANCES_YML, AnsibleResources.CONFIG_YML);
         String DEFAULT_IP_FILE = AnsibleResources.VARS_PATH + "{{ ansible_default_ipv4.address }}.yml";
@@ -83,9 +97,10 @@ public static void writeSiteFile(OutputStream stream,
             }
         }
         master.put("vars_files", vars_files);
-        List<String> roles = new ArrayList<>();
+        List roles = new ArrayList<>();
         roles.add("common");
         roles.add("master");
+        roles.add(createRole("slurm",Arrays.asList("slurm","scale-up","scale-down")));
         for (String role_name : customMasterRoles.keySet()) {
             roles.add("additional/" + role_name);
         }
@@ -105,6 +120,7 @@ public static void writeSiteFile(OutputStream stream,
         roles = new ArrayList<>();
         roles.add("common");
         roles.add("worker");
+        roles.add(createRole("slurm",Arrays.asList("slurm","scale-up","scale-down")));
         for (String role_name : customWorkerRoles.keySet()) {
             roles.add("additional/" + role_name);
         }
@@ -357,7 +373,6 @@ public static void writeConfigFile(OutputStream stream, Configuration config, St
         addBooleanOption(map, "enable_gridengine", config.isOge());
         addBooleanOption(map, "enable_slurm",config.isSlurm());
         addBooleanOption(map, "use_master_as_compute", config.isUseMasterAsCompute());
-        addBooleanOption(map, "enable_ganglia",config.isGanglia());
         addBooleanOption(map, "enable_zabbix", config.isZabbix());
         addBooleanOption(map, "enable_ide", config.isIDE());
         if (config.isNfs()) {
@@ -367,12 +382,12 @@ public static void writeConfigFile(OutputStream stream, Configuration config, St
         if (config.isIDE()) {
             map.put("ideConf", getIdeConfMap(config.getIdeConf()));
         }
+        if (config.isSlurm()) {
+            map.put("slurmConf",getSlurmConfMap(config.getSlurmConf()));
+        }
         if (config.isZabbix()) {
             map.put("zabbix", getZabbixConfMap(config.getZabbixConf()));
         }
-        if (config.isOge()) {
-            map.put("oge", getOgeConfMap(config.getOgeConf()));
-        }
         if (config.hasCustomAnsibleRoles()) {
             map.put("ansible_roles", getAnsibleRoles(config.getAnsibleRoles()));
         }
@@ -473,6 +488,14 @@ private static Map<String, Object> getZabbixConfMap(Configuration.ZabbixConf zc)
         return zabbixConf;
     }
 
+    private static Map<String, Object> getSlurmConfMap(Configuration.SlurmConf sc) {
+        Map<String, Object> slurmConf = new LinkedHashMap<>();
+        slurmConf.put("db",sc.getDatabase());
+        slurmConf.put("db_user",sc.getDb_user());
+        slurmConf.put("db_password",sc.getDb_password());
+        return slurmConf;
+    }
+
     private static Map<String, String> getOgeConfMap(Properties oc) {
         Map<String, String> ogeConf = new HashMap<>();
         for (final String name : oc.stringPropertyNames()) {

diff --git a/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/model/Configuration.java b/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/model/Configuration.java
@@ -116,21 +116,19 @@ public static Configuration loadConfiguration(Class<? extends Configuration> con
     private List<WorkerInstanceConfiguration> workerInstances = new ArrayList<>();
     private boolean oge;
     private boolean slurm;
+    private SlurmConf slurmConf = new SlurmConf();
     private boolean localDNSLookup;
     private String mungeKey;
     private boolean nfs = true;
     private String serviceCIDR;
     private IdeConf ideConf = new IdeConf();
-    private boolean ganglia;
     private boolean zabbix;
     private ZabbixConf zabbixConf = new ZabbixConf();
     private List<String> nfsShares = new ArrayList<>(Collections.singletonList("/vol/spool"));
     private List<MountPoint> masterMounts = new ArrayList<>();
     private List<MountPoint> extNfsShares = new ArrayList<>();
     private FS localFS = FS.XFS;
     private boolean debugRequests;
-    @Deprecated
-    private Properties ogeConf = OgeConf.initOgeConfProperties();
     private List<AnsibleRoles> ansibleRoles = new ArrayList<>();
     private List<AnsibleGalaxyRoles> ansibleGalaxyRoles = new ArrayList<>();
     private boolean useHostnames = false;
@@ -515,6 +513,14 @@ public void setSlurm(boolean slurm) {
         this.slurm = slurm;
     }
 
+    public SlurmConf getSlurmConf() {
+        return slurmConf;
+    }
+
+    public void setSlurmConf(SlurmConf slurmConf) {
+        this.slurmConf = slurmConf;
+    }
+
     public String getMungeKey() {
         if (mungeKey == null) {
             // create a unique hash
@@ -539,18 +545,6 @@ public void setMungeKey(String mungeKey) {
         this.mungeKey = mungeKey;
     }
 
-    public boolean isGanglia() {
-        return ganglia;
-    }
-
-    public void setGanglia(boolean ganglia) {
-        this.ganglia = ganglia;
-        if (ganglia) {
-            LOG.warn("Ganglia (oge) support is deprecated (only supported using Ubuntu 16.04.) " +
-                     "and will be removed in the near future. Please use Zabbix instead.");
-        }
-    }
-
     public boolean isZabbix() {
         return zabbix;
     }
@@ -771,20 +765,6 @@ private static String bytesToHex(byte[] hash) {
         return hexString.toString();
     }
 
-    public Properties getOgeConf() {
-        return ogeConf;
-    }
-
-    /**
-     * Saves given values to ogeConf Properties.
-     * @param ogeConf Properties
-     */
-    public void setOgeConf(Properties ogeConf) {
-        for (String key : ogeConf.stringPropertyNames()) {
-            this.ogeConf.setProperty(key, ogeConf.getProperty(key));
-        }
-    }
-
     /**
      * Provides support for GridEngine global configuration.
      */
@@ -811,29 +791,6 @@ private static OgeConf initOgeConfProperties() {
         }
     }
 
-    @Deprecated
-    public boolean isCloud9() {
-        return ideConf.isIde();
-    }
-
-    @Deprecated
-    public void setCloud9(boolean cloud9) {
-        LOG.warn("cloud9 parameter is deprecated. Please use IdeConf instead.");
-        LOG.warn("Cloud9 will not longer be supported and is replaced by the Theia Web IDE.");
-        ideConf.setIde(cloud9);
-    }
-
-    @Deprecated
-    public boolean isTheia() {
-        return ideConf.isIde();
-    }
-
-    @Deprecated
-    public void setTheia(boolean theia) {
-        LOG.warn("theia parameter is deprecated. Please use IdeConf instead.");
-        ideConf.setIde(theia);
-    }
-
     public boolean isIDE() {
         if (ideConf == null) {
             return false;
@@ -907,6 +864,45 @@ public void setBuild(boolean build) {
         }
     }
 
+    /**
+     * Configuration of Slurm.
+     * Currently, all values are hard-coded.
+     */
+    public static class SlurmConf {
+        private boolean slurm = true;
+        private String database = "slurm";
+        private String db_user = "slurm";
+        private String db_password = "changeme";
+
+        public boolean isSlurm() {
+            return slurm;
+        }
+
+        public String getDatabase() {
+            return database;
+        }
+
+        public void setDatabase(String database) {
+            this.database = database;
+        }
+
+        public String getDb_user() {
+            return db_user;
+        }
+
+        public void setDb_user(String db_user) {
+            this.db_user = db_user;
+        }
+
+        public String getDb_password() {
+            return db_password;
+        }
+
+        public void setDb_password(String db_password) {
+            this.db_password = db_password;
+        }
+    }
+
     /**
      * Checks if custom ansible roles used.
      *

diff --git a/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/util/SshFactory.java b/bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/util/SshFactory.java
@@ -198,6 +198,9 @@ class LineReaderRunnable implements Runnable {
     }
 
     private void work_on_line(String line) {
+        if (line.isEmpty()) {
+            return;
+        }
         if (regular) {
             if (line.contains("CONFIGURATION FINISHED")) {
                 returnCode = 0;

diff --git a/bibigrid-core/src/main/resources/playbook/ansible.cfg b/bibigrid-core/src/main/resources/playbook/ansible.cfg
@@ -0,0 +1,8 @@
+[defaults]
+inventory = ./ansible_hosts
+host_key_checking = False
+forks=50
+pipelining = True
+
+[ssh_connection]
+ssh_args = -o ControlMaster=auto -o ControlPersist=60s
diff --git a/bibigrid-core/src/main/resources/playbook/roles/common/handlers/main.yml b/bibigrid-core/src/main/resources/playbook/roles/common/handlers/main.yml
@@ -0,0 +1 @@
+
diff --git a/bibigrid-core/src/main/resources/playbook/roles/common/tasks/001-apt.yml b/bibigrid-core/src/main/resources/playbook/roles/common/tasks/001-apt.yml
@@ -10,6 +10,16 @@
     group: root
     mode: 0644
 
+- name: Populate service facts
+  ansible.builtin.service_facts:
+
+- name: Wait till Apt_Mirror de.NBI Bielefeld Service is done
+  ansible.builtin.service_facts:
+  until: services['de.NBI_Bielefeld_environment.service'].state == 'stopped'
+  retries: 35
+  delay: 10
+  when: services['de.NBI_Bielefeld_environment.service'] is defined
+
 - name: Update
   apt:
     update_cache: "yes"
@@ -100,11 +110,14 @@
     deb: 'https://repo.zabbix.com/zabbix/5.0/{{ ansible_distribution | lower }}/pool/main/z/zabbix-release/zabbix-release_5.0-1+{{ ansible_distribution_release }}_all.deb'
     state: present
 
-- name: Add ondrej/php repository
-  apt_repository:
-    repo: ppa:ondrej/php
+- name: Add apt.bi.denbi.de repository key
+  apt_key:
+    url: 'https://apt.bi.denbi.de/repo_key.key'
     state: present
-  when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'xenial'"
+
+- name: Add apt.bi.denbi.de repository
+  apt_repository:
+    repo: 'deb https://apt.bi.denbi.de/repos/apt/{{ ansible_distribution_release | lower }} {{ ansible_distribution_release | lower }} main'
 
 - name: Update apt cache
   apt: