Skip to content

Commit

Permalink
Release 2.3.1 (#322)
Browse files Browse the repository at this point in the history 
* fix : print possible configuration errors to stdout to be more specific that "an error occurs"

* fix #222: print possible configuration errors to stdout

* enhancement #281: add support for cloud site region (Openstack)

* Fix/#275 - Integrated 'useHostname' config parameter (#280)

* Adjusted ideConf Readme

* Extended configuration by useHostname parameter, adjusted AnsibleHostsConfig writing if useHostname is enabled

* Added useHostname config parameter to config schema, adjusted default parameters marked with ""

* Fixed 'useHostname' instead of 'useHostnames' typo

* Added description to 'useHostnames' parameter

* enhancement #283: add option to overwrite service CIDR mask

* enhancement #283: adjust changelog

* enhancement #283: adjust documentation

* Moved serviceCIDR validation from config to validator, added method description for validateServiceCIDR(string), Added string pattern description to CONFIGURATION_SCHEMA.md

* bump version to 2.2.2 (main bugfix release)
- readd missing 'break' in intentMode switch/case directive

* fix/#217 :
- adjust theia task
- remove unused (unlinked) cloud9 task

* #285: move CIDR mask validation into Configuration setter method

* security/#291: change default ip range for opened ports from 0.0.0.0/0 to current

* enhancement/#289 :
- bump node version manager to 0.37.0
- bump prebuild theia-version to 1.8

* fix/#297: disable auto-upgrade as early as possible (#298)

* fix/#297: disable auto-upgrade as early as possible

* #297 add check for /var/lib/dpkg/lock

* Move client2 module/#299 (#300)

* Adjusted ideConf Readme

* Moved client to providerModule

* update ChangeLog.md

* bump version from 2.2.2 to 2.3

* Determine quotas/#257 (#306)

* Adjusted ideConf Readme

* Moved Intents for Openstack to separate directory

* Changed interface OSClient to correct implementation via OSClientV3

* ProviderTypes have not been tested and remained unused, check config independently only in -ch and -c case

* Moved config validation to validator

* Removed unnecessary duplicate block in checking instance and image resources, added documentation to ValidateIntent

* Renamed ValidatorOpenstack method validate in validateCredentials to be more precise

* Added OpenStack Implementation of ValidateIntent

* Check quotas exceeded before launching worker instances in scaling up,
Changed scale up method createWorkerInstances into createAdditionalWorkerInstances,
Assign clusterId in runIntent() in StartUp in forehand,
Removed help intentMode from runIntent(), since it is handled before,
Added ValidateIntentOpenstack,
Added todos

* Created configureClusterInstances() including master and worker instance configuration to check quotas in forehand,
changed list of map entries for instanceTypes to map,
removed unnecessary 'prepare' value permanently,
Fixed some issues in google cloud, aws and azure and other warnings

* Fixed ValidateIntent checkQuotas() wrong parameters issue

* Fixed issue with master and worker same providerType,
enhanced logging

* adjust ChangeLog.md
bump jackson databind to 2.9.10.7

* adjust ChangeLog.md

* Bump guava from 28.0-jre to 29.0-jre in /bibigrid-openstack

Bumps [guava](https://github.com/google/guava) from 28.0-jre to 29.0-jre.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](https://github.com/google/guava/commits)

Signed-off-by: dependabot[bot] <[email protected]>

* Bump snakeyaml from 1.25 to 1.26 in /bibigrid-core

Bumps [snakeyaml](https://bitbucket.org/asomov/snakeyaml) from 1.25 to 1.26.
- [Commits](https://bitbucket.org/asomov/snakeyaml/branches/compare/snakeyaml-1.26..snakeyaml-1.25)

---
updated-dependencies:
- dependency-name: org.yaml:snakeyaml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Fix ide config check/#293 (#313)

* Adjusted ideConf Readme

* Remove ConfigurationFile.java which has been fully replaced by Configuration

* Moved CommandLine interpretation to separate class.

* Moved config validation into validate and create only,
minor fixes

* Removed deprecated slave instances configuration

* Added missing loadConfiguration() method doc

* Added opportunity to install IDE subsequently

* Added cluster initialization in CreateCluster

* Fixed rest api createCluster components

* Minor enhancements

* Moved loaded cluster to IdeIntent instead of loading in intent

* Moved Yaml Interpretation into separate model, minor bug fixes

* Enhanced YamlInterpreter logging

* Moved isIpAdressFile() to YamlInterpreter

* Added LoadIdeConfiguration Method in LoadClusterConfigurationIntent,
minor code enhancements

* Fix rebase merging errors

* error handling when ide not installed, separate installSubsequently() method to install afterwards (not yet implemented)

* Abort IDE start when configuration not loaded successfully

* fix(BibigridInfoIdGetHandler):just added a space to avoid confusion

* Fixes missing clusterMap loading in BibigridTerminateIdDeleteHandler (#316)

* Added ansible ping check in shell script, Removed unnecessary warn logging (#319)

* add workaround for buggy openstack4j identity api v3 handling (#321)

* bump version to 2.3.1

Co-authored-by: tdilger <[email protected]>
Co-authored-by: tdilger <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dweinholz <[email protected]>
  • Loading branch information
@jkrue @tdilger
@dependabot @dweinholz
5 people authored Jan 5, 2022
1 parent 393ba11 commit 9c459ab
Show file tree
Hide file tree
Showing 35 changed files with 1,116 additions and 520 deletions.
2 changes: 1 addition & 1 deletion bibigrid-aws/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>bibigrid</artifactId>
<groupId>de.unibi.cebitec.bibigrid</groupId>
<version>2.3</version>
<version>2.3.1</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
4 changes: 2 additions & 2 deletions bibigrid-core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<parent>
<artifactId>bibigrid</artifactId>
<groupId>de.unibi.cebitec.bibigrid</groupId>
<version>2.3</version>
<version>2.3.1</version>
</parent>

<artifactId>bibigrid-core</artifactId>
Expand Down Expand Up @@ -49,7 +49,7 @@
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.25</version>
<version>1.26</version>
</dependency>

<dependency>
Expand Down
87 changes: 41 additions & 46 deletions bibigrid-core/src/main/java/de/unibi/cebitec/bibigrid/core/intents/CreateCluster.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,7 @@ public abstract class CreateCluster extends Intent {

protected final ProviderModule providerModule;
protected final Configuration config;
protected final String clusterId;
protected Instance masterInstance;
protected List<Instance> workerInstances;
protected Cluster cluster;
protected CreateClusterEnvironment environment;

protected DeviceMapper masterDeviceMapper;
Expand All @@ -54,12 +52,13 @@ public abstract class CreateCluster extends Intent {
* @param clusterId optional if cluster already started and has to be scaled
*/
protected CreateCluster(ProviderModule providerModule, Configuration config, String clusterId) {
this.clusterId = clusterId != null ? clusterId : generateClusterId();
String cid = clusterId != null ? clusterId : generateClusterId();
cluster = new Cluster(cid);
this.providerModule = providerModule;
this.config = config;
this.interruptionMessageHook = new Thread(() ->
LOG.error("Cluster setup was interrupted!\n\n" +
"Please clean up the remains using: -t {}\n\n", this.clusterId));
"Please clean up the remains using: -t {}\n\n", cluster.getClusterId()));
}

/**
Expand All @@ -78,10 +77,6 @@ static String generateClusterId() {
return clusterIdBase64.substring(0, len).toLowerCase(Locale.US);
}

public String getClusterId() {
return clusterId;
}

/**
* The environment creation procedure. For a successful environment creation
* you will need an Environment-Instance which implements the
Expand Down Expand Up @@ -115,15 +110,13 @@ public CreateClusterEnvironment createClusterEnvironment() throws ConfigurationE
public boolean configureClusterInstances() {
Map<InstanceType, Integer> instanceTypes = new HashMap<>();
instanceTypes.put(config.getMasterInstance().getProviderType(), 1);
LOG.warn("Put master instance type {} into quotas map.", config.getMasterInstance().getProviderType());
for (Configuration.WorkerInstanceConfiguration worker : config.getWorkerInstances()) {
InstanceType providerType = worker.getProviderType();
if (instanceTypes.containsKey(providerType)) {
instanceTypes.put(worker.getProviderType(), instanceTypes.get(providerType) + worker.getCount());
} else {
instanceTypes.put(worker.getProviderType(), worker.getCount());
}
LOG.warn("Put worker instance type {} into quotas map.", worker.getProviderType());
}
if (providerModule.getValidateIntent(config).
checkQuotasExceeded(instanceTypes)) {
Expand Down Expand Up @@ -163,12 +156,12 @@ public void configureClusterWorkerInstance() {
*/
public boolean launchClusterInstances() {
try {
String masterNameTag = MASTER_NAME_PREFIX + SEPARATOR + clusterId;
masterInstance = launchClusterMasterInstance(masterNameTag);
String masterNameTag = MASTER_NAME_PREFIX + SEPARATOR + cluster.getClusterId();
Instance masterInstance = launchClusterMasterInstance(masterNameTag);
if (masterInstance == null) {
return false;
}
workerInstances = new ArrayList<>();
List<Instance> workerInstances = new ArrayList<>();
int totalWorkerInstanceCount = config.getWorkerInstanceCount();
if (totalWorkerInstanceCount > 0) {
LOG.info("Requesting {} worker instance(s) with {} different configurations...",
Expand All @@ -178,7 +171,7 @@ public boolean launchClusterInstances() {
Configuration.WorkerInstanceConfiguration instanceConfiguration = config.getWorkerInstances().get(i);
LOG.info("Requesting {} worker instance(s) with same configuration...",
instanceConfiguration.getCount());
String workerNameTag = WORKER_NAME_PREFIX + SEPARATOR + clusterId;
String workerNameTag = WORKER_NAME_PREFIX + SEPARATOR + cluster.getClusterId();
List<Instance> workersBatch = launchClusterWorkerInstances(i, instanceConfiguration, workerNameTag);
if (workersBatch == null) {
return false;
Expand All @@ -188,9 +181,14 @@ public boolean launchClusterInstances() {
} else {
LOG.info("No Worker instance(s) requested!");
}
// set cluster initialization values
cluster.setMasterInstance(masterInstance);
cluster.setWorkerInstances(workerInstances);
cluster.setPublicIp(masterInstance.getPublicIp());
cluster.setPrivateIp(masterInstance.getPrivateIp());
// just to be sure, everything is present, wait x seconds
sleep(4);
LOG.info("Cluster (ID: {}) successfully created!", clusterId);
LOG.info("Cluster (ID: {}) successfully created!", cluster.getClusterId());
final String masterIp = config.isUseMasterWithPublicIp() ? masterInstance.getPublicIp() :
masterInstance.getPrivateIp();
configureAnsible();
Expand All @@ -205,7 +203,8 @@ public boolean launchClusterInstances() {
}
if (Configuration.DEBUG) {
logFinishedInfoMessage(
config.isUseMasterWithPublicIp() ? masterInstance.getPublicIp() : masterInstance.getPrivateIp());
config.isUseMasterWithPublicIp() ?
cluster.getMasterInstance().getPublicIp() : cluster.getMasterInstance().getPrivateIp());
}

Runtime.getRuntime().removeShutdownHook(this.interruptionMessageHook);
Expand All @@ -216,17 +215,13 @@ public boolean launchClusterInstances() {
}

/**
* TODO Should not rely on config, since configurationFile might not be valid for current cluster, keypair + ssh user?
* Adds additional worker instance(s) with specified batch to cluster.
* Adopts the configuration from the other worker instances in batch
* @return true, if worker instance(s) created successfully
*/
public boolean createAdditionalWorkerInstances(int batchIndex, int count) {
LoadClusterConfigurationIntent loadIntent = providerModule.getLoadClusterConfigurationIntent(config);
loadIntent.loadClusterConfiguration(clusterId);
Cluster cluster = loadIntent.getCluster(clusterId);
public boolean createWorkerInstances(int batchIndex, int count) {
if (cluster == null) {
LOG.error("No cluster with specified clusterId {} found", clusterId);
LOG.error("No cluster with specified clusterId {} found", cluster.getClusterId());
return false;
}
try {
Expand Down Expand Up @@ -291,18 +286,20 @@ public boolean createAdditionalWorkerInstances(int batchIndex, int count) {
}
LOG.info("Creating {} worker " + (count == 1 ? "instance" : "instances") + " for batch {}.", count, batchIndex);
instanceConfiguration.setCount(count);
String workerNameTag = WORKER_NAME_PREFIX + "-" + clusterId;
String workerNameTag = WORKER_NAME_PREFIX + "-" + cluster.getClusterId();
int workerIndex = workersBatch.size() + 1;
List<Instance> additionalWorkers =
launchAdditionalClusterWorkerInstances(cluster, batchIndex, workerIndex, instanceConfiguration, workerNameTag);
launchAdditionalClusterWorkerInstances(batchIndex, workerIndex, instanceConfiguration, workerNameTag);
if (additionalWorkers == null) {
LOG.error("No additional workers could be launched.");
channelSftp.disconnect();
sshSession.disconnect();
return false;
} else {
workerInstances = cluster.getWorkerInstances();
workerInstances.addAll(additionalWorkers);
// loadIntent cluster ...
// workerInstances = cluster.getWorkerInstances();
// workerInstances.addAll(additionalWorkers);
cluster.addWorkerInstances(additionalWorkers);
}
config.getClusterKeyPair().setName(CreateCluster.PREFIX + cluster.getClusterId());
config.getClusterKeyPair().load();
Expand Down Expand Up @@ -352,10 +349,11 @@ public boolean createAdditionalWorkerInstances(int batchIndex, int count) {
* @return List of worker Instances
*/
protected abstract List<Instance> launchAdditionalClusterWorkerInstances(
Cluster cluster, int batchIndex, int workerIndex, Configuration.WorkerInstanceConfiguration instanceConfiguration, String workerNameTag);
int batchIndex, int workerIndex,
Configuration.WorkerInstanceConfiguration instanceConfiguration, String workerNameTag);

protected String buildWorkerInstanceName(int batchIndex, int workerIndex) {
return WORKER_NAME_PREFIX + SEPARATOR + (batchIndex) + SEPARATOR + (workerIndex) + "-" + clusterId;
return WORKER_NAME_PREFIX + SEPARATOR + (batchIndex) + SEPARATOR + (workerIndex) + "-" + cluster.getClusterId();
}

private void logFinishedInfoMessage(final String masterPublicIp) {
Expand All @@ -376,9 +374,9 @@ private void logFinishedInfoMessageWindows(final String masterPublicIp) {
.append("putty -i ")
.append(Configuration.KEYS_DIR).append("\\").append(config.getClusterKeyPair().getName())
.append(" ").append(config.getSshUser()).append("@%BIBIGRID_MASTER%\n\n");
sb.append("The cluster id of your started cluster is: ").append(clusterId).append("\n\n");
sb.append("The cluster id of your started cluster is: ").append(cluster.getClusterId()).append("\n\n");
sb.append("You can easily terminate the cluster at any time with:\n")
.append("./bibigrid -t ").append(clusterId).append(" ");
.append("./bibigrid -t ").append(cluster.getClusterId()).append(" ");
if (config.isAlternativeConfigFile()) {
sb.append("-o ").append(config.getAlternativeConfigPath()).append(" ");
}
Expand All @@ -394,9 +392,9 @@ private void logFinishedInfoMessageUnix(final String masterPublicIp) {
.append("ssh -i ")
.append(Configuration.KEYS_DIR).append("/").append(config.getClusterKeyPair().getName())
.append(" ").append(config.getSshUser()).append("@$BIBIGRID_MASTER\n\n");
sb.append("The cluster id of your started cluster is: ").append(clusterId).append("\n\n");
sb.append("The cluster id of your started cluster is: ").append(cluster.getClusterId()).append("\n\n");
sb.append("You can easily terminate the cluster at any time with:\n")
.append("./bibigrid -t ").append(clusterId).append(" ");
.append("./bibigrid -t ").append(cluster.getClusterId()).append(" ");
if (config.isAlternativeConfigFile()) {
sb.append("-o ").append(config.getAlternativeConfigPath()).append(" ");
}
Expand All @@ -410,7 +408,7 @@ private void saveGridPropertiesFile(String masterPublicIp) {
gp.setProperty("BIBIGRID_MASTER", masterPublicIp);
gp.setProperty("SSHPublicKeyFile", config.getSshPublicKeyFile());
gp.setProperty("SSHPrivateKeyFile", config.getSshPrivateKeyFile());
gp.setProperty("clusterId", clusterId);
gp.setProperty("clusterId", cluster.getClusterId());
if (config.isAlternativeConfigFile()) {
gp.setProperty("AlternativeConfigFile", config.getAlternativeConfigPath());
}
Expand All @@ -430,8 +428,8 @@ private void saveGridPropertiesFile(String masterPublicIp) {
* ConfigurationException, close the sshSession and throw a new ConfigurationException
*/
private void configureAnsible() throws ConfigurationException {
final String masterIp = config.isUseMasterWithPublicIp() ? masterInstance.getPublicIp() :
masterInstance.getPrivateIp();
final String masterIp = config.isUseMasterWithPublicIp() ? cluster.getPublicIp() :
cluster.getPrivateIp();
LOG.info("Now configuring...");
boolean sshPortIsReady = SshFactory.pollSshPortIsAvailable(masterIp);
if (sshPortIsReady) {
Expand Down Expand Up @@ -565,7 +563,7 @@ private void uploadAnsibleToMaster(Session sshSession) throws JSchException, Con
AnsibleConfig.writeSiteFile(channel.put(site_file),
customMasterRoles, customWorkerRoles);

AnsibleConfig.writeHostsFile(channel, config.getSshUser(), workerInstances, config.useHostnames());
AnsibleConfig.writeHostsFile(channel, config.getSshUser(), cluster.getWorkerInstances(), config.useHostnames());

// files for common configuration
String login_file = channel_dir + AnsibleResources.COMMONS_LOGIN_FILE;
Expand All @@ -581,7 +579,7 @@ private void uploadAnsibleToMaster(Session sshSession) throws JSchException, Con

// write files using stream
AnsibleConfig.writeLoginFile(login_stream, config);
AnsibleConfig.writeInstancesFile(instances_stream, masterInstance, workerInstances, masterDeviceMapper, providerModule.getBlockDeviceBase());
AnsibleConfig.writeInstancesFile(instances_stream, cluster.getMasterInstance(), cluster.getWorkerInstances(), masterDeviceMapper, providerModule.getBlockDeviceBase());
AnsibleConfig.writeConfigFile(config_stream, config, environment.getSubnet().getCidr());
// TODO network should be written in instance configuration when initializing
// security group und server group
Expand All @@ -594,7 +592,7 @@ private void uploadAnsibleToMaster(Session sshSession) throws JSchException, Con
}

// Write worker instance specific configuration file
for (Instance worker : workerInstances) {
for (Instance worker : cluster.getWorkerInstances()) {
String filename = channel_dir + AnsibleResources.CONFIG_ROOT_PATH + "/"
+ worker.getPrivateIp() + ".yml";
AnsibleConfig.writeSpecificInstanceFile(channel.put(filename), worker, providerModule.getBlockDeviceBase());
Expand Down Expand Up @@ -691,19 +689,16 @@ private String getSingleFileName(String roleFile) {
return pathway[pathway.length - 1];
}



public Configuration getConfig() {
return config;
}

public Instance getMasterInstance() {
return masterInstance;
public Cluster getCluster() {
return cluster;
}

public List<Instance> getWorkerInstances() {
return workerInstances;
public void setCluster(Cluster cluster) {
this.cluster = cluster;
}

}

2 changes: 1 addition & 1 deletion ...d-core/src/main/java/de/unibi/cebitec/bibigrid/core/intents/CreateClusterEnvironment.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ private void generateClusterKeyPair() throws ConfigurationException {
bao = new ByteArrayOutputStream();
keypair.writePublicKey(bao, "generated by BiBiGrid");
clusterKeyPair.setPublicKey(bao.toString());
clusterKeyPair.setName(CreateCluster.PREFIX+cluster.clusterId);
clusterKeyPair.setName(CreateCluster.PREFIX+cluster.cluster.getClusterId());
clusterKeyPair.store();
} catch (Exception ex) {
if (Configuration.DEBUG) {
Expand Down
Loading

0 comments on commit 9c459ab

Please sign in to comment.