Skip to content
This repository has been archived by the owner on May 27, 2024. It is now read-only.

Commit

Permalink
Merge pull request #659 from deNBI/staging
Browse files Browse the repository at this point in the history
Staging
  • Loading branch information
eKatchko authored Mar 29, 2021
2 parents c906446 + e7a0a16 commit 46a4e47
Show file tree
Hide file tree
Showing 18 changed files with 167 additions and 172 deletions.
3 changes: 3 additions & 0 deletions .clog.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[clog]
repository = "https://github.com/deNBI/cloud-portal-client"
from-latest-tag = true
16 changes: 0 additions & 16 deletions .env

This file was deleted.

21 changes: 10 additions & 11 deletions .env.in
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
CLOUD_CLIENT_TAG=0.1.0-beta.0.13.0
BIBIGRID_TAG=dev-30-9
# Client and Bibigrid
CLOUD_CLIENT_TAG=0.1.0-beta.0.15.1
BIBIGRID_TAG=bibigrid-rest-2.3
CLIENT_PORT=
FORC_API_KEY=
ELASTIC_USER=
ELASTIC_PASSWORD=
FILEBEAT_TAG=7.1.0
HOST=portal-dev.denbi.de
REGION=Giessen-Staging

OS_AUTH_URL=
OS_PROJECT_ID=
OS_PROJECT_NAME=
OS_USERNAME=
OS_USER_DOMAIN_NAME=
OS_PROJECT_DOMAIN_ID=
OS_PROJECT_DOMAIN_NAME=
OS_PASSWORD=

GITHUB_PLAYBOOKS_REPO=https://api.github.com/repos/deNBI/resenvs/contents/
# Filebeat
FILEBEAT_TAG=7.10.0
ELASTIC_USER=
ELASTIC_PASSWORD=
HOST=portal-dev.denbi.de
REGION=Giessen-Staging
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ __pycache__/

# C extensions
*.so
.env
# Distribution / packaging
.Python
build/
Expand Down Expand Up @@ -155,3 +154,5 @@ fabric.properties
# additional data
*.pem
#*.sh
.env_*
VirtualMachineService/config/config_*
3 changes: 0 additions & 3 deletions .secrets.in

This file was deleted.

10 changes: 10 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,17 @@ bibigrid_logs: ## Logs from Bibigrid
enter_client_container: ## Enter Client container
docker exec -it client_portal-client_1 bash

check_env: ## Checks if your .env contains every key set in .env.in.
python3 check_env.py .env.in .env env

check_manual_env: ## Checks if your specified .env_* contains every key set in .env.in.
python3 check_env.py .env.in $(env-file) env

check_local_config: ## Check if your config_local.yml contains every key set in config.yml
python3 check_env.py VirtualMachineService/config/config.yml VirtualMachineService/config/config_local.yml config

check_manual_config: ## Check if your specified config_*.yml contains every key set in config.yml
python3 check_env.py VirtualMachineService/config/config.yml VirtualMachineService/config/$(config-file) config


.PHONY: help lint docs thrift_py
26 changes: 15 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,14 @@ source NameOfRcFile.sh

#### Configuration
You can view (almost) all existing parameters in the [yaml file](VirtualMachineService/config/config.yml).
For local development:
Please copy this file and rename it to `config_local.yml` and fill in missing parameters.
For staging/production setup:
Please copy this file and rename it to `config_YOUR_LOCATION.yml` and fill in missing parameters.
Also you need to provide the path to your config file as the first param when starting a server.

Furthermore there are some parameters you can set in the [.env.in](.env.in) file, which are read only when starting with docker.
Important: You need to rename .env.in to .env in order for it to be read by docker.
Furthermore there are some parameters you must set in the .env file. Copy the [.env.in](.env.in) to .env and
fill in the missing parameters.
When starting with commandline you will need to export some of them manually.

#### Security Groups
Expand Down Expand Up @@ -221,20 +225,21 @@ You can read how to set an ssh key for the cloud-portal repository on [this webs
ansible-galaxy install -r ansible_requirements.yml
~~~

#### 5 Create your own secrets file

Copy the `.secrets.in` to `.secrets`.

#### 6.Set all variables

Set all variables that can be found in `.env` and `.secrets` file.
Set all variables that can be found in `.env` and `VirtualMachineService/config/config.yml` file.
You can have more than one `.env` file (`.env` and `.env_*` are not tracked by git) and specify which you want to copy
by using the `env_file` variable.
You can have more than one `VirtualMachineService/config/config.yml` file (`VirtualMachineService/config/config_*` are
not tracked by git) and specify which you want to copy by using the `client_config` variable.
These options are useful when maintaining multiple client sites.

#### 8.Run the playbook
#### 7.Run the playbook

You can run the playbook using the following command:

~~~BASH
ansible-playbook --tags "client_only" -i inventory_openstack site.yml
ansible-playbook -i inventory_openstack site.yml
~~~

where
Expand All @@ -244,7 +249,7 @@ where
* If you also want to start bibigrid use the tag "bibigrid"
**Choose different files**

You can also specify different .env , .secrets and server.pem files.
You can also specify different .env, config.yml and server.pem files.

You can also specify branch, tag, commit that should be checked out with `--extra-vars`.

Expand All @@ -256,7 +261,6 @@ ansible-playbook -i inventory_openstack --extra-vars "repo_version=master" site.
Optional Keys are:
+ repo_version
+ env_file
+ secrets_file
+ client_server_pem
+ client_config

Expand Down
29 changes: 25 additions & 4 deletions VirtualMachineService/VirtualMachineHandler.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,6 @@
ch.setFormatter(formatter)
LOG.addHandler(fh)
LOG.addHandler(ch)
GITHUB_PLAYBOOKS_REPO = os.environ["GITHUB_PLAYBOOKS_REPO"]
PLAYBOOKS_DIR = "/code/VirtualMachineService/ancon/playbooks/"

PORT = "port"
Expand Down Expand Up @@ -194,15 +193,32 @@ def __init__(self, config):

try:
self.RE_BACKEND_URL = cfg["forc"]["forc_url"]
self.FORC_API_KEY = os.environ["FORC_API_KEY"]
self.FORC_API_KEY = cfg["forc"]["forc_api_key"]
self.FORC_ALLOWED = {}
self.GITHUB_PLAYBOOKS_REPO = cfg["forc"]["github_playbooks_repo"]
if (
not self.RE_BACKEND_URL
or not self.FORC_API_KEY
or not self.GITHUB_PLAYBOOKS_REPO
):
raise ValueError
LOG.info(msg="Forc-Backend url loaded: {0}".format(self.RE_BACKEND_URL))
except ValueError as ve:
LOG.exception(ve)
LOG.info(
"Forc-Backend not loaded as one of the configurations was empty."
)
self.RE_BACKEND_URL = None
self.FORC_API_KEY = None
self.FORC_ALLOWED = None
self.GITHUB_PLAYBOOKS_REPO = None
except Exception as e:
LOG.exception(e)
LOG.info("Forc-Backend not loaded.")
self.RE_BACKEND_URL = None
self.FORC_API_KEY = None
self.FORC_ALLOWED = None
self.GITHUB_PLAYBOOKS_REPO = None
if self.USE_GATEWAY:
self.GATEWAY_IP = cfg["openstack_connection"]["gateway_ip"]
self.SSH_FORMULAR = cfg["openstack_connection"][
Expand Down Expand Up @@ -2665,8 +2681,13 @@ def get_limits(self):
}

def update_playbooks(self):
LOG.info("STARTED update")
r = req.get(GITHUB_PLAYBOOKS_REPO)
if self.GITHUB_PLAYBOOKS_REPO is None:
LOG.info(
"Github playbooks repo url is None. Aborting download of playbooks."
)
return
LOG.info("STARTED update of playbooks")
r = req.get(self.GITHUB_PLAYBOOKS_REPO)
contents = json.loads(r.content)
# Todo maybe clone entire direcotry
for f in contents:
Expand Down
6 changes: 4 additions & 2 deletions VirtualMachineService/config/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ bibigrid:
- slurm

forc:
forc_url: https://proxy-dev.bi.denbi.de:5000/
forc_url:
forc_api_key:
github_playbooks_repo:

cloud_site: giessen
cloud_site: bielefeld
70 changes: 70 additions & 0 deletions check_env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import sys


def get_keys_from_file(file_name):
all_keys = {}
try:
with open(file_name, "r") as stream:
lines = stream.readlines()
for line in lines:
split_line = line.split("=")
if len(split_line) == 2:
if split_line[1] == "\n":
all_keys[split_line[0]] = False
else:
all_keys[split_line[0]] = True
except FileNotFoundError:
print(f"Could not find file {file_name}. Aborting.")
sys.exit(1)
return all_keys


def get_keys_from_config_file(file_name):
all_keys = {}
try:
with open(file_name, "r") as stream:
lines = stream.readlines()
for line in lines:
split_line = line.split(":")
if len(split_line) >= 2:
if split_line[1] == "\n":
all_keys[split_line[0].strip()] = False
else:
all_keys[split_line[0].strip()] = True
except FileNotFoundError:
print(f"Could not find file {file_name}. Aborting.")
sys.exit(1)
return all_keys


if __name__ == "__main__":
if sys.argv[3] == "env":
env_in = get_keys_from_file(sys.argv[1])
env_to_check = get_keys_from_file(sys.argv[2])
for k, v in env_in.items():
if k in env_to_check:
if env_to_check[k]:
continue
else:
print(f"{k} is not set in your env file.")
else:
print(f"{k} is missing as a key in your env file.")
elif sys.argv[3] == "config":
config_in = get_keys_from_config_file(sys.argv[1])
config_to_check = get_keys_from_config_file(sys.argv[2])
for k, v in config_in.items():
if k in config_to_check:
if config_to_check[k]:
continue
else:
if not v:
print(
f"{k} is not set in your config file. Could also be a yml key."
)
else:
print(f"{k} is not set in your config file.")
else:
print(f"{k} is missing as a key in your config file.")
else:
print("No information which files to check. Aborting.")
sys.exit(1)
32 changes: 4 additions & 28 deletions docker-compose.bibigrid.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,16 @@ services:
portal-client:
image: denbicloud/cloud-portal-client:${CLOUD_CLIENT_TAG}
ports:
- ${CLIENT_PORT}:9090

- ${CLIENT_PORT}:9090
env_file:
- .env

environment:
- OS_AUTH_URL
- OS_PROJECT_ID
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_USER_DOMAIN_NAME
- OS_PROJECT_DOMAIN_ID
- FORC_API_KEY

- .env
volumes:
- ${client_PERSISTENT_PATH}/config.yml:/code/VirtualMachineService/config.yml
- ${client_PERSISTENT_PATH}/config.yml:/code/VirtualMachineService/local_config.yml
- ${client_PERSISTENT_PATH}/server.pem:/code/VirtualMachineService/keys/server.pem
- ./logs/client:/code/VirtualMachineService/log:rw
command: python3 VirtualMachineServer.py /code/VirtualMachineService/config.yml
networks:
- portal
- portal

redis:
image: redis:6.0.9
Expand All @@ -38,14 +26,6 @@ services:
env_file:
- .env
environment:
- OS_AUTH_URL
- OS_PROJECT_ID
- OS_PROJECT_NAME
- OS_USERNAME
- OS_PASSWORD
- OS_USER_DOMAIN_NAME
- OS_PROJECT_DOMAIN_ID
- OS_PROJECT_DOMAIN_NAME
- server.enableHttps=false
- server.enableHttp=true
expose:
Expand All @@ -58,10 +38,6 @@ services:
image: docker.elastic.co/beats/filebeat:${FILEBEAT_TAG}
env_file:
- .env
environment:
- ELASTIC_USER
- ELASTIC_PASSWORD
- HOST
volumes:
- ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
- ./logs:/usr/share/filebeat/log:ro
Expand Down
Loading

0 comments on commit 46a4e47

Please sign in to comment.