Openshift

Introduction

Openshift is used to host NATCom in dev, test, and production. The license plates for these namespaces are cec7ed-dev, cec7ed-test, and cec7ed-prod

Here's the link to the dev namespace

https://console.apps.silver.devops.gov.bc.ca/k8s/cluster/projects/c1c7ed-dev

The dev namespace contains sandboxed instances of NATCom for each PR in GitHub. The test environment contains at most two instances (one for test releases, and one for a hotfix). The prod environment contains at most one instance.

Helm is used for OpenShift deployments.

Helm

• Login to openshift via oc. Ensure you have oc on your path (can be downloaded here https://console.apps.silver.devops.gov.bc.ca/command-line-tools)
• Using the link above, click the "Copy login command" link to get the login token
• In a terminal window, paste the login command
• List the helm projects to find the one you want to uninstall: helm list --all
• helm uninstall {project name} to uninstall the project
• You'll need to manually delete the pvcs in OpenShift, or using the following OC commands
• Note: you also need to install helm (https://helm.sh/docs/intro/install/)

To get a list of pvc names: oc get pvc -n {namespace}

To delete a pvc by name: oc delete pvc {pvc name} -n {namespace}

Note {namespace} is c1c7ed-dev, c1c7ed-test, or c1c7ed-prod (don't delete the prod pvc).

Note, each PR will typically have two PCS:

• data-nr-compliance-enforcement-{PR}-bitnami-pg-0 (this stores the database)
• nr-compliance-enforcement-{PR}-nats-js (this persists any nats/jetstream messages).

You'll want to delete both when uninstalling your PR manually.

If you want to redeploy, you can rerun your github action: https://github.com/bcgov/nr-compliance-enforcement/actions/workflows/pr-open.yml

Install Openshift cli & Helm

For macOS and Windows you can use Homebrew or Chocolatey to install both helm and openshift commands

macOS

Install homebrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install openshift-cli

brew install helm

windows

Install Chocolatey:

https://chocolatey.org/install

choco install openshift-cli

choco install kubernetes-helm

Connecting to database in openshift

1. Ensure you have oc on your path (can be downloaded here https://console.apps.silver.devops.gov.bc.ca/command-line-tools)
2. Using the link above, click the "Copy login command" link to get the login token
3. In a terminal window, paste the login command
4. Select the project that namespace (e.g. oc project c1c7ed-test)
5. Find the database pod by running the command oc get pods
6. Port forward to the database pod oc port-forward {podname} 5433:5432 (note 5433 is arbitrary, this is the port that you're going to connect to locally)
7. Use your database explorer of choice (I like DBeaver) to connect to the database. You can connect to the database using the host localhost, and the port 5433 (or whatever port you indicated on step 6 above).
8. The password for the database can be found in the OpenShift Secret nr-compliance-enforcement-database (or in dev nr-compliance-enforcement-{PR #}-database

Openshift Quotas

Since each PR can create an instance of NATCom in dev, we may run into quota issues where we don't have enough resources to run each sandboxed instance of NATCom.

Troubleshooting

If a deployment is failing, it's probably because the long-running quota has been exceeded. This is mostly a problem in the development environment which scales based on the number of open PR.

How to determine if there is a quota problem?

1. Log into the OpenShift console as an administrator.
2. Under the Administration section select ResourceQuotas.
3. From the list select compute-long-running-quota.

4. If anything is in the Yellow or the Red some attention is needed.

Strategies for freeing up resources

Look for unwanted running pods

1. Log into the OpenShift console as an administrator.
2. Under the Workloads section select Pods.
3. Filter the list by Status to put the running pods to the top of the list.
4. Make sure that all the pods that are running are desirable. For example, if there are any Pods relating to a closed pull request, this indicates that something didn't get cleaned up properly.

Temporarily remove pods for a Pull Request (Development only)

So you looked at the Pods, and you can't find any that don't belong. Unfortunately you are going to have to make a decision about who gets deployed and who doesn't. Examine the pull requests and determine which ones are the priority to be deployed. If there is one that is still a work in progress, it might be a good candidate to get removed from the dev environment.

1. Log into the OpenShift console as an administrator.
2. Under the Workloads section select DeploymentConfigs.
3. Type the number of the Pull Request you want to remove in the search box
4. For each item in the list under the ... menu select DeleteDeploymentConfig

This will delete all of the application Pods until either a new commit is issued for the Pull Request, or the deployment actions are re-run.

Removing the NATS Pods

The NATS container is special as it is deployed through Helm.

1. Log into the OpenShift console as a Developer
2. Select Helm
3. Type the number of the Pull Request you want to remove in the search box
4. From the ... menu select Delete Helm Release

To redeploy the NATS container, go into the pull Re-run the Deploy NATS to OpenShift action in the same fashion as described above

Requesting an increase of quota/limits

Requests for quota updates can be made here: https://registry.developer.gov.bc.ca/registry/user/dashboard/requests

More information is here: https://docs.developer.gov.bc.ca/request-quota-increase-for-openshift-project-set/