Hosting

Teaching Vacancies is hosted on Azure Cloud Infrastructure Platform (CIP), with the services running as an Azure Kubernetes Service.

Environments

Environment	URL	Code branch	CI/CD workflow	AKS Cluster	AKS Namespace
Production	https://teaching-vacancies.service.gov.uk	`main`	build_and_deploy.yml	s189-teacher-services-cloud-production	tv-production
Staging	https://staging.teaching-vacancies.service.gov.uk	`main`	build_and_deploy.yml	s189-teacher-services-cloud-test	tv-staging
QA	https://qa.teaching-vacancies.service.gov.uk	`main`	build_and_deploy.yml	s189-teacher-services-cloud-test	tv-development
Review	`https://teaching-vacancies-review-pr-xxxx.test.teacherservices.cloud`	multiple	build_and_deploy.yml	s189-teacher-services-cloud-test	tv-development

Ephemeral review apps that are created when a PR is created on GitHub, and destroyed when the PR is merged. These have URLs which contain the Pull Request number, like https://teaching-vacancies-review-pr-6441.test.teacherservices.cloud

The QA and Staging environments have integration with DSI.

The Staging environment is a pre-production environment, to identify issues with code before it's promoted to Production. On merging a Pull Request, the same code is deployed first to Staging, and after a successful smoke test, to Production.

QA environment contains the same code branch as Staging and Production environments, and is used by multiple members of the team for general testing about a production-like environment.

Deployments

Detailed information about our deployment process in this link.

Azure Kubernetes Service (AKS) organisation and permission model

Teaching Vacancies members have access to three different Azure Subscriptions. Each subscription gives access to a Kubernetes cluster with the same name.

Each of the Kubernetes clusters is divided into namespaces.

Each namespace hosts one/multiple different environments for Teaching Vacancies service.

Cluster	Name	DfE Platform Identity Group	Access
Development	s189-teacher-services-cloud-development	s189 TV delivery team	Permanently granted
Test	s189-teacher-services-cloud-test	s189 AKS admin test PIM	Permanently granted
Production	s189-teacher-services-cloud-production	s189 TV production PIM	Up to 8 hours. Needs team member approval

Teaching Vacancies application environments are hosted in the following clusters/namespaces:

App environment	Kubernetes Cluster	Kubernetes Namespace
Review	s189-teacher-services-cloud-test	tv-development
QA	s189-teacher-services-cloud-test	tv-development
Staging	s189-teacher-services-cloud-test	tv-staging
Production	s189-teacher-services-cloud-production	tv-production

During onboarding you will have been granted access to selected Azure resources and roles.

By default, you will have access to TV Kubernetes namespaces/envs/apps hosted in the s189-teacher-services-cloud-test cluster.

For the tv-production namespace containing the Production environment running the Teaching Vacancies live app, you will have to request the temporal activation (up to 8 hours per request) for the Member role on the (s189 TV production PIM).

Senior developers/tech lead on the team and DevOps have the will be able to approve each others requests.

Users aren´t able to self-approve their own role requests.

Getting Production access on Azure Platform

Request your access for the s189 TV production PIM role on the Azure Privileged Identity Management (PIM) request under Groups tab in the Azure Portal.
The request will need to be approved.

Installing the Azure Client and Kubectl

Install the Azure Client
Install kubectl
```
az aks install-cli
```

Login from Azure Console

You will need production access first if you want to run commands on the production environment.
Log into the tenant using the Azure Cli. This will launch your browser for the login.
```
az login --tenant 9c7d9dd3-840c-4b3f-818e-552865082e16
```
[OPTIONAL] From Teaching Vacancies root directory, get the credentials for the environment.
- Only needed if you're going to execute kubectl commands directly. make env command shortcuts in our makefile will execute this step automatically.
- For test cluster environments (review, qa, staging):
```
make qa get-cluster-credentials
```
- For production environment:
```
make production get-cluster-credentials CONFIRM_PRODUCTION=YES
```

Makefile shortcuts for kubernetes commands

We have added a series of Makefile definitions to speed up common Rails developer commands over any of our environments:

If the environment has multiple pods running the web/application. The command will be executed over the first listed pod.

Opening a Rails Console

make review pr_id=5432 railsc

make qa/staging railsc

make production railsc CONFIRM_PRODUCTION=YES

Opening a shell

make review pr_id=5432 shell

make qa/staging shell

make production shell CONFIRM_PRODUCTION=YES

Running a rake task

make review pr_id=5432 rake task=audit:email_addresses

make qa/staging rake task=audit:email_addresses

make production rake task=audit:email_addresses CONFIRM_PRODUCTION=YES

Tailing application logs

make review pr_id=5432 logs

make qa/staging logs

make production logs CONFIRM_PRODUCTION=YES

Kubernetes commands

Executing commands with kubectl tool.

Listing the deployments (apps) in the cluster:

kubectl -n tv-development get deployments

Listing the application pods (running instances) in the cluster:

kubectl -n tv-development get pods

Opening a console in a Review App

To open a console for an app deployment (on its first pod):

kubectl -n tv-development exec -ti deployment/teaching-vacancies-review-pr-xxxx -- /bin/sh

To open a console in the particular pod:

kubectl -n tv-development exec -ti teaching-vacancies-review-pr-xxxx-podid -- /bin/sh

Executing commands in a Review App

kubectl -n tv-development exec deployment/teaching-vacancies-review-pr-xxxx -- ps aux

Temporally scaling up our deployment instances

Sometimes we may need to temporally scale up our service resources. That can be done with the kubectl scale command.

Important: These changes are ephemereal and will be overriden to the defaults on the next deployment.

If you need to scale the changes for a period of time and to keep the new number of instances between deployments, you will need to set it up in the terraform configuration and release it through a PR/deployment.

EG: Executing a task that will enqueue thousands of jobs in our workers, this could cause a bottleneck in our Sidekiq queues.

To see the original number of instances:

kubectl -n tv-production get deployments
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
teaching-vacancies-production          8/8     8            8           318d
teaching-vacancies-production-worker   4/4     4            4           314d

To temporally increase our worker instances:

kubectl scale --replicas=16 deployment teaching-vacancies-production-worker -n tv-production

See the new number of instances:

kubectl -n tv-production get deployments
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
teaching-vacancies-production          8/8     8            8           318d
teaching-vacancies-production-worker   16/16   16           16          314d

Scale it back when not needed anymore:

kubectl scale --replicas=4 deployment teaching-vacancies-production-worker -n tv-production

Set up a new environment

Create file terraform/workspace-variables/<env>.tfvars.json
Create file terraform/workspace-variables/<env>_app_env.yml
Create SSM parameters of type SecureString:
- /teaching-vacancies/<env>/app/BIG_QUERY_API_JSON_KEY
- /teaching-vacancies/<env>/app/GOOGLE_API_JSON_KEY
- /teaching-vacancies/<env>/app/secrets
- /teaching-vacancies/<env>/infra/secrets
Add a target to the Makefile

.PHONY: <env>
<env>: ## <env>
		$(eval env=<env>)
		$(eval var_file=<env>)

Run:

make passcode=MyPasscode tag=47fd1475376bbfa16a773693133569b794408995 <env> terraform-app-apply

If you want to have a deployment triggered by a push to a branch, add a trigger to build_and_deploy.yml

on:
  push:
    branches:
      - dev
      - <env>

Optionally, refresh the database with a sanitised copy of the production data

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hosting.md

hosting.md

Hosting

Environments

Deployments

Azure Kubernetes Service (AKS) organisation and permission model

Getting Production access on Azure Platform

Installing the Azure Client and Kubectl

Login from Azure Console

Makefile shortcuts for kubernetes commands

Opening a Rails Console

Opening a shell

Running a rake task

Tailing application logs

Kubernetes commands

Listing the deployments (apps) in the cluster:

Listing the application pods (running instances) in the cluster:

Opening a console in a Review App

Executing commands in a Review App

Temporally scaling up our deployment instances

EG: Executing a task that will enqueue thousands of jobs in our workers, this could cause a bottleneck in our Sidekiq queues.

Set up a new environment

Other documentation

Files

hosting.md

Latest commit

History

hosting.md

File metadata and controls

Hosting

Environments

Deployments

Azure Kubernetes Service (AKS) organisation and permission model

Getting Production access on Azure Platform

Installing the Azure Client and Kubectl

Login from Azure Console

Makefile shortcuts for kubernetes commands

Opening a Rails Console

Opening a shell

Running a rake task

Tailing application logs

Kubernetes commands

Listing the deployments (apps) in the cluster:

Listing the application pods (running instances) in the cluster:

Opening a console in a Review App

Executing commands in a Review App

Temporally scaling up our deployment instances

EG: Executing a task that will enqueue thousands of jobs in our workers, this could cause a bottleneck in our Sidekiq queues.

Set up a new environment

Other documentation