Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added the actions and necessary files for the CI/CD flow #25

Merged
merged 4 commits into from
Aug 28, 2023
Merged

Added the actions and necessary files for the CI/CD flow #25

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b8152e1
added the needed k8s files for the job and cronjob definitions and cr…
Aug 2, 2023
e2dac3e
Update deployColl.yml to check the tag version of the release and the…
NicolaeMarius Aug 25, 2023
4f4dc01
Update deployProd.yml to check the version of the release tag and the…
NicolaeMarius Aug 25, 2023
771febe
Update deployColl.yml to check the tag version of the release with th…
NicolaeMarius Aug 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions .github/workflows/deployColl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
on:
release:
types: [created]
tags:
- 'coll-v*\.*\.*'

name: AWS PCM Build and Deploy Collaudo

jobs:
deploy:
if: ${{ startsWith(github.ref, 'refs/tags/coll') }}
name: Deploy
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v2
- name: Print GITHUB_REF
run: echo "GITHUB_REF=${GITHUB_REF}"
- name: Set output
id: vars
run: echo "tag=${GITHUB_REF#refs/*/}" | sed 's/coll-v//' >> "${GITHUB_OUTPUT}"
- name: Check output
env:
RELEASE_VERSION: ${{ steps.vars.outputs.tag }}
run: |
echo $RELEASE_VERSION
echo ${{ steps.vars.outputs.tag }}

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-south-1

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Build, tag, and push the image to Amazon ECR
id: build-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: ${{ secrets.REPO_NAME }}
IMAGE_TAG: ${{ steps.vars.outputs.tag }}
run: |
# Build a docker container and push it to ECR
docker build --build-arg GEOIP_LICENSE=${{ secrets.GEOIP_LICENSE }} -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
echo "Pushing image to COLL ECR..."
docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
echo "::set-output name=image::$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG"
echo "Pushed image to COLL ECR "

- name: Update kube config for coll EKS
id: update-kube-config-coll
run: aws eks update-kubeconfig --name ${{ secrets.EKS_CLUSTER_NAME }}

- name: Apply the deployment to EKS
id: deploy
env:
IMAGE_TAG: ${{ steps.build-image.outputs.image }}
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: ${{ secrets.REPO_NAME }}
run: |
echo "Image tag: $IMAGE_TAG"
tag=$(echo ${IMAGE_TAG##*:})
echo "coll ecr: $ECR_REGISTRY/$ECR_REPOSITORY:$tag"
echo "Deploying to test EKS..."
cat git-deployment-coll.yml | sed "s|ImagePlaceholder|$ECR_REGISTRY/$ECR_REPOSITORY:$tag|g" | kubectl apply -f -

- name: Check the deploy to EKS COLL
id: deploy-eks-coll
run: |
kubectl get cronjobs -n dtd-crawler-coll
tensor5 marked this conversation as resolved.
Show resolved Hide resolved

- name: Send SNS notification when the deploy completes in collaudo
id: sns-success
if: success()
run: |
aws sns publish --topic-arn ${{ secrets.SNS_TOPIC_ARN }} --subject "[PCM DTD CRAWLER COLL] Deployment della versione ${{ steps.vars.outputs.tag }} avvenuto con successo" --message "Il deployment della versione ${{ steps.vars.outputs.tag }} è avvenuto con successo su EKS COLL"

- name: Send SNS notification when the deploy fails in collaudo
id: sns-failure
if: failure()
run: |
aws sns publish --topic-arn ${{ secrets.SNS_TOPIC_ARN }} --subject "[PCM DTD CRAWLER COLL] Deployment della versione ${{ steps.vars.outputs.tag }} fallito" --message "Il deployment della versione ${{ steps.vars.outputs.tag }} è fallito su EKS COLL"
85 changes: 85 additions & 0 deletions .github/workflows/deployProd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
on:
release:
types: [created]
tags:
- 'prod-v*\.*\.*'

name: AWS PCM Promote and deploy PROD

jobs:
deploy:
if: ${{ startsWith(github.ref, 'refs/tags/prod') }}
name: Deploy
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set output
id: vars
run: echo "tag=${GITHUB_REF#refs/*/}" | sed 's/prod-v//' >> "${GITHUB_OUTPUT}"
- name: Check output
env:
RELEASE_VERSION: ${{ steps.vars.outputs.tag }}
run: |
echo $RELEASE_VERSION
echo ${{ steps.vars.outputs.tag }}

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: eu-south-1

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Build, tag, and push the image to Amazon ECR
id: build-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: ${{ secrets.REPO_NAME }}
IMAGE_TAG: ${{ steps.vars.outputs.tag }}
ECR_REPOSITORY_PROD: ${{ secrets.REPO_NAME_PROD }}

run: |
docker pull $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
docker tag $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY_PROD:$IMAGE_TAG
docker push $ECR_REGISTRY/$ECR_REPOSITORY_PROD:$IMAGE_TAG

- name: Update kube config for prod EKS
id: update-kube-config-prod
run: aws eks update-kubeconfig --name ${{ secrets.EKS_CLUSTER_NAME_PROD }}

- name: Apply the deployment to EKS
id: deploy
env:
IMAGE_TAG: ${{ steps.build-image.outputs.image }}
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: ${{ secrets.REPO_NAME }}
ECR_REPOSITORY_PROD: ${{ secrets.REPO_NAME_PROD }}
run: |
echo "Image tag: $IMAGE_TAG"
tag=$(echo ${IMAGE_TAG##*:})
echo "prod ecr: $ECR_REGISTRY/$ECR_REPOSITORY_PROD:$tag"
echo "Deploying to prod EKS..."
cat git-deployment-prod.yml | sed "s|ImagePlaceholder|$ECR_REGISTRY/$ECR_REPOSITORY_PROD:$tag|g" | kubectl apply -f -

- name: Check the deploy to EKS PROD
id: deploy-eks-prod
run: kubectl get cronjobs -n dtd-crawler-prod


- name: Send SNS notification when the deploy completes in production
id: sns-success
if: success()
run: |
aws sns publish --topic-arn ${{ secrets.SNS_TOPIC_ARN }} --subject "[PCM DTD CRAWLER PROD] Deployment della versione ${{ steps.vars.outputs.tag }} avvenuto con successo" --message "Il deployment della versione ${{ steps.vars.outputs.tag }} è avvenuto con successo su EKS PROD"

- name: Send SNS notification when the deploy fails in production
id: sns-failure
if: failure()
run: |
aws sns publish --topic-arn ${{ secrets.SNS_TOPIC_ARN }} --subject "[PCM DTD CRAWLER PROD] Deployment della versione ${{ steps.vars.outputs.tag }} fallito" --message "Il deployment della versione ${{ steps.vars.outputs.tag }} è fallito su EKS PROD"
143 changes: 143 additions & 0 deletions git-deployment-coll.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
apiVersion: batch/v1
kind: Job
metadata:
name: dtd-crawler-web-server-job-coll
namespace: dtd-crawler-coll
labels:
app: dtd-crawler-web-server-coll
spec:
template:
metadata:
labels:
app: dtd-crawler-web-server-coll
spec:
containers:
- name: dtd-crawler-web-server-coll
image: ImagePlaceholder
imagePullPolicy: Always
command: ["npm", "run", "dist-webserver"]
envFrom:
- configMapRef:
name: dtd-crawler-env-configmap-coll
- secretRef:
name: dtd-crawler-database-secret-coll
- secretRef:
name: dtd-crawler-basicauth-secret-coll
- secretRef:
name: dtd-crawler-aws-s3-secret-coll
- secretRef:
name: dtd-crawler-jwt-secret-coll
- secretRef:
name: dtd-crawler-pa2026-secret-coll
restartPolicy: Never
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: dtd-crawler-pa2026-manager-job-coll
namespace: dtd-crawler-coll
labels:
app: dtd-crawler-pa2026-manager-coll
spec:
schedule: "0 22 * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: dtd-crawler-pa2026-manager-coll
spec:
restartPolicy: Never
containers:
- name: dtd-crawler-pa2026-manager-coll
image: ImagePlaceholder
imagePullPolicy: Always
command: ["npm", "run", "dist-PA2026-manager"]
envFrom:
- configMapRef:
name: dtd-crawler-env-configmap-coll
- secretRef:
name: dtd-crawler-database-secret-coll
- secretRef:
name: dtd-crawler-basicauth-secret-coll
- secretRef:
name: dtd-crawler-aws-s3-secret-coll
- secretRef:
name: dtd-crawler-jwt-secret-coll
- secretRef:
name: dtd-crawler-pa2026-secret-coll
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: dtd-crawler-queue-manager-job-coll
namespace: dtd-crawler-coll
labels:
app: dtd-crawler-queue-manager-coll
spec:
schedule: "0 0,12 * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: dtd-crawler-queue-manager-coll
spec:
restartPolicy: Never
containers:
- name: dtd-crawler-queue-manager-coll
image: ImagePlaceholder
imagePullPolicy: Always
command:
- /bin/sh
- -c
- node --max-old-space-size=8192 --no-warnings --experimental-modules --es-module-specifier-resolution=node ./dist/command/queueManager.js --maxItems 1000 --passedOlderThanDays 0 --failedOlderThanDays 0 --asservationOlderThanDays 0 --manualScanLogic true
envFrom:
- configMapRef:
name: dtd-crawler-env-configmap-coll
- secretRef:
name: dtd-crawler-database-secret-coll
- secretRef:
name: dtd-crawler-basicauth-secret-coll
- secretRef:
name: dtd-crawler-aws-s3-secret-coll
- secretRef:
name: dtd-crawler-jwt-secret-coll
- secretRef:
name: dtd-crawler-pa2026-secret-coll
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: dtd-crawler-scan-manager-job-coll
namespace: dtd-crawler-coll
labels:
app: dtd-crawler-scan-manager-coll
spec:
schedule: "0 2,14 * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: dtd-crawler-scan-manager-coll
spec:
restartPolicy: Never
containers:
- name: dtd-crawler-scan-manager-coll
image: ImagePlaceholder
imagePullPolicy: Always
command: ["npm", "run", "dist-scan-manager"]
envFrom:
- configMapRef:
name: dtd-crawler-env-configmap-coll
- secretRef:
name: dtd-crawler-database-secret-coll
- secretRef:
name: dtd-crawler-basicauth-secret-coll
- secretRef:
name: dtd-crawler-aws-s3-secret-coll
- secretRef:
name: dtd-crawler-jwt-secret-coll
- secretRef:
name: dtd-crawler-pa2026-secret-coll
Loading