Video Manipulation Repo Set Up
This boilerplate repo initially housed the endpoints (audio extraction, video concatenation, video watermark, etc,) for the core product video manipulation. To make the video processing endpoints independent of the boilerplate, a aivideo repo was set up. The repository can be found at https://github.com/hngprojects/aivideo_be_java.
The aivideo repository now houses the endpoints that communicate with the background processes at https://github.com/hngprojects/Hng_videoSuite_java through RabbitMQ and a shared database connection. When a client sends a video request to audio extraction endpoint for example, this application converts the video into bytes and then JSON strings and sends it through RabbitMQ on video concat queue.
The second application which is the background process then manipulates the video and extracts the audio, and sends it through finished concat queue. This repository also has a listener on that queue and converts the received audio from JSON string to bytes and saves it on the server pending when the client requests for the file and converts it to the expected media format.
It communicates with the application in this repo for video processing.
The core product: https://staging.tifi.tv/. This houses all other features being built by Team Anchor.
The application is accessible at this endpoint: https://staging.tifi.tv/videosuite/
For the CI
name: CI
on:
pull_request:
jobs:
build-and-test:
runs-on: aividci
strategy:
matrix:
maven: [ '3.8.8' ]
services:
postgres:
image: postgres:latest
env:
POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
POSTGRES_DB: ${{ secrets.POSTGRES_DB }}
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
rabbitmq:
image: rabbitmq:3-management
env:
RABBITMQ_DEFAULT_USER: ${{ secrets.RABBITMQ_USER }}
RABBITMQ_DEFAULT_PASS: ${{ secrets.RABBITMQ_PASSWORD }}
ports:
- 5672:5672
- 15672:15672
options: >-
--health-cmd "rabbitmqctl node_health_check"
--health-interval 10s
--health-timeout 5s
--health-retries 5
env:
MAVEN_OPTS: "-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
steps:
- name: Checkout repository base
uses: actions/checkout@v4
- name: Setup Maven Action
uses: s4u/[email protected]
with:
java-version: 17
maven-version: ${{ matrix.maven }}
- name: Create application.properties
run: |
echo "spring.profiles.active=ci" > src/main/resources/application.properties
echo "${{ secrets.APPLICATION_PROPERTIES_CI }}" | base64 --decode > src/main/resources/application-ci.properties
- name: Check Maven Version
run: mvn --version
- name: Build the application
run: mvn verify -B
- name: Start Application
run: java -jar target/aivideo_be_java-0.0.1-SNAPSHOT.jar & sleep 15
This configuration builds and starts the application when a PR is made. This checks the code changes and ensures no errors are made and the application starts successfully. If the CI passes, the application can be merged into the dev branch for deployment.
For the CD
name: Dev Deployment
on:
workflow_dispatch:
push:
branches:
- dev
jobs:
build-and-push:
if: github.event.repository.fork == false
name: Build for dev
runs-on: aividci
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Maven Action
uses: s4u/[email protected]
with:
java-version: 17
maven-version: 3.8.8
- name: Create application-dev.properties
run: |
echo "spring.profiles.active=dev" > src/main/resources/application.properties
echo "${{ secrets.APPLICATION_PROPERTIES_DEV }}" | base64 --decode > src/main/resources/application-dev.properties
- name: Build the application
run: mvn clean package -DskipTests
- name: List target directory
run: ls -l target/*.jar
- name: Copy Jar file to Server
uses: appleboy/[email protected]
with:
host: ${{ secrets.SSH_HOST }}
username: ${{ secrets.SSH_USERNAME }}
password: ${{ secrets.SSH_PASSWORD }}
source: "target/*.jar"
target: "/home/teamjava/aivideo_be_java/"
- name: Deploy to server
uses: appleboy/ssh-action@master
with:
host: ${{ secrets.SSH_HOST }}
username: ${{ secrets.SSH_USERNAME }}
password: ${{ secrets.SSH_PASSWORD }}
script: |
set -x
cd /home/teamjava/aivideo_be_java
git fetch origin dev
git stash
git checkout dev
git pull origin dev
sudo cp .github/scripts/aivideo_java_dev.service /etc/systemd/system/aivideo_java_dev.service
sudo systemctl daemon-reload
sudo systemctl restart aivideo_java_dev.service
This configuration uses the aividci runner to build the application deployed on the dev branch, copies the Jar file to the deployment server, fetches changes from the dev branch, and starts the application in the server.
To prevent app downtime in case of server restart, the application was configured to start as a service using systemd.
[Unit]
Description=AI Video Development
After=network.target
[Service]
User=teamjava
Group=teamjava
WorkingDirectory=/home/teamjava/aivideo_be_java
ExecStart=/usr/bin/java -jar /home/teamjava/aivideo_be_java/target/aivideo_be_java-0.0.1-SNAPSHOT.jar --spring.profiles.active=dev
[Install]
WantedBy=multi-user.target
Nginx configuration was updated in the server so that the application can be accessible at the /videosuite endpoint
# Define upstream services
upstream aivideo_staging {
server localhost:3010;
}
upstream aivideo {
server localhost:3011;
}
upstream videosuite_dev {
server localhost:8085;
}
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
server_name status.tifi.tv;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:3010/status;
}
location ~ ^/(_next|static|api) {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Pass these requests to the root of your application
proxy_pass http://127.0.0.1:3010$request_uri;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/status.tifi.tv/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/status.tifi.tv/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
# Redirect HTTP to HTTPS for tifi.tv
server {
if ($host = tifi.tv) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name tifi.tv;
return 301 https://$host$request_uri;
}
# HTTPS server for tifi.tv
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name tifi.tv;
# SSL certificate paths will be configured by Certbot
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffers 16 16k;
proxy_buffer_size 16k;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass http://aivideo;
}
ssl_certificate /etc/letsencrypt/live/staging.tifi.tv/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/staging.tifi.tv/privkey.pem; # managed by Certbot
}
# Redirect HTTP to HTTPS for staging.tifi.tv
server {
if ($host = staging.tifi.tv) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name staging.tifi.tv;
return 301 https://$host$request_uri;
}
# HTTPS server for staging.tifi.tv
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name staging.tifi.tv;
# SSL certificate paths will be configured by Certbot
location / {
#auth_basic "Restricted";
#auth_basic_user_file /etc/nginx/.htpasswd;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# proxy_buffers 16 16k;
# proxy_buffer_size 16k;
proxy_buffering off;
proxy_http_version 1.1;
chunked_transfer_encoding on;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_pass http://aivideo_staging;
}
location /videosuite {
proxy_pass http://videosuite_dev/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
rewrite ^/videosuite/(.*) /$1 break;
proxy_set_header X-Forwarded-Prefix /videosuite;
}
ssl_certificate /etc/letsencrypt/live/staging.tifi.tv/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/staging.tifi.tv/privkey.pem; # managed by Certbot
}
server {
if ($host = status.tifi.tv) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name status.tifi.tv;
return 404; # managed by Certbot
}
