Message Queue Integration

Overview

This section provides a detailed guide on integrating a messaging queue using Redis and setting up workers with Supervisor for the langlearnai project. Supervisor allows the management of the worker processes that will handle tasks from the queue.
This ensures that tasks are processed asynchronously, improving the application's performance and scalability.

Integration

The Laravel application sends jobs to a Redis queue. These jobs require asynchronous processing. Redis temporarily holds these jobs until they are picked up by workers and processed.

Worker processes, managed by Supervisor, are always listening to the Redis queue for new jobs. When a job is added to the Redis queue, a worker retrieves it and processes it. This include things such as sending mails or any other background activity.

Logging and Monitoring: Worker processes report their operations and are monitored to verify they are running properly. If a worker fails, the Supervisor guarantees that it restarts automatically. Supervisor manages worker processes and edefines how workers operate, guaranteeing that they execute orders to process the Laravel queue.

Configuration

Setting up Redis

Installing Redis

sudo apt-get install redis

Editing the Redis Configuration file

sudo vim /etc/redis/redis.conf

Make the following chnages to these lines

bind 0.0.0.0
requirepass <user>
protected-mode no

• bind 0.0.0.0: This allows Redis to accept connections from any IP address
• requirepass : Sets a password that ensures only authorized clients can connect to Redis
• protected-mode no: Disabling protected mode allows connections from any IP without additional security checks(can be useful in a controlled environment)

Restart the service to apply changes

sudo systemctl restart redis

Setting up Supervisor

Installing Supervisor

sudo apt install supervisor -y

Creating Configuration Files for the Workers

cd /etc/supervisor/conf.d
sudo touch langlearn-be-main.conf langlearn-be-staging.conf

Edit the Configuration Files
Specify the worker settings for both the main and staging environments.

sudo vim langlearn-be-<enviroment>.conf #edit for the 

Add the following contents

# Add the following content for the main or staging environment
[program:langlearnai-<env>]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/langlearnai-be/<env>/artisan queue:work
autostart=true
autorestart=true
startsecs=10
startretries=10
stopasgroup=true
killasgroup=true
user=root
numprocs=8
redirect_stderr=true
stdout_logfile=/var/log/langlearn-<env>.log
stopwaitsecs=3600

• [program]: Defines the program name.
• process_name: Sets the naming pattern for the worker processes.
• command: Specifies the command to run the worker.
• autostart and autorestart: Ensure that the worker starts automatically and restarts if it fails.
• startsecs and startretries: Define how long to wait before considering the worker started and how many times to retry starting it.
• user: Specifies the user under which the worker runs.
• redirect_stderr and stdout_logfile: Handle logging for the worker processes.

Update Supervisor

sudo supervisorctl reread # Tells Supervisor to re-read its configuration files
sudo supervisorctl update # Applies the new configurations, making the workers ready to run.

Starting Workers

sudo supervisorctl start "langlearn-<env>:*"