Analyze Installation assistant YUM lock behavior

[teddytpc1]

Description

During the 4.9.0-apha3 Wazuh Installation assistant test and this issue it was found that the assistant fails to install Wazuh in AL2 with a YUM lock unexpected behavior:
23/07/2024 11:13:38 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (10/10)

This does not happen with the v4.8.1 script.

Tasks

• Validate (again) if it works fine with v4.8.1
• Analyze if some change introduced in 4.9.0 can generate this behavior
• Troubleshoot the issue
• Provide a fix and test it

[davidcr01]

Update Report

Validation

The environment is the following:

[root@ip-172-31-90-194 ec2-user]# cat /etc/os-release 
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

🟢 I have downloaded the 4.8.1 Wazuh Installation assistant and I could not reproduce the problem:

[root@ip-172-31-90-194 ec2-user]# bash wazuh-install.sh -a
06/08/2024 10:07:19 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.1
06/08/2024 10:07:19 INFO: Verbose logging redirected to /var/log/wazuh-install.log
06/08/2024 10:07:20 INFO: Verifying that your system meets the recommended minimum hardware requirements.
06/08/2024 10:07:23 INFO: Wazuh web interface port will be 443.
06/08/2024 10:07:24 INFO: Wazuh repository added.
06/08/2024 10:07:24 INFO: --- Configuration files ---
06/08/2024 10:07:24 INFO: Generating configuration files.
06/08/2024 10:07:24 INFO: Generating the root certificate.
06/08/2024 10:07:24 INFO: Generating Admin certificates.
06/08/2024 10:07:24 INFO: Generating Wazuh indexer certificates.
06/08/2024 10:07:24 INFO: Generating Filebeat certificates.
06/08/2024 10:07:25 INFO: Generating Wazuh dashboard certificates.
06/08/2024 10:07:25 INFO: Created wazuh-install-files.tar. It contains the Wazuh cluster key, certificates, and passwords necessary for installation.
06/08/2024 10:07:25 INFO: --- Wazuh indexer ---
06/08/2024 10:07:25 INFO: Starting Wazuh indexer installation.
06/08/2024 10:08:21 INFO: Wazuh indexer installation finished.
06/08/2024 10:08:21 INFO: Wazuh indexer post-install configuration finished.
06/08/2024 10:08:21 INFO: Starting service wazuh-indexer.
06/08/2024 10:08:32 INFO: wazuh-indexer service started.
06/08/2024 10:08:32 INFO: Initializing Wazuh indexer cluster security settings.
06/08/2024 10:08:42 INFO: Wazuh indexer cluster security configuration initialized.
06/08/2024 10:08:42 INFO: Wazuh indexer cluster initialized.
06/08/2024 10:08:42 INFO: --- Wazuh server ---
06/08/2024 10:08:42 INFO: Starting the Wazuh manager installation.
06/08/2024 10:09:07 INFO: Wazuh manager installation finished.
06/08/2024 10:09:07 INFO: Wazuh manager vulnerability detection configuration finished.
06/08/2024 10:09:07 INFO: Starting service wazuh-manager.
06/08/2024 10:09:19 INFO: wazuh-manager service started.
06/08/2024 10:09:19 INFO: Starting Filebeat installation.
06/08/2024 10:09:37 INFO: Filebeat installation finished.
06/08/2024 10:09:38 INFO: Filebeat post-install configuration finished.
06/08/2024 10:09:38 INFO: Starting service filebeat.
06/08/2024 10:09:38 INFO: filebeat service started.
06/08/2024 10:09:38 INFO: --- Wazuh dashboard ---
06/08/2024 10:09:38 INFO: Starting Wazuh dashboard installation.
06/08/2024 10:10:45 INFO: Wazuh dashboard installation finished.
06/08/2024 10:10:45 INFO: Wazuh dashboard post-install configuration finished.
06/08/2024 10:10:45 INFO: Starting service wazuh-dashboard.
06/08/2024 10:10:46 INFO: wazuh-dashboard service started.
06/08/2024 10:10:48 INFO: Updating the internal users.
06/08/2024 10:10:51 INFO: A backup of the internal users has been saved in the /etc/wazuh-indexer/internalusers-backup folder.
06/08/2024 10:11:25 INFO: Initializing Wazuh dashboard web application.
06/08/2024 10:11:25 INFO: Wazuh dashboard web application initialized.
06/08/2024 10:11:25 INFO: --- Summary ---
06/08/2024 10:11:25 INFO: You can access the web interface https://<wazuh-dashboard-ip>:443
    User: admin
    Password: xT9FfIOH*SPB94+Ry55FfhIsAgziFnzc
06/08/2024 10:11:25 INFO: Installation finished.

🟢 In 4.9.0, it is observed that the YUM lock behavior is reported twice but the installation was finished successfully:

[root@ip-172-31-90-194 ec2-user]# curl -sO https://packages-dev.wazuh.com/4.9/wazuh-install.sh && bash ./wazuh-install.sh -a
06/08/2024 10:42:50 INFO: Starting Wazuh installation assistant. Wazuh version: 4.9.0
06/08/2024 10:42:50 INFO: Verbose logging redirected to /var/log/wazuh-install.log
06/08/2024 10:42:50 INFO: Verifying that your system meets the recommended minimum hardware requirements.
06/08/2024 10:42:50 INFO: Wazuh web interface port will be 443.
06/08/2024 10:42:50 INFO: Wazuh development repository added.
06/08/2024 10:42:50 INFO: --- Configuration files ---
06/08/2024 10:42:50 INFO: Generating configuration files.
06/08/2024 10:42:51 INFO: Generating the root certificate.
06/08/2024 10:42:51 INFO: Generating Admin certificates.
06/08/2024 10:42:51 INFO: Generating Wazuh indexer certificates.
06/08/2024 10:42:51 INFO: Generating Filebeat certificates.
06/08/2024 10:42:51 INFO: Generating Wazuh dashboard certificates.
06/08/2024 10:42:51 INFO: Created wazuh-install-files.tar. It contains the Wazuh cluster key, certificates, and passwords necessary for installation.
06/08/2024 10:42:51 INFO: --- Wazuh indexer ---
06/08/2024 10:42:51 INFO: Starting Wazuh indexer installation.
06/08/2024 10:43:20 INFO: Wazuh indexer installation finished.
06/08/2024 10:43:20 INFO: Wazuh indexer post-install configuration finished.
06/08/2024 10:43:20 INFO: Starting service wazuh-indexer.
06/08/2024 10:43:33 INFO: wazuh-indexer service started.
06/08/2024 10:43:33 INFO: Initializing Wazuh indexer cluster security settings.
06/08/2024 10:43:37 INFO: Wazuh indexer cluster security configuration initialized.
06/08/2024 10:43:37 INFO: Wazuh indexer cluster initialized.
06/08/2024 10:43:37 INFO: --- Wazuh server ---
06/08/2024 10:43:37 INFO: Starting the Wazuh manager installation.
06/08/2024 10:44:02 INFO: Wazuh manager installation finished.
06/08/2024 10:44:02 INFO: Wazuh manager vulnerability detection configuration finished.
06/08/2024 10:44:02 INFO: Starting service wazuh-manager.
06/08/2024 10:44:12 INFO: wazuh-manager service started.
06/08/2024 10:44:12 INFO: Starting Filebeat installation.
06/08/2024 10:44:12 INFO: Another process is using YUM. Waiting for it to release the lock. Next retryYU in 30 seconds (1/10)
06/08/2024 10:44:51 INFO: Filebeat installation finished.
06/08/2024 10:44:52 INFO: Filebeat post-install configuration finished.
06/08/2024 10:44:52 INFO: Starting service filebeat.
06/08/2024 10:44:52 INFO: filebeat service started.
06/08/2024 10:44:52 INFO: --- Wazuh dashboard ---
06/08/2024 10:44:52 INFO: Starting Wazuh dashboard installation.
06/08/2024 10:44:52 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (1/10)
06/08/2024 10:46:20 INFO: Wazuh dashboard installation finished.
06/08/2024 10:46:20 INFO: Wazuh dashboard post-install configuration finished.
06/08/2024 10:46:20 INFO: Starting service wazuh-dashboard.
06/08/2024 10:46:21 INFO: wazuh-dashboard service started.
06/08/2024 10:46:21 INFO: Updating the internal users.
06/08/2024 10:46:26 INFO: A backup of the internal users has been saved in the /etc/wazuh-indexer/internalusers-backup folder.
06/08/2024 10:46:36 INFO: The filebeat.yml file has been updated to use the Filebeat Keystore username and password.
06/08/2024 10:47:02 INFO: Initializing Wazuh dashboard web application.
06/08/2024 10:47:03 INFO: Wazuh dashboard web application initialized.
06/08/2024 10:47:03 INFO: --- Summary ---
06/08/2024 10:47:03 INFO: You can access the web interface https://<wazuh-dashboard-ip>:443
    User: admin
    Password: F5IrzrxZjqs?vrFn?Z6C3*Fil3A51EZU
06/08/2024 10:47:03 INFO: Installation finished.
[root@ip-172-31-90-194 ec2-user]# 

[davidcr01]

Update Report

Research

In the AL2 machine of the wazuh/wazuh#24874, I could reproduce the YUM lock problem:

07/08/2024 07:50:20 INFO: Removing Wazuh manager.
07/08/2024 07:50:20 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (1/10)
07/08/2024 07:50:50 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (2/10)
07/08/2024 07:51:20 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (3/10)
07/08/2024 07:51:50 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (4/10)
07/08/2024 07:52:20 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (5/10)
07/08/2024 07:52:50 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (6/10)
07/08/2024 07:53:20 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (7/10)
07/08/2024 07:53:50 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (8/10)
07/08/2024 07:54:20 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (9/10)

⚠️ While the YUM lock messages are being generated, the PID that reports the YUM lock file does not exist in the system, so we can think that an unknown reason corrupts the YUM lock file. This is the reason why the Installation assistant can not finish the installation successfully, as the YUM lock is always active, although no process is using it.

[root@ip-172-31-41-84 ec2-user]# cat /var/run/yum.pid 
8374
[root@ip-172-31-41-84 ec2-user]# ps aux | grep yum
root      9745  0.0  0.0 119424   964 pts/1    S+   07:50   0:00 grep --color=auto yum
[root@ip-172-31-41-84 ec2-user]# ps aux | grep 8374
root      9793  0.0  0.0 119424   948 pts/1    S+   07:52   0:00 grep --color=auto 8374

⁉️ After finishing the installation, the YUM lock still exists, but if I execute a yum update command, the YUM lock is removed:

8374[root@ip-172-31-41-84 ec2-user]# cat /var/run/yum.pid 
8374[root@ip-172-31-41-84 ec2-user]# ps aux | grep yum
root     10019  0.0  0.0 119424   988 pts/1    S+   08:05   0:00 grep --color=auto yum
[root@ip-172-31-41-84 ec2-user]# yum update
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                    | 3.6 kB  00:00:00     
No packages marked for update
[root@ip-172-31-41-84 ec2-user]# cat /var/run/yum.pid 
cat: /var/run/yum.pid: No such file or directory
[root@ip-172-31-41-84 ec2-user]# 

[davidcr01]

Update Report

I have investigated the differences between the 4.8.1 and 4.9.0 Installation assistant and no differences that may impact into the YUM lock behavior were found.
Differences: https://www.diffchecker.com/2eVm9AL4/

Analyzing the behavior

I have been investigating the behavior of the YUM lock when the Installation assistant is being executed, and I come up with the following conclusions:

• Everytime a package is being installed (such as the Wazuh central components), YUM runs the following command /usr/bin/python /usr/bin/yum --debuglevel 2 --security check-update, which performs a security update check and provides detailed debug information.

• After this process completes, the lock file /var/run/yum.pid remains, even though the process associated with the PID in the file has finished. This lock file is supposed to be removed automatically when YUM completes its task.

• The persistence of this lock file seems to prevent YUM from functioning properly, as other operations that rely on YUM cannot proceed until the lock is cleared.

In the following code, it is checked that the YUM lock is enabled, but no process is using YUM:

07/08/2024 09:19:02 INFO: Starting Wazuh dashboard installation.
07/08/2024 09:19:03 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (1/10)
07/08/2024 09:19:33 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (2/10)
07/08/2024 09:20:03 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (3/10)
07/08/2024 09:20:33 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (4/10)
07/08/2024 09:21:03 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (5/10)
07/08/2024 09:21:33 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (6/10)
07/08/2024 09:22:03 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (7/10)
07/08/2024 09:22:33 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (8/10)
07/08/2024 09:23:03 INFO: Another process is using YUM. Waiting for it to release the lock. Next retry in 30 seconds (9/10)

In another terminal:

[root@ip-172-31-41-84 ec2-user]# ps -ef | grep yum | grep -v grep
[root@ip-172-31-41-84 ec2-user]# 

[root@ip-172-31-41-84 ec2-user]# cat /var/run/yum.pid
14201[root@ip-172-31-41-84 ec2-user]# 

[root@ip-172-31-41-84 ec2-user]# ps aux | grep 14201
root     15723  0.0  0.0 119424   920 pts/0    S+   09:22   0:00 grep --color=auto 14201

[davidcr01]

Update Report

Conclusions

❌ In conclusion, the Installation Assistant will not manage, modify, or delete the YUM lock file due to potential system intrusion risks, as this file is an internal component of the system's package manager.

🟡 If this issue occurs again, we will report it as a known issue.

💻 This decision is based on tests conducted, which indicate that the problem appears to be random and not dependent on the AMI or the kernel version used.

🧪 If this behavior is reproduced in our testing, we should decide whether to delete the YUM lock manually or not in the testing process. Also, I have observed that if the yum update command is executed when the lock is corrupted (not released when it is not used), the YUM package manager will delete the lock.

💡 Also, we should consider changing the OSs of the Installation assistant test, as we intend to use the latest OSs, we should consider changing AL2 to AL2023 and changing Ubuntu 22.04 to Ubuntu 24.04. This should be discussed with the management team.

[c-bordon]

LGTM