Test cluster worker logs order is detecting old format of logs in cluster.log

[pro-akim]

Description

Workload benchmark testing were performed for Release of v4.6.0-pre-alpha1
Running manually reliability/test_cluster/test_cluster_logs/test_cluster_worker_logs_order
/test_cluster_worker_logs_order.py::test_check_logs_order_workers over artifacts created by workload benchmark pipeline (It should be fixed #4299) it is possible to find that the log messages provided by Wazuh are different than what the test is expecting. Small format changes could be detected, however, the meaning of the messages are still the same as the old format.

Current behavior

Detected errors in the test:

E            - Log type: Integrity check
E              Expected logs: ['Sending zip file to master.*']
E              Found log: 2023/07/25 09:03:05 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Integrity check] Sending zip file.

E            - Log type: Agent-groups recv
E              Expected logs: ['The checksum of both databases match.*Reset the attempt counter.', 'The checksum of both databases match.*', 'Checksum comparison failed. Attempt 10/10.*', 'Checksum comparison failed. Attempt .*', "The master's checksum and the worker's checksum are different. Local checksum:.*| Master checksum:.*."]
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Agent-groups recv] The checksum of master (0107654add85786339eba7c5fa72cee25783dd6c) and worker (4ba9e4d09c32baef1d763fd81e3cf804aa06b607) are different.

Expected behavior

No errors should be detected.

[EduLeon12]

Issue Update

Probably related to #4364, Once the issue is resolved this will be worked on.

[pro-akim]

Update

After running the test again over the same artifacts, some changes were detected.

From:

Frist

E            - Log type: Integrity check
E              Expected logs: ['Sending zip file to master.*']
E              Found log: 2023/07/25 09:03:05 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Integrity check] Sending zip file.

And

Second:

E            - Log type: Agent-groups recv
E              Expected logs: ['The checksum of both databases match.*Reset the attempt counter.', 'The checksum of both databases match.*', 'Checksum comparison failed. Attempt 10/10.*', 'Checksum comparison failed. Attempt .*', "The master's checksum and the worker's checksum are different. Local checksum:.*| Master checksum:.*."]
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Agent-groups recv] The checksum of master (0107654add85786339eba7c5fa72cee25783dd6c) and worker (4ba9e4d09c32baef1d763fd81e3cf804aa06b607) are different.

Only the second error is still visible

[RamosFe]

Update

Replicated the error locally using the artifacts of build 274:

python3 -m pytest test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py --artifacts_path='/Users/framos/Desktop/artifacts'
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.9.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/framos/Documents/Wazuh/Repositories/wazuh-qa/tests, configfile: pytest.ini
plugins: testinfra-5.0.0, html-3.1.1, metadata-2.0.4
collected 1 item

test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py F                                                                          [100%]

==================================================================================== FAILURES ====================================================================================
_________________________________________________________________________ test_check_logs_order_workers __________________________________________________________________________

artifacts_path = '/Users/framos/Desktop/artifacts'

    def test_check_logs_order_workers(artifacts_path):
        """Check that cluster logs appear in the expected order.

        Check that for each group of logs (agent-info, integrity-check, etc), each message
        appears in the order it should. If any log is duplicated, skipped, etc. the test will fail.

        Args:
            artifacts_path (str): Path where folders with cluster information can be found.
        """
        if not artifacts_path:
            pytest.fail('Parameter "--artifacts_path=<path>" is required.')

        cluster_log_files = glob(os.path.join(artifacts_path, 'worker_*', 'logs', 'cluster.log'))
        if len(cluster_log_files) == 0:
            pytest.fail(f'No files found inside {artifacts_path}.')

        for log_file in cluster_log_files:
            failed_tasks = set()

            with open(log_file) as file:
                for line in file.readlines():
                    result = worker_logs_format.search(line)
                    if result:
                        if result.group(1) in logs_order and result.group(1) not in failed_tasks:
                            tree_info = logs_order[result.group(1)]
                            for child in tree_info['tree'].children(tree_info['node']):
                                if re.search(child.tag, result.group(2)):
                                    # Current node is updated so the tree points to the next expected log.
                                    logs_order[result.group(1)]['node'] = child.identifier if \
                                        tree_info['tree'].children(child.identifier) else 'root'

                                    break
                            else:
                                # Log can be different to the expected one only if permission was not granted.
                                if "Master didn't grant permission to start a new" not in result.group(2):
                                    if node_name.search(log_file)[1] not in incorrect_order:
                                        incorrect_order[node_name.search(log_file)[1]] = []
                                    incorrect_order[node_name.search(log_file)[1]].append({
                                        'log_type': result.group(1),
                                        'found_log': result.group(0),
                                        'expected_logs': [log.tag for log in tree_info['tree'].children(tree_info['node'])]
                                    })

                                    failed_tasks.add(result.group(1))

            # Update status of all logs so they point to their tree root.
            for log_type, tree_info in logs_order.items():
                tree_info['node'] = 'root'

        if incorrect_order:
            result = ''
            for node, info in incorrect_order.items():
                result += f"\n\n{node}"
                for items in info:
                    result += '\n - Log type: {log_type}\n' \
                              '   Expected logs: {expected_logs}\n' \
                              '   Found log: {found_log}\n'.format(**items)

>           pytest.fail(result)
E           Failed:
E
E           worker_2
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_2] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_5
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_5] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_4
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:33 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_4] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_3
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_3] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_15
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_15] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_12
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_12] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_24
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_24] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_23
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_23] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_22
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_22] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_25
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_25] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_13
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_13] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_14
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_14] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_6
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_6] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_1
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_1] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_8
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_8] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_9
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_9] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_7
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_7] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_11
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_11] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_16
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_16] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_20
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_20] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_18
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_18] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_19
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_19] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_21
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_17
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_17] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_10
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_10] [Agent-groups recv] Checksum comparison failed (1/5).

test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py:103: Failed
============================================================================ short test summary info =============================================================================
FAILED test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py::test_check_logs_order_workers - Failed:
========================================================================== 1 failed in 66.22s (0:01:06) ==========================================================================

After changing the format of the log to match the new format, the following error occur:

python3 -m pytest test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py --artifacts_path='/Users/framos/Desktop/artifacts'
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.9.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/framos/Documents/Wazuh/Repositories/wazuh-qa/tests, configfile: pytest.ini
plugins: testinfra-5.0.0, html-3.1.1, metadata-2.0.4
collected 1 item

test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py F                                                                          [100%]

==================================================================================== FAILURES ====================================================================================
_________________________________________________________________________ test_check_logs_order_workers __________________________________________________________________________

artifacts_path = '/Users/framos/Desktop/artifacts'

    def test_check_logs_order_workers(artifacts_path):
        """Check that cluster logs appear in the expected order.

        Check that for each group of logs (agent-info, integrity-check, etc), each message
        appears in the order it should. If any log is duplicated, skipped, etc. the test will fail.

        Args:
            artifacts_path (str): Path where folders with cluster information can be found.
        """
        if not artifacts_path:
            pytest.fail('Parameter "--artifacts_path=<path>" is required.')

        cluster_log_files = glob(os.path.join(artifacts_path, 'worker_*', 'logs', 'cluster.log'))
        if len(cluster_log_files) == 0:
            pytest.fail(f'No files found inside {artifacts_path}.')

        for log_file in cluster_log_files:
            failed_tasks = set()

            with open(log_file) as file:
                for line in file.readlines():
                    result = worker_logs_format.search(line)
                    if result:
                        if result.group(1) in logs_order and result.group(1) not in failed_tasks:
                            tree_info = logs_order[result.group(1)]
                            for child in tree_info['tree'].children(tree_info['node']):
                                if re.search(child.tag, result.group(2)):
                                    # Current node is updated so the tree points to the next expected log.
                                    logs_order[result.group(1)]['node'] = child.identifier if \
                                        tree_info['tree'].children(child.identifier) else 'root'

                                    break
                            else:
                                # Log can be different to the expected one only if permission was not granted.
                                if "Master didn't grant permission to start a new" not in result.group(2):
                                    if node_name.search(log_file)[1] not in incorrect_order:
                                        incorrect_order[node_name.search(log_file)[1]] = []
                                    incorrect_order[node_name.search(log_file)[1]].append({
                                        'log_type': result.group(1),
                                        'found_log': result.group(0),
                                        'expected_logs': [log.tag for log in tree_info['tree'].children(tree_info['node'])]
                                    })

                                    failed_tasks.add(result.group(1))

            # Update status of all logs so they point to their tree root.
            for log_type, tree_info in logs_order.items():
                tree_info['node'] = 'root'

        if incorrect_order:
            result = ''
            for node, info in incorrect_order.items():
                result += f"\n\n{node}"
                for items in info:
                    result += '\n - Log type: {log_type}\n' \
                              '   Expected logs: {expected_logs}\n' \
                              '   Found log: {found_log}\n'.format(**items)

>           pytest.fail(result)
E           Failed:
E
E           worker_2
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_2] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_5
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_5] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_4
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:33 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_4] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_3
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_3] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_15
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_15] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_12
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_12] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_24
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_24] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_23
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_23] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_22
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_22] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_25
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_25] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_13
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_13] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_14
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_14] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_6
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_6] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_1
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_1] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_8
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_8] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_9
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_9] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_7
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_7] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_11
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_11] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_16
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_16] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_20
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_20] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_18
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_18] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_19
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_19] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_21
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_21] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_17
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_17] [Agent-groups recv] Checksum comparison failed (1/5).
E
E
E           worker_10
E            - Log type: Agent-groups recv
E              Expected logs: ['Sent request to obtain all agent-groups information from the master node.*']
E              Found log: 2023/07/25 09:25:34 DEBUG: [Worker CLUSTER-Workload_benchmarks_metrics_B274_manager_10] [Agent-groups recv] Checksum comparison failed (1/5).

test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py:103: Failed
============================================================================ short test summary info =============================================================================
FAILED test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py::test_check_logs_order_workers - Failed:

This is caused by the dependency created in Agent-groups_recv. The YAML file has a list of logs and their dependencies that the script then parses into a tree, once the tests match one of the logs with a tree node, they proceed to follow the child nodes and validate that the next log has the message of the child node and thus validates the order of the logs.

wazuh-qa/tests/reliability/test_cluster/test_cluster_logs/test_cluster_worker_logs_order/data/Agent-groups_recv.yml

Lines 52 to 61 in b184c9e

	# The checksum differs
	- log_id: log9
	parent: log2
	tag: "The master's checksum and the worker's checksum are different. Local checksum:.*| Master checksum:.*."
	- log_id: log10
	parent: log9
	tag: 'Sent request to obtain all agent-groups information from the master node.*'
	- log_id: log11
	parent: log10
	tag: 'Finished in.*Updated.*chunks.*'

In this case, the following log should only appear if the agent_groups_mismatch_limit from the cluster.json is exceeded, but the test seems not to take this condition into account.

I'm still debugging the code to validate the behavior described above.

[RamosFe]

Update

Added new nodes to the tree depending on the agent_groups_mismatch_limit condition:

wazuh-qa/tests/reliability/test_cluster/test_cluster_logs/test_cluster_worker_logs_order/data/Agent-groups_recv.yml

Lines 52 to 71 in 9a0d894

	# The checksum differs
	- log_id: log9
	parent: log2
	tag: 'The checksum of master (.*) and worker (.*) are different.'
	# The checksums differ and the agent_groups_mismatch_limit was not exceeded
	- log_id: log17
	parent: log9
	tag: 'Checksum comparison failed.*'
	- log_id: log18
	parent: log17
	tag: 'Finished in.*Updated.*chunks.*'
	# The checksums differ and the agent_groups_mismatch_limit was exceeded
	- log_id: log10
	parent: log9
	tag: 'Sent request to obtain all agent-groups information from the master node.*'
	- log_id: log11
	parent: log10
	tag: 'Finished in.*Updated.*chunks.*'

Note: In the future, if we have more cases like this one, with multiple conditions and repeated logs (Like log18 and log11), we can change the data structure to a Directed Acyclic Graph to have a node with multiple parents and avoid code repetition.

Results:

python3 -m pytest test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py --artifacts_path='/Users/framos/Desktop/artifacts'
============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.9.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/framos/Documents/Wazuh/Repositories/wazuh-qa/tests, configfile: pytest.ini
plugins: testinfra-5.0.0, html-3.1.1, metadata-2.0.4
collected 1 item

test_cluster/test_cluster_logs/test_cluster_worker_logs_order/test_cluster_worker_logs_order.py .                                                                          [100%]

========================================================================== 1 passed in 66.47s (0:01:06) ==========================================================================

[GGP1]

Update

• Changed the extension of the tests/reliability/test_cluster/test_cluster_logs/test_cluster_worker_logs_order/data/Agent-groups_recv.yml file to .yaml
• Changed the commit names to conform the convention

Moved state to final review since the first one was moved to 'on hold' just for the checks