Workload benchmark performance tier 1

[juliamagan]

Description

This issue outlines the work required to create an initial workload performance test and conduct a preliminary benchmark for the Vulnerability Detector's performance after the changes done in 4.8.0. This will provide valuable insights into system behavior under load and identify areas for potential optimization. In addition, the possibility of deploying (and monitoring) a complete environment (including Indexer and Dashboard) will be added.

Functional Requirements

VD performance refactor

• As the QA team, we need to define workload scenarios representative of real-world usage to simulate and benchmark the Vulnerability Detector feature.
• As the QA team, we need to create new and update current benchmarking tools and procedures to measure the performance metrics accurately.
• As the Core 2 team, we need to validate the benchmark established for Vulnerability Detector after the changes done in 4.8.0.

Central components deployment

• As the QA team, we need to add the deployment of all central components.
• As the QA team, we need to update the deployment to be parametrizable (#nodes, indexer, dashboard).
• As the QA team, we need to create new and update current tool to obtain metrics of every component.
• As the QA team, we must ensure that the deployment uses cost-effective techniques.

Non-functional Requirements

• The workload should be as modular and scalable as possible to accommodate future performance testing needs.
• The chosen workload and benchmarking tools should be compatible with existing QA infrastructure.
• The benchmarking tools and procedures must be reliable and provide accurate performance metrics for effective analysis.

Implementation Restrictions

• The Indexer and Dashboard should be deployed with packages to allow the test of development branches.

Plan

The plan for implementing these objectives is divided into several stages, each focusing on a specific aspect of the improvements:

Stage 1: Workload scenario definition and tool preparation

• Objective: Define workload scenarios, prepare benchmarking tools, and address pipeline issues.

• Tasks:

  1. Define requirements for workload scenarios.
  2. Implement fixes and improvements in the pipeline reported previously.
  3. Develop or update benchmarking tools to accurately measure performance metrics of Vulnerability Detector after the changes introduced in 4.8.0.

• Issues:

  • Investigate the possibility of using the workload benchmarks metrics pipeline in agents mode to send syscollector events wazuh#20777
  • Modify the workload benchmarks metrics pipeline to send custom EPS in hybrid mode wazuh#20779
  • Adapt agent simulator in order to generate Vulnerability detector alerts #4781
  • Update Agent Simulator Syscollector format messages #4783
  • https://github.com/wazuh/wazuh-jenkins/issues/6182
  • https://github.com/wazuh/wazuh-jenkins/issues/6203
  • https://github.com/wazuh/wazuh-jenkins/issues/6204
  • Update the data visualization script to support API statistics files #5022
  • https://github.com/wazuh/wazuh-jenkins/issues/6304
  • Vulnerability Detector Discards Events #5222
  • https://github.com/wazuh/wazuh-jenkins/issues/6475
  • https://github.com/wazuh/wazuh-jenkins/issues/5388
  • https://github.com/wazuh/wazuh-jenkins/issues/6585

• Outcome: Workload scenarios defined, benchmarking tools prepared, and pipeline issues fixed.

Stage 2: Central components deployment enhancement

• Objective: Enhance deployment process for central components.

• Tasks:

  1. Update the pipeline to add the deployment of central components.
  2. Develop or update tools to monitor metrics of deployed components.

• Issues:

  • https://github.com/wazuh/wazuh-jenkins/issues/6080
  • https://github.com/wazuh/wazuh-jenkins/issues/6207
  • https://github.com/wazuh/wazuh-jenkins/issues/6208
  • https://github.com/wazuh/wazuh-jenkins/issues/6443
  • https://github.com/wazuh/wazuh-jenkins/issues/6444
  • https://github.com/wazuh/wazuh-jenkins/issues/6473
  • https://github.com/wazuh/wazuh-jenkins/issues/6474
  • https://github.com/wazuh/wazuh-jenkins/issues/6571
  • Unify monitoring metrics values for child processes with those of the main process #5479

• Outcome: Deployment process enhanced, monitoring tools ready, and pipeline improvements implemented.

Stage 3: Workload performance test execution

• Objective: Execute workload performance tests and ensure pipeline readiness.

• Tasks:

  1. Execute workload scenarios defined in Stage 1.
  2. Review and validate performance metrics collected during tests.
  3. Prepare a comprehensive report detailing performance test results and analysis.

• Issues:

  • Vulnerability detector refactor performance tests wazuh#20776
  • Benchmark vulnerability detector performance after the refactor wazuh#20781

• Outcome: Performance test results collected, and pipeline readiness confirmed.

Requirements Traceability Matrix (RTM)

Type	Responsible(s)	Description	Plan Part
Functional	QA Team	Define workload scenarios representative of real-world usage for the Vulnerability Detector feature.	Stage 1
Functional	QA and Framework Team	Fix known issues that preventing the proper operation of the workload benchmark performance	Stage 1
Functional	QA Team	Prepare benchmarking tools and procedures for workload testing.	Stage 1
Functional	QA Team	Add deployment of all central components.	Stage 2
Functional	QA Team	Create new and update current tools to obtain metrics of every component.	Stage 2
Functional	QA Team	Conduct workload benchmarking tests using predefined scenarios.	Stage 3
Functional	Core 2 Team	Validate the benchmark established for Vulnerability Detector after the changes done in 4.8.0.	Stage 3
Non-functional	QA Team	Report performance bottlenecks in the codebase	All Stages
Non-functional	QA Team	Ensure reliability and accuracy of benchmarking tools and procedures	All Stages
Implementation	QA Team	Adhere to existing development and testing frameworks	All Stages
Implementation	QA Team	Ensure compatibility of new benchmarking procedures with existing testing infrastructure and tools	All Stages

Approval

DRI name: @juliamagan
Director: @davidjiglesias
Objective: Workload benchmark performance tier 1

[santipadilla]

LGTM