Add Vulnerability Detector System End-to-End tests

[Deblintrake09]

Target version	Related issue	Related PR/dev branch
4.8.0 Alpha 1	#4369

Description

This issue aims to create the End-to-end test cases defined in #4531. This test cases use the basic cases defined in #4590

Proposed test cases

• E2E-VD-3: Installation of a vulnerable package
• E2E-VD-4: Updating a vulnerable package that remains vulnerable to the same CVE
• E2E-VD-5: Updating a vulnerable package that becomes vulnerable to another
• E2E-VD-6: Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one
• E2E-VD-7: Updating a vulnerable package that ceases to be vulnerable
• E2E-VD-8: Deleting a vulnerable package
• E2E-VD-9: Installation of a non-vulnerable package
• E2E-VD-10: Updating a non-vulnerable package that becomes vulnerable
• E2E-VD-11: Updating a non-vulnerable package that remains non-vulnerable
• E2E-VD-12: Deleting a non-vulnerable package
• E2E-VD-13: Updating the vulnerability feed with a new applicable vulnerability
• E2E-VD-14: Updating the vulnerability feed with a new non-applicable vulnerability
• E2E-VD-15: Deleting the states index
• E2E-VD-16: Updating the vulnerability feed removes a CVE and causes a package to cease to be vulnerable
• Adapt E2E-VD1 and E2E-VD2 to the refactored VD module
• E2E-VD-17: Installation of a vulnerable package when agent is offline
• E2E-VD-18: Enable vulnerability detector when some agents are already registered
• E2E-VD-19: Change agents' manager and install a vulnerable package

Considerations

• This tests should use the packages proposed in Research vulnerable packages for usage in VDT System tests #4529

[Rebits]

Examining testing scenarios through the provided design at #4590 (comment). Certain challenges have surfaced concerning the monitoring of alerts across arm architectures. We are actively addressing these issues and working towards resolution.

In addition we are working on including methods for handling indices

[Rebits]

Refactor schema in order to check generated vulnerabilities in different ways.
Done in b9f1101

[Rebits]

Marked On Hold until #4703 is stable

[Rebits]

Continue with the development using #4703 as base development

[Rebits]

• Fixed bug in remove_package for windows endpoint
• Refactor test
• Implemented VD 3, 4, 5 and 6

It has been detected issues in the Vulnerability detector module regarding the detection of some packages. Further research is required

[Rebits]

Adjusted the ETA to align with @wazuh/data-pirates at wazuh/wazuh#14153 development. Finalizing test modifications to seamlessly integrate with the ongoing development.

[Rebits]

• Adapted tests to match changes of VD refactor Vulnerability Detector refactor wazuh#14153
  • Change configurations in order to include vulnerability-detection and indexer
  • Deprecate Vuln Endpoints and include methods to ensure state index is not empty
  • Minor changes in callback for waiters in order to adapt to content gathering stage
  • Adapt the rest of tests cases to the common yaml structure
  • Adapt final methods for alerts checking in the index state

In addition it is necessary to research some not expected behaviors:

• Syscollector seems to missed packages if these are installed in certain time gap in during the scan
• Syscollector seems to ignore grafana package for ubuntu system. This should be confirmed for the rest of the systems
• Vulnerability Detector seems not work after the refactor. This seems to pull correctly the content, but after that stage this does not work.

[Rebits]

• Uploaded packages into a common bucket
• Review detect alerts method
• Refactor detect alert method in order to take into accound the timestamp
• Get feedback from @wazuh/data-pirates regarding the VD functionality in the environment
  • @Dwordcito https://github.com/wazuh/wazuh/blob/dev-14153-vulndet-refactor/src/wazuh_modules/vulnerability_scanner/indexer/template/legacy-template.json give us the strucutre of the indices state. We can proceed with the comparision methods regarding this template
• Implement indices state validator methods
• Implement the rest of the YAML cases

[Rebits]

• Implement the rest of the YAML cases

Marked as blocked until development is over

[Rebits]

Adjusted ETA to December 20, 2023, with approval from @davidjiglesias.
This modification is prompted by a development delay now anticipated for December 18: wazuh/wazuh#14153

[Rebits]

Starting the migration of tests to accommodate the recent of the Vulnerability Detector.

---

Generated packages

https://ci.wazuh.info/job/Packages_builder_tier/3288/console

---

Meeting with @Dwordcito @davidjiglesias about current status of the VD refactor

[Rebits]

We have successfully conducted the first Proof of Concept (POC) for the testing environment, leveraging the newly refactored Vulnerability Detector. Unfortunately, the analysis has revealed several critical errors that currently hinder the progression of testing.

For further details and a comprehensive overview of the identified issues, please refer to the following GitHub link: wazuh/wazuh#20785.

[Rebits]

Created new packages with latests changes in Vulnerability detector

Build: https://ci.wazuh.info/job/Packages_builder_tier/3347

However, no indices of alert was created

root@ip-172-31-9-35:/home/qa# curl -k -u admin:changeme https://172.31.9.35:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .opensearch-observability   eCUPM1U6RXiXnioci6-_8g   1   0          0            0       208b           208b
green  open   .plugins-ml-config          RATIwdbhSuCvw0AcMDQs1A   1   0          1            0      3.9kb          3.9kb
green  open   wazuh-alerts-4.x-2023.12.28 gplYSwV1T1SVg35Hy2xicg   3   0       2351            0      3.4mb          3.4mb
green  open   .opendistro_security        MmlFVltMQ22KxT2nJLNg-A   1   0         10            0     64.7kb         64.7kb

In addition no vulnerability alerts were generated.
It is necessary to achieve feedback from developer team in order to continue with tests

[Rebits]

New packages were created with wazuh/wazuh@937acfc changes. However, a segmentation fault has been identified in the current development

• Segmentation fault in Vulnerability Detector Refactor wazuh#21176

Testing cannot proceed until a stable version is delivered

[Rebits]

wazuh/wazuh#21176 seems to be solved. We can proceed with the development of the tests

[Rebits]

Currently working on tests adaptation

[Rebits]

• Improve debug messages
• Include methods to handle agent
• Include in setup fixture agent removal to avoid known bug Vulnerability Detector Refactor: No vulnerability alerts were triggered for agents previously registered wazuh#21185
• Fixed check_consistency method

Detected bug in decoded alerts regarding the package name. Currently researching

• Fixed timeouts and waiters method to adjust the development

[Rebits]

• Refactor tests in order to allow continue even if one or multiple agents or stages failed. This change has been motivated due to the high time costing of the deploy/provision, to all the possible information in one test iteration. In addition this change was necessary in order to check the behaviour of all the test stages
• Refactor reporting. This was necessary in order to get readable information regarding the environment. Without theses changes developer will have serious difficulties troubleshooting the environment
• Improve performance of alert gathering methods.
• It should be necessary refactor basic tests due to the complexity that they have been get
• ETA for first POC for the first basic tests planned for 15/01/2024

Changes in 52b70be

[Rebits]

• Working on the development of tools for collecting evidence, aiming to amass comprehensive information that empowers our developer team in effectively debugging the environment.

[Rebits]

• Stabilize initial scans tests
• Created evidence collection tool
• Improve report to allow readability
• Fix specific package cases tests tools

Changes can be found in: f5c96e2

Currently working on:

• Improve readability to specific package tests
• Improve logging for specific package tests
• Research possible issues with the product in regard to the test result
• Research possible consistency error in the product
• Improve error resistance to all tests and library to avoid aborting the suite in case of not expected data format
• Uncomment the rest of tests

[Rebits]

• Recreate packages in order to avoid reported bug that does not allow continue with testing
• Refactored install and remove functions
• Refactored packages metadata

[Rebits]

• Refactored code
• Create new update_package function
• Replace some packages in order to fulfill with the repository
• Detected vulnerable packages not detected by the scan

Pending task

• Improve logging messages
• Include error resistance
• Uncomment rest of cases
• Testing in real environment

[Rebits]

• Detected that .pkg are not detected by syscollector. This will make all macOS checks fails
• Refactored update method
• Detected failure in windows package uninstallation. Work in progress
• Fix evidence gathering for consistency check tests

[Rebits]

• Fixed Windows VLC uninstall process
• Fixed macOS installation method
• Included all tests cases. Ready to perform a PoC. Planned 31/01/2024

Minor issues to fix:

• Include fixture session to check connection to all hosts
• Fix macOS uninstall program
• Fix function to check if any vulnerability were triggered (non vulnerable package installation case)
• Update documentation in all methods and modules
• Include logic to avoid not handle bad installation
• Improve logic in case host OS or architecture provided is not included in the supported tests
• Improve logging messages
• Refactor libraries to improve performance

[davidjiglesias]

LGTM