Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cloud Security] Fix flaky metering tests #197950

Merged
merged 3 commits into from
Oct 29, 2024

Conversation

CohenIdo
Copy link
Contributor

Solves:

Summary

The error logs indicate that while we index N documents, the metering API intercepts only N-X documents. Previously, to address a similar issue, we switched to bulk indexing to ensure all test-related documents were indexed before tests began.

Although this adjustment resolved the problem in some flaky test runs, we still encountered intermittent failures in one of the hundreds runs.

Upon further investigation, I found that even with bulk indexing, there are rare cases where tests start before all relevant documents have finished indexing. I confirmed this by increasing the number of documents from 5 to 150, which made the error appear more consistently.

To address this, I updated the test logic to retry the expect assertion, reducing the likelihood of these occasional failures.

@CohenIdo CohenIdo added release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related labels Oct 28, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7276

[✅] x-pack/test_serverless/api_integration/test_suites/security/config.ts: 50/50 tests passed.

see run history

@CohenIdo CohenIdo added the backport:skip This commit does not require backporting label Oct 28, 2024
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

@CohenIdo CohenIdo merged commit 3c7268d into elastic:main Oct 29, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment