Skip to content

Commit

Permalink
[Security Solution] Unskip get_prebuilt_rules_status test (elastic#…
Browse files Browse the repository at this point in the history
…192160)

**Addresses:** elastic#190952 and elastic#190960

## Summary

This PR unskips flaky test in `x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts`.

## Details

There are troubles with reproducing the flakiness detected in elastic#190952 and elastic#190960. Build logs inspections didn't reveal anything clear. I suspect that some failures happened while prebuilt rules were being deleted by query. Existing logic handles only conflicts `409` error while the other are ignored.

This PR adds extra error logging in `retryIfDeleteByQueryConflicts()` used in `deleteAllPrebuiltRuleAssets()`. And unskips `x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/management/trial_license_complete_tier/get_prebuilt_rules_status.ts`. This should help in further debugging.

## Flaky test runner

-  🟢  (100 runs) https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6864
  • Loading branch information
maximpn authored Sep 11, 2024
1 parent 3956ba0 commit b78ab47
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 21 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,7 @@ export default ({ getService }: FtrProviderContext): void => {
const es = getService('es');
const log = getService('log');

// Failing: See https://github.com/elastic/kibana/issues/190960
// Failing: See https://github.com/elastic/kibana/issues/190952
describe.skip('@ess @serverless @skipInServerlessMKI Prebuilt Rules status', () => {
describe('@ess @serverless @skipInServerlessMKI Prebuilt Rules status', () => {
describe('get_prebuilt_rules_status', () => {
beforeEach(async () => {
await deleteAllPrebuiltRuleAssets(es, log);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,34 +18,29 @@ const RETRY_DELAY = 200;
* Retry an Elasticsearch deleteByQuery operation if it runs into 409 Conflicts,
* up to a maximum number of attempts.
*/
export async function retryIfDeleteByQueryConflicts<T>(
export async function retryIfDeleteByQueryConflicts(
logger: ToolingLog,
name: string,
operation: () => Promise<DeleteByQueryResponse>,
retries: number = RETRY_ATTEMPTS,
retryDelay: number = RETRY_DELAY
): Promise<DeleteByQueryResponse> {
const operationResult = await operation();
if (!operationResult.failures || operationResult.failures?.length === 0) {
return operationResult;
}
for (let retriesLeft = retries; retriesLeft > 0; retriesLeft--) {
const operationResult = await operation();

for (const failure of operationResult.failures) {
if (failure.status === 409) {
// if no retries left, throw it
if (retries <= 0) {
logger.error(`${name} conflict, exceeded retries`);
throw new Error(`${name} conflict, exceeded retries`);
}

// Otherwise, delay a bit before retrying
logger.debug(`${name} conflict, retrying ...`);
await waitBeforeNextRetry(retryDelay);
return await retryIfDeleteByQueryConflicts(logger, name, operation, retries - 1);
if (!operationResult.failures || operationResult.failures?.length === 0) {
logger.info(`${name} finished successfully`);
return operationResult;
}

const failureCause = operationResult.failures.map((failure) => failure.cause).join(', ');

logger.warning(`Unable to delete by query ${name}. Caused by: "${failureCause}". Retrying ...`);

await waitBeforeNextRetry(retryDelay);
}

return operationResult;
throw new Error(`${name} failed, exceeded retries`);
}

async function waitBeforeNextRetry(retryDelay: number): Promise<void> {
Expand Down

0 comments on commit b78ab47

Please sign in to comment.