Add error handling/retry logic for search source alert tests

[lukasolson]

Summary

Resolves #193842.

Adds error handling & retry logic for search source alerts that are causing failures on MKI.

Checklist

• Flaky Test Runner was used on any tests changed

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7157

[✅] x-pack/test/functional_with_es_ssl/apps/discover_ml_uptime/config.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/observability/common_configs/config.group6.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts: 25/25 tests passed.
[✅] x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts: 25/25 tests passed.

see run history

[lukasolson]

@wayneseymour Do you mind taking a quick look at this and see if it's along the lines of what you were thinking?

[wayneseymour]

@lukasolson I would strongly recommend running the serverless tests through MKI, from your local

One major reason is to ensure the default timeout durations encoded in x-pack/test_serverless/functional/test_suites/common/discover_ml_uptime/discover/search_source_alert.ts, actually work for real serverless as well.
PR CI only runs fake-serverless.
Running against real serverless (mki) will give you useful feedback.

• network access is typically slower, etc

Ran From my Local

Ran the stateful test, no issues.
Ran the serverless test against a real serverless oblt project (mki), I am seeing errors:

└-: Discover alerting
  └-> "before all" hook: beforeTestSuite.trigger in "Discover alerting"
  └-> "before all" hook in "Discover alerting"
  └-: Search source Alert
    └-> "before all" hook: beforeTestSuite.trigger for "should create an alert when there is no data view"
    └-> "before all" hook for "should create an alert when there is no data view"
            │ERROR timeout of 60000ms exceeded
            │      Waiting 15000 ms before the next attempt
    └-> should create an alert when there is no data view
      └-> "before each" hook: global before each for "should create an alert when there is no data view"
      └- ✓ pass  (23.1s)
    └-> should show time field validation error
      └-> "before each" hook: global before each for "should show time field validation error"
      └- ✓ pass  (14.8s)
    └-> should navigate to alert results via view in app link
      └-> "before each" hook: global before each for "should navigate to alert results via view in app link"
      └- ✓ pass  (40.0s)
    └-> should navigate to alert results via link provided in notification
      └-> "before each" hook: global before each for "should navigate to alert results via link provided in notification"
      └- ✓ pass  (37.8s)
    └-> should display prev rule state after params update on clicking prev generated link
      └-> "before each" hook: global before each for "should display prev rule state after params update on clicking prev generated link"
      └- ✖ fail: Discover alerting Search source Alert should display prev rule state after params update on clicking prev generated link
      │      Error: retry.try reached timeout 120000 ms
      │ TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="saveEditedRuleButton"])
      │ Wait timed out after 10054ms
      │     at /Users/trezworkbox/dev/main/node_modules/selenium-webdriver/lib/webdriver.js:923:22
      │     at processTicksAndRejections (node:internal/process/task_queues:95:5)
      │       at onFailure (retry_for_success.ts:18:9)
      │       at retryForSuccess (retry_for_success.ts:85:7)
      │       at Proxy.try (retry.ts:52:12)
      │       at Proxy.clickByCssSelector (find.ts:420:5)
      │       at Proxy.click (test_subjects.ts:169:5)
      │       at Context.<anonymous> (search_source_alert.ts:509:7)
      │       at Object.apply (wrap_function.js:74:16)
      │
      │
    └-> should display actual state after rule params update on clicking viewInApp link
      └-> "before each" hook: global before each for "should display actual state after rule params update on clicking viewInApp link"
      └- ✖ fail: Discover alerting Search source Alert should display actual state after rule params update on clicking viewInApp link
      │      Error: expected '' to equal 'message:msg-1'
      │       at Assertion.assert (expect.js:100:11)
      │       at Assertion.apply (expect.js:227:8)
      │       at Function.equal (expect.js:531:15)
      │       at checkUpdatedRuleParamsState (search_source_alert.ts:343:31)
      │       at processTicksAndRejections (node:internal/process/task_queues:95:5)
      │       at Context.<anonymous> (search_source_alert.ts:522:7)
      │       at Object.apply (wrap_function.js:74:16)
      │
      │
    └-> should display prev data view state after update on clicking prev generated link
      └-> "before each" hook: global before each for "should display prev data view state after update on clicking prev generated link"
     
     ....
     

I recommend running it against search and security projects as well.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kibanamachine]

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7454

[✅] x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts: 25/25 tests passed.

see run history

[davismcphee]

Code changes LGTM, but it looks like a different test is failing now in this PR. Possibly related to #198987.

[lukasolson]

Code changes LGTM, but it looks like a different test is failing now in this PR. Possibly related to #198987.

I've skipped this test for now, looks to be an actual bug. I've filed an issue here: #203045

[davismcphee]

LGTM, thanks for fixing them and adding the issue!

[davismcphee]

I'm guessing the conditionals for buttons and inputs are for O11y projects? We should probably consider splitting these out since they could diverge more, but let's get them unskipped for now.

[lukasolson]

That's right, and yes I totally agree. There was already conditionals like these in these tests but I agree, it's probably time now to split these.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: dd053d4

Metrics [docs]

✅ unchanged

History

• 💛 Build #257671 was flaky 3cfd4ff
• 💔 Build #257215 failed 96fc8ba
• 💔 Build #254715 failed 2eddc47
• 💔 Build #253375 failed 4a09619
• 💔 Build #252959 failed f38bb92
• 💚 Build #243226 succeeded 62a04a2

cc @lukasolson

[davismcphee]

@lukasolson Did we only want this merged for 9.0 or should it also be backported to an 8.x branch?

[lukasolson]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation