[Fleet] Unskip test suite for agentless and replace waitForNextUpdate with waitFor

[criamico]

Closes #189038
Closes #192126

Summary

Attempting again to fix failures related to waitForNextUpdate. I merged #197951 but it failed again right after merging it, this time I'm removing altogether waitForNextUpdate from the whole file and using waitFor or rerender as needed.

NOTE: This test never fails locally, I tried it many times and it never occurs. So it could be some setting in the c.i. that makes it different from the local test. There's also currently no way to use the flaky test runner.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

NOTE: This test never fails locally, I tried it many times and it never occurs. So it could be some setting in the c.i. that makes it different from the local test. There's also currently no way to use the flaky test runner.

The only way I know of to get this to run a bunch of times is to wrap the whole test suite in a for loop, but even doing that I've never seen a failure come out of it 😞

[criamico]

@elasticmachine merge upstream

[criamico]

The only way I know of to get this to run a bunch of times is to wrap the whole test suite in a for loop, but even doing that I've never seen a failure come out of it

@kpollich I suspect that the ci env is in some way different from the local env so it only fails there. However, even the changes merged yesterday to the testRenderer function didn't help :(

[opauloh]

/ci

[opauloh]

@criamico / @kpollich Since our team introduced most of these tests, would you like me to take over these flaky issues as part of the new Reliability Epic?

There are tasks to migrate waitForNextUpdate to waitFor , as well for identifying and documenting flakiness.

I can also see we actually missed adding await to a couple of other places that use waitFor as well, leading the assertions to be executed earlier than they should without actually waiting for the waitFor method. It probably fails on CI but not locally due to the slow execution time that can occasionally occur on CI.

Fixing those will also help with the React 18 preparation PR, where some tests are falling due to the missing await.

Also, since it's very easy to miss the addition of await, I'm also considering writing an ES Lint rule for our plugin to prevent missing adding await when using waitFor.

[criamico]

@elasticmachine merge upstream

[criamico]

I'm also considering writing an ES Lint rule for our plugin to prevent missing adding await when using waitFor.

@opauloh I fixed it the missing await, but it would be great to have a rule to enforce it, it's pretty easy to miss it.

Since our team introduced most of these tests, would you like me to take over these flaky issues as part of the new Reliability Epic?

Let me know if you want me to keep this PR open or it's better to proceed with your epic. The only thing that concerns me is that keeping too many tests skipped might lead to not catching some bugs early, especially on serverless/agentless. @kpollich what do you think?

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 3507556

Failed CI Steps

• Jest Tests #15
• Jest Tests #15
• FTR Configs #78

Test Failures

• [job] [logs] FTR Configs #78 / alerting api integration security and spaces enabled - Group 4 Alerts alerts alerts connector adapters should use connector adapters correctly on system actions
• [job] [logs] Jest Tests #15 / useSetupTechnology should update agentless policy name to match integration name if agentless is enabled
• [job] [logs] Jest Tests #15 / useSetupTechnology should update agentless policy name to match integration name if agentless is enabled

Metrics [docs]

✅ unchanged

History

• 💚 Build #246851 succeeded 17d77bd
• 💛 Build #246695 was flaky 4b3a224
• 💔 Build #246561 failed 4b3a224

cc @criamico

[kpollich]

@kpollich what do you think?

Since @opauloh's team is already tracking work here, it might make sense to have them take this over the finish line. @criamico do you feel like we have a path forward to get this PR green and land it, or is scope of work needed here larger? It seems like there's a lot of moving parts in the reliability epic Paulo linked, and I am not sure if we need to do a lot of that foundational work to fix the root cause of the flakiness here rather than just patching smaller things to get this PR green.

[criamico]

Since @opauloh's team is already tracking work here, it might make sense to have them take this over the finish line.

I think it would be better if @opauloh team takes over this work, since they wrote most of the tests and intend to take care of the larger scope of it. I'll close this PR in favor of their solution.

[opauloh]

Thanks! I already updated the relevant tickets to include the agentless scope of Fleet!

• https://github.com/elastic/security-team/issues/10979
• Update Deprecated waitForNextUpdate Hook to waitFor #198186

The only thing that concerns me is that keeping too many tests skipped might lead to not catching some bugs early, especially on serverless/agentless.

Yes, great concern, those tasks are top priority for this sprint, so we ensure they are not skipped for longer but also fixed in the large scope.