[Proposal] Automate detection and quarantine of flakey tests

[dblock]

Is your feature request related to a problem? Please describe.

The amount of flakey tests (#1715) is growing and gradle checks fail roughly 50% of the time. This impedes velocity, but also reduces the trust in the tests.

Describe the solution you'd like

1. Label flakey tests, @Ignore them, and link to a GitHub issue.
2. Automatically detect repeated flakey test failures and link them to a GitHub issue. Automatically keep count of flakey tests.
3. Remove flakey tests that have been flakey for over a period of time, e.g. 2 weeks, via an auto-PR.

Describe alternatives you've considered

Fix all flakey tests one-by-one.

[reta]

@dblock we could also try to use Gradle's flaky test mitigation tooling, wdyt?

[1] https://blog.gradle.org/gradle-flaky-test-retry-plugin

[saratvemulapalli]

@dblock love the idea. Are you picking this up or do you need help?

[dblock]

@dblock love the idea. Are you picking this up or do you need help?

I am not picking it up, I stole the idea from a discussion with another engineer that shall remain nameless for now (will give them credit sometime in the future), so go for it!

[dblock]

@dblock we could also try to use Gradle's flaky test mitigation tooling, wdyt?

[1] https://blog.gradle.org/gradle-flaky-test-retry-plugin

#2547