flaky-test-extractor-maven-plugin

Maven plugin that scans the test output of Surefire for flaky tests and writes test output files for just the flaky tests. These can then be displayed in reports or in CI tools

Why does this exist?

Flaky tests are tricky to diagnose and fix. Surefire has an option to rerun failed tests. This will detect flaky tests, and indeed information about flaky tests is available in the XML reports generated by Surefire. However, in this mode surefire will not fail the build and the flaky test runs will not appear in the test reports generated by either Surefire or Jenkins JUnit result publisher. Which is not ideal if the goal is to eliminate flaky tests.

This plugin scans the XML file for flaky tests results and then generates additional XML files just for the flaky tests. In these new XML files the flaky tests will be reported as test failures. Thus they can be picked up by downstream tools.

What does it do exactly?

• It scans a folder for Surefire test run XML files
• If it finds flaky test results in a file TEST-com.example.Test.xml
• It creates a new file TEST-com.example.Test-FLAKY.xml
• This file will contain only the flaky tests and flaky errors of the original file
• They will be reported as test failures
• The test methods will also get the suffix ... (FlakyTest) so that they stand out in test reports
• The plugin will also fail the build if flaky tests were found (this can be configured)

How to use the plugin?

<plugin>
  <groupId>io.zeebe</groupId>
  <artifactId>flaky-test-extractor-maven-plugin</artifactId>
  <version>x.x.x</version>
  <executions>
    <execution>
      <phase>post-integration-test</phase>
      <goals>
        <goal>extract-flaky-tests</goal>
      </goals>
    </execution>
  </executions>
</plugin>

How can I configure this plugin?

There are three configuration options:

• reportDir (defaultValue ${project.build.directory}/surefire-reports) - this is the directory that will be scanned for XML report file. It is also the directory to which new XML files will be written
• failBuild (defaultValue true) - flag to make the plugin fail the build if flaky test were discovered
• skipFlakyTestExtractor (defaultValue false) - flag to skip this plugin

What are known shortcomings?

• JUnit distinguishes between errors and failures. Likewise, it distinguishes between flaky errors and flaky failures. This plugin will put both in the same bucket and report them as failures in the generated XML
• The group attribute is not carried over when generating new XML files
• XML files generated by this plugin will not contain the environment variables section
• The elapsed time is simply copied from the original file. It is not adjusted to reflect only the time consumed by the flaky tests

How to release this project?

Manually trigger a release workflow run with the desired parameters for current and next version.