Add support for subresults (experimental flavor) to junit report plugin

[seberm]

This PR adds a subresults junit report plugin flavor to show the tmt.result.SubResult instances within the tmt.Result. It tries to map the tmt.Result to <testsuite> and tmt.result.SubResult to <testcase> respectively.

Discussion from 10.9. 2024:

This flavor will be probably used by testing-farm to generate the results-junit.xml in the future.

There are two results files in Testing Farm:

• results.xml (converted from tmt’s results.yaml)
• results-junit.xml (in the future should be generated by tmt)

For more info see:

• Add support for test phases #2826 (comment)
• JUnit examples

Related to:

• Remove support for junit-xml, use Jinja2 instead #3150
• Add support for test phases #2826

TODO:

• Test with Use Jinja2 and polarion JUnit flavor to generate XUnit in polarion report plugin #3166
• Add tests to cover the subresults
• Maybe use classname to define a parent test name and name for a subresult test name - Add support for test phases #2826 (comment)

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[seberm]

@kkaarreell Hello,
I am studying various formats of JUnit. In this PR, I created a new subresults JUnit flavor with nested <testsuite> elements. Unfortunately, it seems that the <testsuite> elements cannot contain the same elements as <testcase> - especially tags like <error>, <failure>, and <skipped>. At least according to these specs/examples:

• https://github.com/testmoapp/junitxml/blob/main/examples/junit-complete.xml
• https://github.com/junit-team/junit5/blob/main/platform-tests/src/test/resources/jenkins-junit.xsd

This basically means if there are failed/skipped/... tests that do not contain any subresults, we will be unable to export their result. This probably means that all the <testsuite> elements should contain <testcase> elements and we must somehow create a test hierarchy using class attributes.

The example of the currently generated JUnit subresults flavor:

<?xml version='1.0' encoding='utf-8'?>
<testsuites disabled="0" errors="0" failures="1" tests="3" time="0">
  <testsuite name="/tests/call-dmesg" disabled="0" errors="0" failures="0" skipped="0" tests="0" time="0" timestamp="2024-08-13T12:00:22.022008+00:00">
  </testsuite>
  <testsuite name="/tests/mine-rep-multi" disabled="0" errors="1" failures="1" skipped="0" tests="3" time="0" timestamp="2024-08-13T12:00:22.881823+00:00">
    <testcase name="/tests/mine-rep-multi/test/good">
    </testcase>
    <testcase name="/tests/mine-rep-multi/test/fail">
      <failure type="failure" message="fail"/>
    </testcase>
    <testcase name="/tests/mine-rep-multi/test/weird">
      <error type="error" message="warn"/>
    </testcase>
  </testsuite>
  <testsuite name="/tests/mine-rep-pass" disabled="0" errors="0" failures="0" skipped="0" tests="0" time="0" timestamp="2024-08-13T12:00:23.752135+00:00">
  </testsuite>
</testsuites>

Any hints / ideas?

[happz]

Maybe we could always include at least one testcase, representing the test itself?

[kkaarreell]

Hi @seberm ,
I think we have been discussing this topic in one of the mtgs. You are right that jUnit doesn't allow nested testsuites and therefore jUnit won't be able to implement this feature. There was a proposal to identify which parts of the pipeline (which apps, scripts etc.) would be using this "per-phase" reporting and based on this findings implement changes in tmt. Maybe we would end up producing brand new output file for this.

[seberm]

Maybe we could always include at least one testcase, representing the test itself?

Yeah, I had the same idea - I did the modifications. The output is now:

<?xml version='1.0' encoding='utf-8'?>
<testsuites disabled="0" errors="0" failures="1" tests="3" time="0">
  <testsuite name="/tests/call-dmesg" disabled="0" errors="0" failures="0" skipped="0" tests="0" time="0" timestamp="2024-08-13T12:00:22.022008+00:00">
      <testcase name="/tests/call-dmesg">
      </testcase>
  </testsuite>

  <testsuite name="/tests/mine-rep-multi" disabled="0" errors="1" failures="1" skipped="0" tests="3" time="0" timestamp="2024-08-13T12:00:22.881823+00:00">
    <testcase name="/tests/mine-rep-multi">
      <failure type="failure" message="fail"/>
    </testcase>
    <testcase name="/tests/mine-rep-multi/test/good">
    </testcase>
    <testcase name="/tests/mine-rep-multi/test/fail">
      <failure type="failure" message="fail"/>
    </testcase>
    <testcase name="/tests/mine-rep-multi/test/weird">
      <error type="error" message="warn"/>
    </testcase>
  </testsuite>

  <testsuite name="/tests/mine-rep-pass" disabled="0" errors="0" failures="0" skipped="0" tests="0" time="0" timestamp="2024-08-13T12:00:23.752135+00:00">
    <testcase name="/tests/mine-rep-pass">
    </testcase>
  </testsuite>
</testsuites>

And now we need to test against the various systems/scripts which would be using this flavor. E.g. ReportPortal.

[kkaarreell]

I am sorry but this seems like introducing yet another jUnit format. What tools are expected to work such a XML file? Why not introducing new report plugin for it?

[seberm]

I am sorry but this seems like introducing yet another jUnit format. What tools are expected to work such a XML file? Why not introducing new report plugin for it?

You’re basically right. It is another JUnit format. That’s why it’s implemented as an optional template/flavor (see #3150 for more info).

The challenge is that different systems (GitLab, Jenkins, ReportPortal, Polarion) support slightly different variants of JUnit formats. While these formats are largely similar, there are small differences. The current JUnit report plugin only supports a single, non-customizable JUnit "default" flavor. The changes in #3150 introduce support for custom flavors, allowing users to provide their own JUnit templates.

The motivation for this PR is that we plan to add support for tmt subresults and we need to have (based on requirements from Jira ticket) the ability to export subresults into the JUnit format.

I don’t think we need to merge this and officially support the format, but we should at least ensure that generating this kind of JUnit file is possible. I'll reach out to the people from the Jira ticket and try to clarify their specific needs and ask them for testing.