Modifying the logic for calculating historyId

[anjerukare]

Hi 👋

I'm considering using Allure Report in a Java + TestNG autotesting project I'm working on. In its run configuration, there are several repeated tests in different places that have no difference in method's fully classified name or parameters, so they show up in the report as a single test with multiple retries. This is very confusing because, from my perspective, a test report is, among other things, a description of the test execution sequence.

The repeated tests, however, have different parentSuite and suite labels. I think they can be used to calculate the historyId and that would solve my problem.

Is it possible to modify the logic for calculating historyId? If so, how to do that?

[baev]

Could you please explain the issue in more detail? Am I right that you have multiple TestNG tests declared in testng.xml that reuse the same test methods?

[anjerukare]

Could you please explain the issue in more detail?

The use cases where I use repeated tests can be split into three categories:

1. repeated tests that change the state of something

Revert VM to snapshot example

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
    <test name="Revert the VM to a snapshot before all tests">
        <classes>
            <class name="RevertVmToSnapshot" />
        </classes>
    </test>
    <!-- This test also changes the VM state in some way -->
    <test name="Test something">
        <classes>
            <class name="Test1" />
        </classes>
    </test>
    <!-- This test also changes the VM state in some way -->
    <test name="Test something">
        <classes>
            <class name="Test2" />
        </classes>
    </test>
    <!-- We need to revert the VM to a snapshot because the rest of the tests
         need a "clean" VM  -->
    <test name="Revert the VM to snapshot before all tests">
        <classes>
            <class name="RevertVmToSnapshot" />
        </classes>
    </test>
    <!-- This test also changes the VM state in some way -->
    <test name="Test something">
        <classes>
            <class name="Test3" />
        </classes>
    </test>
</suite>

In this example the repeatable test (actually a test class) is RevertVmToSnapshot and it changes the state of the VM.

2. repeated tests that repeat small amount of times

Login block example

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
    <test name="Login with the wrong credentials">
        <classes>
            <class name="LoginWithWrongCredentials" />
        </classes>
    </test>
    <test name="Login with the wrong credentials again">
        <classes>
            <class name="LoginWithWrongCredentials" />
        </classes>
    </test>
    <!-- After two unsuccessful login attempts, the login is blocked -->
    <test name="Test login block">
        <classes>
            <class name="TestLoginBlock" />
        </classes>
    </test>
</suite>

In this example the repeatable test (actually a test class) is LoginWithWrongCredentials which has been repeated two times.

3. repeated tests that use the global state of something

Role model tests example

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
    <test name="Login as user">
        <parameter name="role" value="user" />
        <classes>
            <class name="Login" />
        </classes>
    </test>
    <test name="Test that the user can read the article">
        <classes>
            <class name="TestReadArticle" />
        </classes>
    </test>
    <test name="Logout">
        <classes>
            <class name="Logout" />
        </classes>
    </test>
    <test name="Login as admin">
        <parameter name="role" value="admin" />
        <classes>
            <class name="Login" />
        </classes>
    </test>
    <test name="Test that the admin can read the article">
        <classes>
            <class name="TestReadArticle" />
        </classes>
    </test>
    <test name="Test that the admin can remove the article">
        <classes>
            <class name="TestRemoveArticle" />
        </classes>
    </test>
</suite>

In this example the repeatable test (actually a test class) is TestReadArticle and the global state is the role: user or admin.

The examples I gave are quite abstract, but they capture the essence of the categories of my use cases. I hope this helps to understand the issue.

Am I right that you have multiple TestNG tests declared in testng.xml that reuse the same test methods?

Yes, you are right.

[baev]

The workaround is to add a parameter to testng.xml for tests that use the same method. Allure calculates historyId as md5(method + method arguments + parameters from testng.xml)

[anjerukare]

Is there any way to modify historyId via a listener or something?

Add a parameter in testng.xml to all repeated tests that are already described is quite problematic because of their number. Another problem is that this nuance of Allure Report should be kept in mind for describing future tests.

[marlet]

I guess you may try smth like

Allure.getLifecycle().updateTestCase( t -> {
      t.setHistoryId("")
});

[anjerukare]

Thanks for the reply!
I will try this solution and report back.