Every retried test results are added to log.json file, instead of the final status of the execution

[mmahmutyilmaz]

Current behavior

If we set a retry value on Cypress config and the test case fails, every test execution is added to JSON report.

Desired behavior

Only the last execution should be included in the JSON report. Please see Cypress Retry documentation. Cypress is also adding the final status to its report at the end of the execution. Especially, it is important to skip failures, which were passed after the following reruns.

Test code to reproduce

Steps to reproduce:

1. Run a feature file which includes some test cases to be failed.
2. Open log.json file
3. Check the failed test cases (Better to search for the "id" in the json)
4. You will see the failed test cases repeat. If we have 5 attempts, then we have the same test case 5 times in log.json

See the shortened log.json file (Tes execution with 1 retry):

[
.
.

  {
    "description": "",
    "id": "1-menu-1--scrolling--ta",
    "keyword": "Scenario",
    "line": 7,
    "name": "1-Menu 1- Scrolling -TA",
    "steps": [
      {
        "keyword": "Given ",
        "line": 8,
        .
                .
                        .

  {
    "description": "",
    "id": "1-menu-1--scrolling--ta",
    "keyword": "Scenario",
    "line": 7,
    "name": "1- Menu 1- Scrolling -TA",
    .
            .
                    .

]

Versions

• Cypress version: 10.7.0
• Preprocessor version:12.2.0

[badeball]

This is a known issue with the json formatter, reported here.

[WTK]

Since no one seem eager to fix the bug on the cucumber-json-formatter, could the work around be to not emit messages that are about failed attempt? In our use case we wouldn't care if something was retried as long as it eventually passed.

[idasilva1]

I'm facing the same issue. Any work around yet?

[badeball]

Since no one seem eager to fix the bug on the cucumber-json-formatter, could the work around be to not emit messages that are about failed attempt?

Changing messages is out of the questions, as they're just as much a report as the JSON report.

The only acceptable workaround in my eyes is to rewrite it myself, ref. #870.

[mmahmutyilmaz]

I have written a primitive, workaround function. It should be called before reporting. I have put it at the top of my cucumber-html-reporter.js.

function deleteDuplicateTests() {
let cucumberLogs = JSON.parse(fs.readFileSync('./cypress/reports/json/log.json'));
for (const feature of cucumberLogs) {
const index = cucumberLogs.indexOf(feature);
const counts = _.countBy(feature.elements, 'id')
let duplicates = _.filter(feature.elements, x => counts[x.id] > 1)
if (duplicates.length !== 0) {
duplicates.forEach(duplicate=>{
const sameDuplicateTests=duplicates.filter(n=>n.id===duplicate.id);
for (let i = 0; i <sameDuplicateTests.length-1 ; i++) {
_.remove(cucumberLogs[index].elements, (n) => {
return sameDuplicateTests[i] === n
});
}
})
}
}
fs.writeFileSync('./cypress/reports/json/log.json', JSON.stringify(cucumberLogs));
}

[WTK]

Thanks for sharing