Flaky test at the SonarCloud step due to "no artifacts found"

[andybalaam]

Sometimes, when we try to kick off the SonarCloud step, the artifacts from the previous workflow are not yet present, and it fails.

Examples of failures caused by this:

• https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993311321
• https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993512503

[andybalaam]

The failing job is .github/workflows/sonarqube.yml, which is kicked when Tests finishes:

on:
    workflow_run:
        workflows: ["Tests"]
        types:
            - completed

[andybalaam]

sonarqube.yml in matrix-react-sdk just kicks off sonarcube.yml in matrix-js-sdk.

[luixxiul]

Do those PRs, with X-Needs-Percy label, keep failing after being added again until this issue is fixed?

[andybalaam]

Do those PRs, with X-Needs-Percy label, keep failing after being added again until this issue is fixed?

I think this is flaky, so they should pass sometimes.

[t3chguy]

So Sonar shouldn't be running at all in the example of https://github.com/matrix-org/matrix-react-sdk/actions/runs/4946075148/jobs/8843676973 due to us not caring about covg in the merge queue. We look to be missing a line like https://github.com/matrix-org/matrix-js-sdk/blob/develop/.github/workflows/sonarqube.yml#L39 - maybe we can apply it to https://github.com/matrix-org/matrix-react-sdk/blob/develop/.github/workflows/sonarqube.yml and thus react-sdk runs will respect it too

[andybalaam]

Created matrix-org/matrix-react-sdk#10863 to address @t3chguy 's comment (thanks!). Now looking into whether this is a problem more widely than the merge queue.

[andybalaam]

I can't find any recent examples of this failure outside of the merge queue, so I'm going to close this until I see one. The above-mentioned fix should remove the vast majority of the failures.

[andybalaam]

Here is a non-merge-queue example of this that happened today: https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993512503

Re-opening.

[andybalaam]

I re-ran https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993512503 about an hour later, and it failed with the same error. This seems like enough time for any race condition to have fixed itself, and not enough time for artifacts to be cleaned up, but that is just based on my feelings about it, not facts.

[t3chguy]

@andybalaam the Test job for that one got cancelled - https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993450997

[andybalaam]

@andybalaam the Test job for that one got cancelled - https://github.com/matrix-org/matrix-js-sdk/actions/runs/4993450997

Hmm... so maybe we can prevent the downstream tests from running when it's been cancelled?

[t3chguy]

Downstream tests don't run unless local tests pass, the run you linked isn't related to downstream testing. The PR I made above will skip js-sdk's sonar prepare stage if the tests job failed. The other layers don't need this as they don't have a similar Prepare stage (js-sdk needs it to merge coverage results)

[andybalaam]

Ah, I thought you meant that the failure was caused by the Test job being cancelled?

[t3chguy]

It was, but normal tests, not downstream tests. The flow is

Test -> Sonar -> Sonar reusable

in all layers other than js-sdk the Sonar job just calls upon Sonar reusable. js-sdk does a prepare stage, Sonar reusable checks the calling job (Test) was successful, the prepare stage did not.