test: parallelizing cypress tests to speed up pipeline #2155
Conversation
Parallelizing cypress tests to speed up pipeline
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site configuration. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Welcome to AsyncAPI. Thanks a lot for creating your first pull request. Please check out our contributors guide useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.
test: parallelizing cypress tests to speed up pipeline
|
@reachaadrika Kindly review this PR |
AtharvChandratre
requested review from
derberg,
akshatnema,
magicmatatjahu,
Mayaleeeee and
asyncapi-bot-eve
as code owners
|
⚡️ Lighthouse report for the changes in this PR:
Lighthouse ran on https://deploy-preview-2155--asyncapi-website.netlify.app/ |
package.json
Outdated
| "fs": "^0.0.1-security", | ||
| "path": "^0.12.7" |
There was a problem hiding this comment.
| "fs": "^0.0.1-security", | |
| "path": "^0.12.7" |
These packages are included by default in node js. Kindly don't add it in package.json.
There was a problem hiding this comment.
Thanks, changed
There was a problem hiding this comment.
Kindly share the workflow test links you did in fork repository.
@akshatnema - https://github.com/AtharvChandratre/website/actions/runs/6218086013
There was a problem hiding this comment.
You haven't pushed your changes 😃
There was a problem hiding this comment.
Yep, just did! Was looking into why build-newsroom-videos.cy.js flakes sometimes. Seems to be because it's calling the YouTube API, which doesn't respond in time intermittently. I can create a separate issue to look into this later.
|
Also, I see that each container is taking a separate workflow/jobs inside the checks, can't we run all containers in single job only? |
|
Hi @AtharvChandratre , you can instead of mentioning the entire command in the workflow yml , can also create a |
.github/workflows/cypress-tests.yml
Outdated
| - name: Install dependencies | ||
| run: npm install | ||
|
|
||
| - name: Cypress Tests are running | ||
| run : node ./scripts/index.js && npm run test | ||
| uses: nick-fields/retry@v2 |
There was a problem hiding this comment.
how often timeouts happen?
I'm a bit afraid that default rerun for tests, will take over GH infra too much.
also I'm pretty sure cypress have native support for retry and timeout
There was a problem hiding this comment.
@AtharvChandratre Yeah, you can try cypress retry and timeout by adding it to the cypress config. Reference - https://docs.cypress.io/guides/guides/test-retries#Configure-Test-Retries
There was a problem hiding this comment.
@derberg thank you for the review. The suggestion makes complete sense. The timeout is in place because, as you pointed out, Cypress might keep running taking over a GH runner for too long. But I looked into the default wait times for various cypress commands (https://docs.cypress.io/guides/core-concepts/introduction-to-cypress#Default-Values) and they seem to be sufficiently large for our use case, while not taking too much GH Runner compute time, so I'm getting rid of that part.
Yeah, we should parallelize the tests locally as well, but the problem is, we can't use the same parallel tests implementation for GH actions (as you tried before). Our main concern here is to optimise the time taken by Cypress tests in GH Actions. |
@reachaadrika - Thanks for the suggestion, this makes sense. Will make the change. Edit - I tried to make this change, but in the cypress-tests.yml file it seems that we are unable to call the cypress-parallel.js file to extract all of the cypress spec files we have. This is because we can't run |
Addressing comments
Addressing Comments
|
@AtharvChandratre Kindly recheck the workflow file. All the tests are running in single container only. |
@akshatnema - Yep, @VidhiRambhia is doing some research to see if there is a way to make all of the tests to run in a single job. Will let you know if it is possible soon. Thanks! |
Hi @akshatnema, It's a great suggestion and I have been researching this. The matrix strategy approach used for parallelization (https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) runs the same job but with runs spread across different containers. Because of the logic we have defined in There is an approach to achieve this more optimally - https://github.com/cypress-io/github-action#parallel. But, unfortunately, this requires a Cypress Cloud account which is paid. I understand that it might be confusing to see multiple runs of the same job as shown in your screenshot. Please let us know if you have any suggestions to change how workflow jobs are displayed in the check summary |
I and @derberg discussed this and got that we can have matrix containers and different jobs running in the same workflow. It works for us. For now, I can see that all the tests are running in each job, despite having multiple containers. You can revert to previous changes and they were good and efficient. |
Thank you for your feedback, the necessary changes have been made. The tests are now running in parallel like before. |
|
@derberg Do you know why the workflow is still waiting to get executed? It's not getting finished although the changes were pushed last night. Is it because we changed a workflow which is under |
|
@derberg @AtharvChandratre I'm not linking the issue with this PR because test coverage is still not completed. Apart from that, it's approved from my side ✅ |
|
@derberg final review from your side? |
|
/rtm |
akshatnema
added
the
hacktoberfest-accepted
Description
This PR enables the Cypress Tests to run in parallel in Github Actions, bringing down the time it took for the Cypress Tests part of the pipeline to run from ~8 min to ~2 min.
Related issue:
#2104
Test Results:
Time taken by non-parallel cypress runs:
Time taken by parallel cypress runs (with this PR's changes):