ci: don't fail fast #838
Merged
ci: don't fail fast #838
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The matrix strategy defaults to failing fast: If any one of the test cases fails all other cases get cancelled as well. The problem with this is that sometimes tests fail not because the code is bad, but because the test is flaky. In an ideal world, we'd make the tests less flaky, but until then we shouldn't abort all other tests in the matrix. The advantage of not failing fast is that when we inevitably have to manually push the re-run button, we don't have to wait for all test cases to complete, but only for the one that failed.
msanft
reviewed
Aug 28, 2024
Following this change failing tests would take the same amount (or less) time as succeeding tests. |
Won't we run all payloads, even if all will fail? So, of course, your statement is correct, but they would take longer than they currently take. |
katexochen
approved these changes
Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The matrix strategy defaults to failing fast: If any one of the test cases fails all other cases get cancelled as well. The problem with this is that sometimes tests fail not because the code is bad, but because the test is flaky. In an ideal world, we'd make the tests less flaky, but until then we shouldn't abort all other tests in the matrix. The advantage of not failing fast is that when we inevitably have to manually push the re-run button, we don't have to wait for all test cases to complete, but only for the one that failed.