Skip to content

Commit

Permalink
Fix CI regression where most Windows failures passed CI
Browse files Browse the repository at this point in the history
This runs `cargo nextest ...` and `cargo test --doc` in separate
steps in the `test-fast` job, so that the job fails when
`cargo nextest ...` fails. Otherwise, with `pwsh` on Windows, test
failures (other than doctests) are masked.

Background: Since 89a0567 (Byron#1556), doctests are run on all three
major platforms, and not only on the full test job done on Ubuntu.
But the way this was done relied on a script failing as soon as
(or, at least, whenever) any command in the script failed. That
works on Ubuntu and macOS, where a `bash` shell is used by default,
with `-e` passed. But on Windows, GitHub Actions uses `pwsh` as the
default shell. `pwsh` is not run in a way that causes it to stop at
the first failing command.

So, on Windows, when the `cargo nextest` command failed but the
`cargo test --doc` command that followed it in the same script
step passed, the step passed, thus allowing the job and workflow to
pass. This was observed in Byron#1429 after a rebase (see comments).

Note that this is not related to the increased use of `nextest`.
While that was also done in Byron#1556, it did not affect the
`test-fast` job where the bug was introduced, which was already
using `nextest`.

This fixes the problem by putting the two commands in separate
steps.

This is simpler than doing anything in PowerShell to make the
script stop, such as using `&&` or attempting to produce `-e`-like
behavior.

Another option could be to use `bash` as the shell, which is a Git
Bash environment suitable for running the tests. The reason I
didn't do that is that I think it is valuable to see the results
when the tests are run from a PowerShell environment.

In particular, continuing to use PowerShell here should help in
investigating Byron#1359 (and shows that the claim I made is overly
strong, since CI on Windows with `pwsh` not itself started from a
Unix-style shell is not "Git Bash or a similar environment").
  • Loading branch information
EliahKagan committed Aug 26, 2024
1 parent 012445a commit 4f2ab5b
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,9 @@ jobs:
with:
tool: nextest
- name: "Test (nextest)"
run: |
cargo nextest run --all --no-fail-fast
cargo test --doc
run: cargo nextest run --all --no-fail-fast
- name: Doctest
run: cargo test --doc

test-32bit:
runs-on: ubuntu-latest
Expand Down

0 comments on commit 4f2ab5b

Please sign in to comment.