You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Many test cases in GatorGrader check the newline count of generated output. It turns out that this count can change depending on an external variable: the number of columns in the terminal used to run pytest. With a $COLUMNS value fewer than 80, certain tests will fail, with more failures the smaller the terminal becomes. The same is true in reverse, with a $COLUMNS value of greater than 82.
To Reproduce
Steps to reproduce the behavior:
Resize a terminal such that echo $COLUMNS prints a number less than 80 or greater than 82
Execute pipenv run cover to see how many tests fail at that given size.
Expected behavior
Tests should not depend on terminal size for correctness.
Environment
GitHub Actions terminal, Elementary OS 5 (based on Ubuntu 18.04) gnome-terminal.
Python 3.9.7 (also reproducible in 3.8.12).
I can confirm that this bug does not appear in Python 3.7.12.
Proposed Solution
This bug is caused by asserting an exact amount of new lines in expected produced output; instead of asserting an exact amount, the minimum amount possible should be "at least" (with >=) asserted. Alternatively, new line count should be dropped as a measure of correctness for produced output -- this may be the desired solution, since new line count can be influenced by many factors, and often does not indicate regressions or errors.
The text was updated successfully, but these errors were encountered:
Describe the bug
Many test cases in GatorGrader check the newline count of generated output. It turns out that this count can change depending on an external variable: the number of columns in the terminal used to run
pytest
. With a$COLUMNS
value fewer than 80, certain tests will fail, with more failures the smaller the terminal becomes. The same is true in reverse, with a$COLUMNS
value of greater than 82.To Reproduce
Steps to reproduce the behavior:
echo $COLUMNS
prints a number less than 80 or greater than 82pipenv run cover
to see how many tests fail at that given size.Expected behavior
Tests should not depend on terminal size for correctness.
Environment
GitHub Actions terminal, Elementary OS 5 (based on Ubuntu 18.04) gnome-terminal.
Python 3.9.7 (also reproducible in 3.8.12).
I can confirm that this bug does not appear in Python 3.7.12.
Proposed Solution
This bug is caused by asserting an exact amount of new lines in expected produced output; instead of asserting an exact amount, the minimum amount possible should be "at least" (with
>=
) asserted. Alternatively, new line count should be dropped as a measure of correctness for produced output -- this may be the desired solution, since new line count can be influenced by many factors, and often does not indicate regressions or errors.The text was updated successfully, but these errors were encountered: