Better flaky test debugging tools
The headline features for this release are focused around debugging flaky tests. See the asciicast for a demo of how to use it. Please file an issue if you have any trouble!
The other major change for this release is that qa is now only available as a rubygem. It's no longer trivially distributable as a tgz, as it has dependencies on other rubygems. For now we accept this limitation as it keeps the build process simple and these third-party dependencies provide a dramatically better user experience.
Other usability improvements:
- Improve messaging around seeds used for multiple
test runs. - Errors now always capture locals, since we're
using the more stable binding_of_caller gem. - Support for
qa help - Support for
-runsargument, so qa can run your tests many times - Use
TEST_ENV_NUMBERinstead ofQA_WORKER, like parallel_tests gem - Add support for
QA_ARCHIVEenvironment variable to avoid needing to specify-archivefor each command. - Add
qa flaky topsubcommand. - Add
qa flaky reprosubcommand to reproduce a specific flaky outcome and attach a pry-remote session to it. - Make
qa flakyan (experimental) REPL for working with flaky tests. The two subcommands you can use with it arereproandtop.
Along with a host of fixes:
- Fix local variable capture in cases where bindings do not exactly match the exception backtrace.
- Move (experimental) ActiveRecord model eager loading to
-eager-loadoption (which is disabled by default) - Concat
/lib, etc to$LOAD_PATH, rather than prepending it. - Fix yet another hang-on-ctrl-c.