Cram tests: $TMP is not a per-test directory

[victorlin]

Filing this issue mostly to document behavior that I came across today.

Context

Unintuitively, the $TMP directory defined by Cram is the same across all tests invoked by the same cram command. This means it isn't cleaned up at the end of each test, and lingering output files from one test may affect the outcome of another test.

Instead, the starting directory of a test file is a directory under $TMP that is truly unique for that test file.

How to reproduce

See this GitHub gist.

Possible solutions

1. Clean up $TMP after every test

This is what most tests do already (e.g. tests/functional/mask.t). However, some tests still leave things under $TMP (e.g. tests/builds/zika.t leaves a directory $TMP/out/). Find these lingering files and add cleanup commands to prevent unintended side effects.

2. Run tests from the starting directory

1. Remove pushd $TESTDIR … commands to stay in the per-test temporary directory.
2. Change the export AUGUR commands accordingly.
3. Replace all usage of $TMP with relative paths.
4. Update the relative paths with $TESTDIR to use existing test files.

[corneliusroemer]

Thanks for tackling this! I've noticed this in the past, too.

[victorlin]

This has been factored into dev docs and concrete solutions proposed in #1176.