Added task_id to Tests and Test Reports

[BethanyG]

No description provided.

[BethanyG]

WIP-ing this until I get the golden tests updated.

[BethanyG]

@cmccandless - So now all my added tests are passing. But I am worried. I had to generate the golden files for my added tests using the run-in-docker.sh script, which tells me that we do indeed have different results when using run.sh vs run-in-docker.sh vs plain pytest. Exactly the scenario we're trying to avoid.

We also definitely have a bug weirdness when capturing stdout in conjunction with a subtest. It appears that stdout is not flushed when using the subtest context -- or at least that is not happening when you invoke run.sh from the command line.

To see this in action, edit example_partial_failure_with_subtest.py to include a print("Captured stdout from user!!") statement, and then run run.sh for it. The results.json ends up with output from all subtests written into the main test result.

I feel like it's not fatal - just really ugly (and should be fixed), so I'll log a bug in the morning, and see if I can figure out how to fix it. Any thoughts/insights/pointers you have to that end would be wonderful. 😄

Logged as Issue 67

[cmccandless]

That does sound messy... I love the maintainability of subtests/parameterization, but it's proving to be a real headache with tooling. We might want to discuss the possibility of breaking out the subtests into individual tests.

[BethanyG]

Yup. I don't want to, but if we keep running into issues like this we'll have to. I am hoping this is a small adjustment. But if not, we will have to either do individual tests, or maybe use PyTest to parameterize, as opposed to unittest.subtest. However, replacing unittest.subtest carries a larger refactor of code, since we filter the AST using unittest class definitions.

[BethanyG]

Huh. It appears the bug is not about subtests. It looks like stdout is not being captured when a test fails -- only when it succeeds. Or in the case of the subtest scenario, the parent is collecting the stdout from the children, which is ... actually as expected , when you think about it. In a success, you'd only have the stdout for each subtest, because there isn't a report generated for a successful test.

But now we have to figure out how we get the stdout from failing tests, because that's not coming through. But I don't think it ever did. 🤔 And thats a huge problem, since that doesn't conform to the runner spec.

Logged as Issue 66

[BethanyG]

TL;DR: I think this can be reviewed/merged, and then we work on the bugs.

[cmccandless]

Does the subtests module not already do this?

[BethanyG]

🤣 No. The subtests module makes it possible to use subttest without PyTest freaking out. There is actually an ongoing discussion on the pytest-subtest repo on how to handle failures and test counting right now. And - as the discussion outlines - this is actually behavior inherited from unittest. So for our plugin, I decided to make it "cleaner". But it is still weird to have a count mis-match when all tests pass vs when some that have sub-tests fail (this happens with all parameterization, in Unittest/Pytest if I am remembering correctly).

[BethanyG]

...I'd actually love to do a refactor of our runner that treats parameterization more like a matrix -- so that there is a clean count of which tests are parameterized, then a process that "explodes" the matrix into individual cases, then makes a summary. But I don't think we want to do that right now.

[cmccandless]

No probably right now, but that sounds like a good idea.