Behavior of the status and result in testRunEnd

[gary083]

I find it strange that we receive a "testRunEnd": {"status": "COMPLETE", "result": "PASS"} even when encountering a FAIL diagnosis in TestStep or when using the TestRun.add_error() method.

For instance, in the provided example at this link: https://github.com/opencomputeproject/ocp-diag-core-python/blob/4671c0f4f1591311851c7437162dc804e3848fdb/examples/sample_output.txt#L187C1-L187C24

We still obtain the PASS result even though we use add_error().

I believe we can manually collect the status/result ourselves and raise the TestRunError if any error or failed diagnostic occurs.

I would like to inquire whether this behavior is intentional or a bug. Additionally, I am interested in knowing the best practice for handling such situations.

Thank you!

[mimir-d]

sorry for the delay; just saw this now. i'll add it to my todos to look at

[mimir-d]

Looks like we may need to add some text to the readme to explain this situation. The python api will not make decisions on whether a test step/run succeeded or failed because there is no clear definition that it could use. Measurements may not validate, errors could be emitted, etc and yet a diag developer would still reason that the step/run succeeded.

To note, there was a prior feature request that the validation of measurements should trigger an automatic test step failure (the TODO file doesnt seem to have made it into the repo), and I did leave extensibility options in the architecture of the codebase as it is right now, specifically for this kind of request.

However, call for action here; if anyone is interested in implementing such a functionality (which would be opt-in, can be done with the lib setup api), feel free to publish a PR.

[mimir-d]

Gonna keep this issue open until we have some text in the README to explain the current design decision.