-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Specify check severity in test scenario documentation only #404
Labels
Comments
This was referenced Dec 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Currently, an understanding of the expected overall behavior of a test scenario from the test scenario documentation is missing information regarding what will/should happen when a check fails. This is important because failed checks sometimes/often serve as flow control to, for instance, stop the test scenario early or even stop the test run early. Instead, this information is provided only in Python when
check.record_failed
is called.Describe the solution you'd like
We already have the ability to specify the severity of a failed check in documentation, and that documentation information is already available programmatically. We should migrate all check severity information into documentation and remove the ability to specify severity at runtime. This will clarify and simplify test behavior.
Describe alternatives you've considered
We could leave maximum flexibility to specify severity in documentation or in code or override documentation in code. While that would provide maximum developer flexibility, I am unaware of any value we would gain/retain under that approach and we would lose the clarity and simplicity that would accompany specification of check severity in documentation only.
Execution plan
The
record_failed
method has wide usage throughout the codebase (~260 instances) and currentlyseverity
is a required parameter. Therefore, we will probably need to phase this change in over many PRs. I suspect a strategy like the following would be effective:severity
an optional parameter torecord_failed
and have it default to the documented value when not specified (throwing ValueError if not specified via either means)severity
parameter fromrecord_failed
calls where severity is specified in documentationseverity
parameter ofrecord_failed
have been removedseverity
parameter fromrecord_failed
The text was updated successfully, but these errors were encountered: