-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[uss_qualifier] Improve test run report validation #298
[uss_qualifier] Improve test run report validation #298
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please note the comment inline
else: | ||
raise ValueError(f"Cannot compare Severity to {type(other)}") | ||
|
||
if self == Severity.Critical: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I may have misunderstood but if we assume that Critical is bigger than Low severity, then it looks like the operator implementation is not completely correct.
self | other | expected gt | implemented gt | expected == implemented ? |
---|---|---|---|---|
Critical | Critical | False | False | ✅ |
Critical | High | True | False | ? |
Critical | Medium | True | False | ? |
Critical | Low | True | False | ? |
High | Critical | False | True | ? |
High | High | False | False | ✅ |
High | Medium | True | False | ? |
High | Low | True | False | ? |
Medium | Critical | False | False | ✅ |
Medium | High | False | False | ✅ |
Medium | Medium | False | False | ✅ |
Medium | Low | True | True | ✅ |
Low | * | False | True | ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤦 thanks. Ok, fix verified with:
from monitoring.uss_qualifier.common_data_definitions import Severity
severities = (Severity.Critical, Severity.High, Severity.Medium, Severity.Low)
operators = ("==", "!=", ">", ">=", "<", "<=")
for op in operators:
corner = f"↓ {op} →"
print(f"{corner:<10}" + "".join(f"{s.value:<10}" for s in severities))
for r in severities:
line = f"{r.value:<10}"
for c in severities:
v = eval(f"r {op} c")
line += f"{v:<10}"
print(line)
print()
* Improve test run report validation * Fix Severity comparison logic 89aaa4b
Currently, all checks in all dev tests (used for CI) pass. This limits test coverage substantially as no logic involving failed checks is ever exercised. This PR reworks test run report validation to allow for more sophisticated test run validation criteria, including allowing (and requiring) the test runs of some configurations to contain failed checks (and, in the future, additional and more specific characteristics).