-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(validation)!: validation testing framework #667
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor comments as I look at general structure otherwise no major concerns.
|
||
- `name`: The name of the test | ||
- `changes`: An array of changes or transformations to be applied to the resources used in the test validation | ||
- `expected-result`: The expected result of the test - satisfied or not-satisfied |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The expected result of the test - satisfied or not-satisfied
This is an interesting point as we look to clear a path for validations used outside of OSCAL. We map the result of a policy (pass/fail) to align to findings/observations using the OSCAL terminology (satisfied/not-satisfied).
We might look at what terminology OPA/Kyverno use and/or decide on what Lulas will be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's fair - I initially was thinking like pass/fail - that also sort of conflates with the overall test result (which is pass/fail) so walked back the terminology. I mean it could be more like policy/provider-result -> true/false? Or accept/reject?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Walk me through the expected result <> test result of pass/fail and how that might conflate?
Seems like pass/fail at the test result layer would align well to the expected result also being pass/fail
Co-authored-by: Brandt Keller <[email protected]>
…ew-fields-process' into 460-validation-review-fields-process
Made a couple updates to incorporate the templating for dev commands work
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nearly finished with review - no major modifications noticed as of yet.
Description
Initial work to specify Lula-native validation testing
inject
function totransform
- tried to do a bit of refactor to pull out functionality to support more granular testsRunTests
method to theLulaValidation
+ updated schema to supportdev validate
to handle--run-tests
flag, and because this was getting annoying to debug a--print-test-resources
that will print out the transformed resources.json to the validation directory -> This is basically just a print-to-screen implementation right now. Probably further discussion needed on dumping a test-result object or how we want to handle that.Follow-ups to this
dev validate
cmd to support the e2e testing structure we have--run-tests
tolula validate
-> will need to include some exploration/decision on how the test report gets dumpedRelated Issue
Fixes #460
Type of change
Checklist before merging