[uss_qualifier] Add execution control #292
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With uss_qualifier's configurations growing larger, there is often a need to focus execution on a smaller part of the final configuration for the purpose of rapid development. Currently, users must create a new configuration sufficiently similar to the final configuration to accomplish this, and this can be, tedious, error-prone, and time-consuming.
This PR introduces execution control to uss_qualifier by allowing users to adjust their existing configurations only slightly in order to selectively execute only parts of the normal test run. This includes, for instance, targeting a particular scenario, a particular instance of a particular scenario, or a test suite, or excluding the same. As such, the configurations exercising particular components of the F3548-21 test suite in the configurations/f3548 folder are now obsolete and are removed.
Execution control is accomplished by an additional
execution
parameter in theTestConfiguration
object; see configuration.py for its structure. The addition of this configuration field results in auto-generating a number of schema (the JSON files in this PR are all autogenerated).Implementation of this functionality is in suite.py where a new
ExecutionContext
object is added. It is similar to the stack in normal programming languages, as it keeps track of what is running and has been run. The logic for whether to skip a particular action is the bulk ofExecutionContext
's code.This new functionality is documented with a number of examples in configurations/README.md.
A number of small bugs and improvements discovered while exercising this functionality are addressed.
All of the examples were tested by inserting the
execution
content shown into the configurations.dev.uspace configuration.