Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add unit test metaschema conformant with ADR#6 #826

Merged
merged 8 commits into from
Oct 29, 2024
Merged

Add unit test metaschema conformant with ADR#6 #826

Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
df65a4f
[skip ci] Metaschema module for unit test struct
aj-stein-gsa Oct 28, 2024
e205177
[skip ci] Document unit test metaschema elements
aj-stein-gsa Oct 28, 2024
add8f9b
[skip ci] Do not explicitly use default types
aj-stein-gsa Oct 28, 2024
66384cb
[skip ci] Add missing `@id` and `@level` to model
aj-stein-gsa Oct 28, 2024
528f342
[skip ci] Add pipeline to pipeline, thx @wandmagic
aj-stein-gsa Oct 28, 2024
74750a7
[skip ci] Backport #219 docs into model remarks
aj-stein-gsa Oct 28, 2024
3a245cf
[skip ci] Fix max-occurs re bad model cardinality
aj-stein-gsa Oct 29, 2024
bd294a3
[skip ci] Is a pipeline with a ref even a pipeline at all?
aj-stein-gsa Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions src/validations/constraints/unit-tests/unit_test_metaschema.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<?xml version="1.0" encoding="UTF-8"?>
<METASCHEMA xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://csrc.nist.gov/ns/oscal/metaschema/1.0 https://raw.githubusercontent.com/metaschema-framework/metaschema/0441e6d4c9bce5b6c40b4647148019e4f47bed08/schema/xml/metaschema.xsd"
xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0">
<schema-name>OSCAL Unit Test Module</schema-name>
<schema-version>1.0.0</schema-version>
<short-name>oscal-unit-tests</short-name>
<namespace>http://github.com/gsa/fedramp-automation/constraint/test</namespace>
<json-base-uri>http://github.com/gsa/fedramp-automation/constraint/test</json-base-uri>
<define-assembly name="test-case">
<formal-name>Test Case Document</formal-name>
<description>This is the root element of the FedRAMP Automation Test Suite format for collections of unit tests.</description>
<root-name>test-case</root-name>
<model>
<field ref="name" min-occurs="1" max-occurs="1"/>
<field ref="description" min-occurs="1" max-occurs="1"/>
<field ref="content" min-occurs="0" max-occurs="unbounded">
<group-as name="content" in-json="SINGLETON_OR_ARRAY"/>
</field>
aj-stein-gsa marked this conversation as resolved.
Show resolved Hide resolved
<assembly ref="expectation" min-occurs="1" max-occurs="unbounded">
<group-as name="expectations" in-json="ARRAY"/>
</assembly>
</model>
</define-assembly>
<define-field name="name">
<formal-name>Test Case Name</formal-name>
<description>The name of the test case collection within a test cases file. </description>
</define-field>
<define-field name="description">
<formal-name>Test Case Description</formal-name>
<description>The description of the test case collection within a test case file.</description>
</define-field>
<define-field name="content">
<formal-name>Test Case Content Path</formal-name>
<description>A relative or absolute file path or HTTP URL to the OSCAL document instance for which the test runs.</description>
<constraint>
<expect id="test-case-content-url-invalid" level="CRITICAL" target="." test=". => doc-available()" >
<message>Test content URL does not point to a valid file or HTTP resource. If the path is invalid, tests are not operable. Are you sure it's correct?</message>
</expect>
</constraint>
<remarks>
<p>In this format, we often use relative paths that are not strictly conformant to the URI datatype in Metaschema.</p>
<p>There, we use the default implied <code>as-type="string"</code> and not <code>as-type="uri"</code>.</p>
<p>However, this approach does work and the constraint passes successfully.</p>
<p>At this time, a violation where the test evaluates to `false()` throws a lower-level exception.</p>
<p>See <a href="https://github.com/metaschema-framework/metaschema-java/issues/219">metaschema-framework/metaschema-java#219</a> for details.</p>
</remarks>
</define-field>
<define-assembly name="pipeline">
<formal-name>Test Case Pipeline Action</formal-name>
<description>The test case pipeline is a sequence of one or more <code>actions</code> to execute before the test runner executes the test suite.</description>
<model>
<field ref="action" max-occurs="unbounded">
<group-as name="actions" in-json="ARRAY"/>
</field>
</model>
</define-assembly>
<define-assembly name="expectation">
<formal-name>Test Case Expectation</formal-name>
<description>This assembly defines the relevant constraint ID and the desired result when executing tests over the content. There MAY be one or more expectations.</description>
<model>
<field ref="constraint-id" min-occurs="1" max-occurs="1"/>
<field ref="result" min-occurs="1" max-occurs="1"/>
</model>
</define-assembly>
<define-field name="constraint-id">
<formal-name>Constraint ID for the Expectation</formal-name>
<description>The <code>@id</code> attribute for a given constraint for which the test serves to match the expectation.</description>
</define-field>
<define-field name="result">
<formal-name>Expectation Result</formal-name>
<description>The expected result type in <a href="https://docs.oasis-open.org/sarif/sarif/v2.1.0/csprd01/sarif-v2.1.0-csprd01.html">SARIF format</a> for the expectation. The expectation only supports specific values, defined in a constraint.</description>
<constraint>
<allowed-values id="test-case-expectation-result" target="." allow-other="no" level="ERROR">
<enum value="pass">Use when the desired result of test execution without a constraint passing</enum>
<enum value="fail">Use when the desired result of test execution with a constraint violation</enum>
</allowed-values>
</constraint>
</define-field>
<define-field name="action">
<formal-name>Pipeline Action</formal-name>
<description>An action for a pipeline to perform before the test runner executes the test suite. This field only supports allowed values explicitly defined in the constraint.</description>
<constraint>
<allowed-values id="test-case-pipeline-action-allowed-values" target="." allow-other="no" level="ERROR">
<enum value="resolve-profile">An action declaration to perform profile resolution for a profile associated with expectation test case.</enum>
<remarks>
<p>See the <a href="https://automate.fedramp.gov/documentation/general-concepts/profile-resolution">official FedRAMP's documentation on profile resolution</a> for detailed information.</p>
</remarks>
</allowed-values>
</constraint>
</define-field>
</METASCHEMA>