Style Guide Constraints (#856)

* Create style guide for FedRAMP OSCAL Constraints (#760) * Remove FedRAMP namespace from 'data-center' props (#795) * Hotfix/info (#780) * fix informational constraint handling and make ssp-all valid correct * revert external constraint changes * Update fedramp-external-constraints.xml * Update fedramp_extensions_steps.ts * update info handling * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update src/validations/constraints/fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update src/validations/constraints/fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * Update dev-constraint.js --------- Co-authored-by: Gabeblis <[email protected]> * [skip ci] Create style guide doc for #675 * [skip ci] FSCR-1 re external constraints for #675 * [skip ci] FCSR-1, woops, need formal name for #675 * [skip ci] Tweak FCSR-1 anchor ID in #675 * [skip ci] Stop header hacks for IDs in #675 I read more about these techniques than I would like, but none of them appear to work effectively for making anchors like `#fcsr-1` without adding other content to the anchor which I would like to avoid. https://gist.github.com/asabaylus/3071099?permalink_comment_id=3895584 Either it never worked or something changed. Oh well! * [skip ci] Add FCSR-2 on context sorting for #675 * [skip ci] Add FCSR-3 about alpha sorting for #675 * [skip ci] Add FCSR-4 to require help-url for #675 * [skip ci] Adjust title from style guide to dev style guide per Rene's review * [skip ci] Adjust grammar and style per Rene's review Co-authored-by: Rene Tshiteya <[email protected]> * [skip ci] @Rene2mt's feedback: add ID req for #675 * [skip ci] @Rene2mt's feedback: level req for #675 * [skip ci] @Rene2mt's review: why CRITICAL for #675 * [skip ci] Woops, missed IDs for reqs for #675 * [skip ci] Feedback: add message req for #675 * [skip ci] Fix constraint path in examples for #675 * [skip ci] Add remarks rec guidance for #675 * [skip ci] Add @wandmagic's rec for FCSR-1 for #675 * [skip ci] Add FCSR-10 re active voice for #675 * [skip ci] Remove FCSR-10's incorrect only for #675 * [skip ci] Add FCSR-11 about BCP14 words for #675 * [skip ci] Add no-jargon req FCSR-12 for #675 * [skip ci] Item, not sequence style req for #675 * [skip ci] Add req for sequence ctx hints for #675 * [skip ci] Add FCSR-15 re formal-names for #675 * [skip ci] Remove anchor hack from FCSR-1 for #675 * [skip ci] Wrap up kebab case IDs, reorder for #675 * [skip ci] Fixes from @Rene2mt'2 review for #675 * [skip ci] Add labels for rules in #675 * [skip ci] Simplify rule titles for #675 Follow feedback from @brian-ruf in his review. * [skip ci] Finalize table index with reqs for #645 * [skip ci] Limit informational constraints for #675 * [skip ci] Feedback: FRR1 about OSCAL constraints, not Metaschema constraints Co-authored-by: David Waltermire <[email protected]> * [skip ci] Update FRR1 in table listing too Co-authored-by: David Waltermire <[email protected]> * [skip ci] Add space in status row of table for FRR2 Co-authored-by: David Waltermire <[email protected]> * [skip ci] Add missing word to FRR3 title Co-authored-by: David Waltermire <[email protected]> * [skip ci] Improve the prose in FRR2 guidance Co-authored-by: David Waltermire <[email protected]> * [skip ci] Reorder statements in sentence of FRR2 guidance Co-authored-by: David Waltermire <[email protected]> * [skip ci] Clarify ambiguous wording in FRR5 Co-authored-by: David Waltermire <[email protected]> * [skip ci] Correct typos in FRR6 formal name Co-authored-by: David Waltermire <[email protected]> * [skip ci] Make FRR7 formal name more explicit Co-authored-by: David Waltermire <[email protected]> * [skip ci] Improve FRR8 formal name Co-authored-by: David Waltermire <[email protected]> * [skip ci] Fix FRR8 formal name in table index Co-authored-by: David Waltermire <[email protected]> * [skip ci] Fix FRR9 formal name in table index Co-authored-by: David Waltermire <[email protected]> * [skip ci] Fix FRR9 formal name in table index Co-authored-by: David Waltermire <[email protected]> * [skip ci] Adjust FRR9 guidance to specify expect constraints Co-authored-by: David Waltermire <[email protected]> * [skip ci] Adjust FRR9 constraint examples for correct type Co-authored-by: David Waltermire <[email protected]> * [skip ci] Adjust FRR10 formal name to be more clear Co-authored-by: David Waltermire <[email protected]> * [skip ci] Fix FRR10 formal name in table index Co-authored-by: David Waltermire <[email protected]> * [skip ci] Make FRR11 formal name better sentence fragment Co-authored-by: David Waltermire <[email protected]> * [skip ci] Fix FRR11 above requirement text Co-authored-by: David Waltermire <[email protected]> * [skip ci] Adjust FedRAMP reqs prefix FCSR->FRR Given related work in the program, I want to generalize the prefix to be more general and global for all form of FedRAMP requirements down the road. * [skip ci] Add missing examples to FRR17 for #675 * [skip ci] Align formal names, spacing for #675 I had to fix up some of the formal names where Dave covered some of them in many places, but not all. Also other suggestions add some space. * [skip ci] Add level to many examples, finish #675 * [skip ci] Fold longer bg info for reqs in #675 * [skip ci] Clarify FRR1 bad example is bad in #645 * [skip ci] Clarify context order examples for #675 * [skip ci] Clarify case sorting for FRR3 in #675 * [skip ci] Clean up explanation of FRR10 for #675 * [skip ci] Fix typos in FRR13 and FRR15 for #675 * [skip ci] FRR2 feedback from Kylie for #675 * [skip ci] Reword FRR9 with Kylie's feedback in #675 * [skip ci] Woops, FRR16 twice, no FRR17 for #675 * [skip ci] Last call and let reqs in FRR18 for #675 * [skip ci] Correct ID for FRR18 to anchor in table Co-authored-by: Gabeblis <[email protected]> * [skip ci] Offset req ID sequence Per discussion with others on a call with leads and staff from both FR branches, begin with an offset sequences and reserve the first 100 for other uses for the time being. /cc @kscarf1 * [skip ci] BCP14 keywords in #675 summary text * [skip ci] Tighten up summary text more for #675 * [skip ci] Add back to top anchors for #675 * [skip ci] Better grammar and flow for #675 summary * [skip ci] Improve FRR102 guidance text for #675 * [skip ci] Capitalize and fix FRR110 title for #675 * [skip ci] Fix poor grammar in FRR117 text for #675 * [skip ci] Explicit docs URL in FRR104 for #675 Address missing feedback to @kyhu65867 from review that had not been previously addressed by yours truly. * [skip ci] Fix FRR105 with feedback for #675 Address some feedback about wording and style of the unique ID req. * [skip ci] Fix FRR103 spacing for #675 Completely address feedback from @david-waltermire after checking for final review of style guide left in the comment below. #760 (comment) * [skip ci] Fix FRR108 conformant example for #675 --------- Co-authored-by: Rene Tshiteya <[email protected]> Co-authored-by: wandmagic <[email protected]> Co-authored-by: Gabeblis <[email protected]> Co-authored-by: David Waltermire <[email protected]> * Apply Style Guide To Constraints (#852) * Add props to each constraint that has sufficient existing documentation * Sort constraints alphabetically and sort ascending by metapath specificity * Add missing (and available) help-url * IETF BCP14 Keywords in Constraint Messages * spacing between context blocks for readability * sort alphabetically and use consistent spcaing. * Add style guide constraints Add constraints to enforce style guide Place holders for constraints to add when necessary functions for implementation are supported Add formal names space Add back constraint Delete temporary test file Point help-url to develop Clean up id and formal name Script name change to test:style * Clean up errors by adding place holder urls and change data-center-US to lowercase. Small touch up to style guide step. * make id lowercase * remove space * inject schema Co-authored-by: A.J. Stein <[email protected]> * TODO * Move style guide * TODO * add help-url --------- Co-authored-by: A.J. Stein <[email protected]> Co-authored-by: Rene Tshiteya <[email protected]> Co-authored-by: wandmagic <[email protected]> Co-authored-by: David Waltermire <[email protected]> Co-authored-by: A.J. Stein <[email protected]>
GSA · Nov 6, 2024 · 39d9983 · 39d9983
1 parent 3542c96
commit 39d9983
Show file tree

Hide file tree

Showing 11 changed files with 313 additions and 134 deletions.
diff --git a/features/fedramp_extensions.feature b/features/fedramp_extensions.feature
@@ -1,5 +1,9 @@
 Feature: OSCAL Document Constraints
 
+@style-guide
+Scenario Outline: Validating OSCAL constraints with metaschema constraints
+  Then I should verify that all constraints follow the style guide constraint
+
 @constraints
 Scenario Outline: Validating OSCAL documents with metaschema constraints
   Given I have Metaschema extensions documents
@@ -208,11 +212,11 @@ Examples:
   | cloud-service-model |
   | component-type |
   | control-implementation-status |
-  | data-center-US |
   | data-center-alternate |
   | data-center-count |
   | data-center-country-code |
   | data-center-primary |
+  | data-center-us |
   | deployment-model |
   | fedramp-version |
   | has-authenticator-assurance-level |

diff --git a/features/steps/fedramp_extensions_steps.ts b/features/steps/fedramp_extensions_steps.ts
@@ -665,4 +665,56 @@ Then("I should have both FAIL and PASS tests for constraint ID {string}", functi
     constraintId,
     `Constraint ${constraintId} is not in the extracted constraints list`
   );
+});
+
+Then('I should verify that all constraints follow the style guide constraint', async function () {
+  const baseDir = join(__dirname, '..', '..');
+  const constraintDir = join(baseDir, 'src', 'validations', 'constraints');
+  const styleGuidePath = join(baseDir, 'src', 'validations', 'styleguides', 'fedramp-constraint-style.xml');
+
+  const constraint_files = readdirSync(constraintDir).filter((file) => file.startsWith('fedramp') && file.endsWith('constraints.xml') );
+  const errors = [];
+
+  function filterOutBrackets(input) {
+    return input.replace(/\[.*?\]/g, '');
+  }
+
+  for (const file_name of constraint_files) {
+    const filePath = join(constraintDir, file_name.trim());
+    console.log(filePath);
+    try {
+      console.log(filePath);
+      const [result, error] = await executeOscalCliCommand('metaschema', [
+        'validate',
+        filePath,
+        '-c',
+        styleGuidePath,
+        '--disable-schema-validation'
+      ]);
+
+      console.log(`Validation result for ${file_name}:`, result);
+      if (error) {
+        console.error(`Validation error for ${file_name}:`, error);
+      }
+
+      const filteredError = filterOutBrackets(error);
+      if (filteredError) {
+        errors.push(`Style guide validation failed for ${file_name}: ${filteredError}`);
+      }
+      if (result.includes("ERROR")) {
+        errors.push(`Style guide validation found errors in ${file_name}: ${result}`);
+      }
+    } catch (error) {
+      errors.push(`Error processing ${file_name}: ${error}`);
+    }
+  }
+
+  // Display all errors at the end
+  if (errors.length > 0) {
+    console.error("Validation errors found:");
+
+    throw new Error("Style guide validation failed. "+errors.join("\n"));
+  }
+
+  expect(errors, "No style guide validation errors should be found").to.be.empty;
 });
diff --git a/package.json b/package.json
@@ -13,6 +13,7 @@
     "test:failed": "cross-env NODE_OPTIONS=\"--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node\" cucumber-js -p rerun",
     "test:constraints": "cross-env NODE_OPTIONS=\"--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node\" cucumber-js --tags @constraints",
     "test:coverage": "cross-env NODE_OPTIONS=\"--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node\" cucumber-js --tags @full-coverage",
+    "test:style": "cross-env-shell NODE_OPTIONS=\"--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node\" cucumber-js --tags @style-guide",
     "mq": "node ./src/scripts/dev-metaschema-eval.js",
     "constraint": "node ./src/scripts/dev-constraint.js"
   },

diff --git a/src/validations/constraints/content/ssp-data-center-country-code-INVALID.xml b/src/validations/constraints/content/ssp-data-center-country-code-INVALID.xml
@@ -7,7 +7,7 @@
     <location uuid="11111112-0000-4000-9001-000000000009">
       <address>
       </address>
-      <prop name='type' value='data-center' class='primary'/>
+      <prop name="type" value="data-center" class="primary"/>
     </location>
     <location uuid="11117112-0080-4000-9001-001230000009">
       <address>

diff --git a/...ts/content/ssp-data-center-US-INVALID.xml → ...ts/content/ssp-data-center-us-INVALID.xml b/...ts/content/ssp-data-center-US-INVALID.xml → ...ts/content/ssp-data-center-us-INVALID.xml
diff --git a/src/validations/constraints/fedramp-external-allowed-values.xml b/src/validations/constraints/fedramp-external-allowed-values.xml
diff --git a/src/validations/constraints/fedramp-external-constraints.xml b/src/validations/constraints/fedramp-external-constraints.xml
diff --git a/src/validations/constraints/unit-tests/data-center-us-FAIL.yaml b/src/validations/constraints/unit-tests/data-center-us-FAIL.yaml
@@ -1,7 +1,7 @@
 test-case:
-  name: Negative Test for data-center-US
-  description: This test case validates the behavior of constraint data-center-US
-  content: ../content/ssp-data-center-US-INVALID.xml
+  name: Negative Test for data-center-us
+  description: This test case validates the behavior of constraint data-center-us
+  content: ../content/ssp-data-center-us-INVALID.xml
   expectations:
-    - constraint-id: data-center-US
+    - constraint-id: data-center-us
       result: fail
diff --git a/src/validations/constraints/unit-tests/data-center-us-PASS.yaml b/src/validations/constraints/unit-tests/data-center-us-PASS.yaml
@@ -1,7 +1,7 @@
 test-case:
-  name: Positive Test for data-center-US
-  description: This test case validates the behavior of constraint data-center-US
+  name: Positive Test for data-center-us
+  description: This test case validates the behavior of constraint data-center-us
   content: ../content/ssp-all-VALID.xml
   expectations:
-    - constraint-id: data-center-US
+    - constraint-id: data-center-us
       result: pass
diff --git a/src/validations/constraints/STYLE.md → src/validations/styleguides/STYLE.md b/src/validations/constraints/STYLE.md → src/validations/styleguides/STYLE.md
diff --git a/src/validations/styleguides/fedramp-constraint-style.xml b/src/validations/styleguides/fedramp-constraint-style.xml
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<metaschema-meta-constraints xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xsi:schemaLocation="http://csrc.nist.gov/ns/oscal/metaschema/1.0 https://raw.githubusercontent.com/metaschema-framework/metaschema/0441e6d4c9bce5b6c40b4647148019e4f47bed08/schema/xml/metaschema-meta-constraints.xsd">
+    <!-- ============================== -->
+    <!-- FedRAMP Constraint Style guide -->
+    <!-- ============================== -->
+
+    <context>
+        <metapath target="/metaschema-meta-constraints"/>
+        <constraints>
+            <!-- TODO: Implement frr103 when axes supported after new oscal-cli release uses metaschema-java release with metaschema-framework/metaschema-java#229 included. -->
+            <!-- <expect id="frr103" target="//expect | //allowed-values | //index-has-key" test="" level="ERROR">
+                <formal-name>Constraints in the Context Sorted Alphabetically by ID</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr103"/>
+                <message>A FedRAMP constraint MUST be sorted Alphabetically within it's specific context block.</message>
+            </expect> -->
+            <expect id="frr104" target="//expect | //index-has-key" test="count(prop[@namespace = 'https://docs.oasis-open.org/sarif/sarif/v2.1.0' and @name = 'help-url']) eq 1" level="ERROR">
+                <formal-name>Constraints Have a Help URL Property</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr104"/>
+                <message>A FedRAMP constraint MUST define a help URL.</message>
+            </expect>
+            <!-- TODO: Uniqueness check needs to be added to frr105 once support is added for a function like distinct-values() or some other method of ensuring that an id is unique-->
+            <expect id="frr105" target="//expect | //allowed-values | //index-has-key" test="@id" level="ERROR">
+                <formal-name>Constraints Have a Unique ID</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr105"/>
+                <message>A FedRAMP constraint MUST have an id.</message>
+            </expect>
+            <expect id="frr106" target="//expect | //allowed-values | //index-has-key" test="matches(@id, '^[a-z0-9-]+$')" level="ERROR">
+                <formal-name>Constraints Have IDs with Lower Case Letters, Numbers, and Dashes</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr106"/>
+                <message>A FedRAMP constraint id MUST only consist of lowercase letters, numbers 0-9, or "-" characters.</message>
+            </expect>
+            <expect id="frr107" target="//expect | //allowed-values | //index-has-key" test="matches(@level, '\b(CRITICAL|ERROR|WARNING|INFORMATIONAL|DEBUG)\b')" level="ERROR">
+                <formal-name>Constraints Have an Explicit Severity Level</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr107"/>
+                <message>A FedRAMP constraint MUST specify a valid severity level.</message>
+            </expect>
+            <expect id="frr109" target="//expect" test="message" level="ERROR">
+                <formal-name>Expect Constraint Message Field Required</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr109"/>
+                <message>A FedRAMP constraint MUST include a message describing the requirement.</message>
+            </expect>
+            <expect id="frr112" target="//expect" test="matches(message, '(MUST|MUST NOT|REQUIRED|SHALL|SHALL NOT|SHOULD|SHOULD NOT|RECOMMENDED|MAY|OPTIONAL)')" level="ERROR">
+                <formal-name>IETF BCP14 Keywords in Constraint Messages</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/STYLE.md#frr112"/>
+                <message>A FedRAMP constraint MUST include one of the IETF BCP14 keywords in the message.</message>
+            </expect>
+            <expect id="frr116" target="//expect | //allowed-values | //index-has-key" test="formal-name" level="ERROR">
+                <formal-name>Constraints Formal Names Required</formal-name>
+                <prop namespace="https://docs.oasis-open.org/sarif/sarif/v2.1.0" name="help-url" value="https://github.com/GSA/fedramp-automation/blob/develop/src/validations/styleguides/README.md#frr116"/>
+                <message>A FedRAMP constraint MUST include a formal name.</message>
+            </expect>
+        </constraints>
+    </context>
+
+</metaschema-meta-constraints>