chore: add html validation to build process (#1659)

Adds link validation to all links within the documentation

CONTRIBUTING.md

@@ -260,4 +260,4 @@ ______________________________________________________________________
 
 ##### Overview of process to take OSCAL models and upgrade trestle Python code
 
-<img src="images/trestle-OSCAL-upgrade.png" style="width:600px;height:500px;border: 1px solid #000;padding:10px;">
+![](images/trestle-OSCAL-upgrade.png)

Makefile

@@ -86,7 +86,7 @@ docs-automation::
 	python ./scripts/website_automation.py
 
 docs-validate:: docs-automation
-	mkdocs build -c -s
+	mkdocs build -v -c -s
 	rm -rf site
 
 docs-serve: docs-automation
@@ -117,4 +117,4 @@ pylint-test:
 	pylint tests --rcfile=.pylintrc_tests
 
 check-for-changes:
-	python scripts/have_files_changed.py -u
+	python scripts/have_files_changed.py -u

docs/cli.md

@@ -113,11 +113,11 @@ Users can query the contents of files using `trestle describe`, and probe the co
 
 OSCAL models are rich and contain multiple nested data structures. Given this, a mechanism is required to address _elements_ /_attributes_ within an oscal object.
 
-This accessing method is called 'element path' and is similar to _jsonPath_. Commands provide element path by a `-e` argument where available, e.g. trestle split -f catalog.json -e 'catalog.metadata.\*'. This path is used whenever specifying an attribute or model, rather than exposing trestle's underlying object model name. Users can refer to [NIST's json outline](https://pages.nist.gov/OSCAL/reference/latest/complete/json-outline/) to understand object names in trestle.
+This accessing method is called 'element path' and is similar to _jsonPath_. Commands provide element path by a `-e` argument where available, e.g. trestle split -f catalog.json -e 'catalog.metadata.\*'. This path is used whenever specifying an attribute or model, rather than exposing trestle's underlying object model name. Users can refer to [NIST's json outline](https://pages.nist.gov/OSCAL-Reference/models/latest/complete/json-outline/) to understand object names in trestle.
 
 ### Rules for element path
 
-1. Element path is an expression of the attribute names, [in json form](https://pages.nist.gov/OSCAL/reference/latest/complete/json-outline/) , concatenated by a period (`.`).
+1. Element path is an expression of the attribute names, [in json form](https://pages.nist.gov/OSCAL-Reference/models/latest/complete/json-outline/) , concatenated by a period (`.`).
    1. E.g. The metadata in a catalog is referred to as `catalog.metadata`
 1. Element paths are relative to the file.
    1. e.g. For `metadata.json` roles would be referred to as `metadata.roles`, from the catalog file that would be `catalog.metadata.roles`

docs/contributing/trestle_oscal_object_model.md

@@ -9,7 +9,7 @@ This functionality, which is built on [pydantic](https://pydantic-docs.helpmanua
 
 ## Mapping and variance with OSCAL names.
 
-The underlying object model that trestle relies on is the json schema published by NIST [here](https://github.com/usnistgov/OSCAL/tree/main/json/schema). In understanding these models the [model reference page](https://pages.nist.gov/OSCAL/reference/1.0.0/) is an indispensable source.
+The underlying object model that trestle relies on is the json schema published by NIST [here](https://github.com/usnistgov/OSCAL/releases/latest). In understanding these models the [model reference page](https://pages.nist.gov/OSCAL-Reference/models/) is an indispensable source.
 
 When generating the python data class based models we have tried to be as faithful as we can to the naming convention provided by OSCAL. This is the hierarchy of rules that we have used:
 

docs/index.md

@@ -65,7 +65,7 @@ natively supports only `json` and `yaml` formats at this time.
 Future roadmap anticipates that support for xml [import](https://github.com/oscal-compass/compliance-trestle/issues/177) and [upstream references](https://github.com/oscal-compass/compliance-trestle/issues/178) will be enabled. However, it is expected
 that full support will remain only for `json` and  `yaml`.
 
-Users needing to import XML OSCAL artifacts are recommended to look at NIST's XML to json conversion page [here](https://github.com/usnistgov/OSCAL/tree/master/json#oscal-xml-to-json-converters).
+Users needing to import XML OSCAL artifacts are recommended to look at NIST's OSCAL converters page [here](https://github.com/usnistgov/OSCAL/blob/main/build/README.md#converters).
 
 ## Python codebase, easy installation via pip
 
@@ -77,12 +77,12 @@ Compliance trestle is currently stable and is based on NIST OSCAL version 1.0.4,
 
 ## Contributing to Trestle
 
-Our project welcomes external contributions. Please consult [contributing](contributing/mkdocs_contributing/) to get started.
+Our project welcomes external contributions. Please consult [contributing](contributing/mkdocs_contributing.md) to get started.
 
 ## License & Authors
 
-If you would like to see the detailed LICENSE click [here](license/).
-Consult [contributors](https://github.com/oscal-compass/compliance-trestle/graphs/contributors) for a list of authors and [maintainers](maintainers/) for the core team.
+If you would like to see the detailed LICENSE click [here](license.md).
+Consult [contributors](https://github.com/oscal-compass/compliance-trestle/graphs/contributors) for a list of authors and [maintainers](maintainers.md) for the core team.
 
 ```text
 # Copyright (c) 2024 The OSCAL Compass Authors. All rights reserved.

docs/tutorials/continuous-compliance/continuous-compliance.md

@@ -17,30 +17,30 @@ Moreover, assuring continuous compliance across multiple cloud vendors can compl
 
 Common sense dictates that standardization would simplify matters. The National Institute of Standards and Technologies (NIST) is developing the Open Security Controls Assessment Language ([OSCAL](https://pages.nist.gov/OSCAL)).
 
-The compliance-[trestle](https://oscal-compass.github.io/compliance-trestle/) open source github project is an effort to employ [OSCAL](https://pages.nist.gov/OSCAL) for compliance standardization and automation. Of great utility is the [trestle](https://oscal-compass.github.io/compliance-trestle/) oscal module that facilitates transformation of data to/from Python object representations in accordance with the [OSCAL](https://pages.nist.gov/OSCAL) schemas.
+The compliance-[trestle](../../index.md) open source github project is an effort to employ [OSCAL](https://pages.nist.gov/OSCAL) for compliance standardization and automation. Of great utility is the [trestle](../../index.md) oscal module that facilitates transformation of data to/from Python object representations in accordance with the [OSCAL](https://pages.nist.gov/OSCAL) schemas.
 
 #### Simple Continuous Compliance Architecture
 
 ![Continuous Compliance Reporting](ContinuousCompliance.jpg)
 
 Cloud Services can often be configured to monitor (and sometimes enforce) policies. Examples include OpenShift Compliance Operator and Tanium. However, the compliance reporting “raw” data produced is unique to each.
 
-Two steps are needed to ascertain your compliance posture. Step 1 is to transform available compliance “raw” data into standardized form ([OSCAL](https://pages.nist.gov/OSCAL)). Step 2 is to examine the [OSCAL](https://pages.nist.gov/OSCAL) data and assemble a compliance posture for the controls and components of interest. And [trestle](https://oscal-compass.github.io/compliance-trestle/) is the go-to solution.
+Two steps are needed to ascertain your compliance posture. Step 1 is to transform available compliance “raw” data into standardized form ([OSCAL](https://pages.nist.gov/OSCAL)). Step 2 is to examine the [OSCAL](https://pages.nist.gov/OSCAL) data and assemble a compliance posture for the controls and components of interest. And [trestle](../../index.md) is the go-to solution.
 
 #### Step 1 – Transformation
 
 The bad news is that a transformer to [OSCAL](https://pages.nist.gov/OSCAL) is needed for each Cloud Service type.
 
 However, there is plenty of good news:
 
-- a transformer for your Cloud Service type may already exist, such as: [Tanium to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/tanium-result-to-oscal-ar.py), [OpenShift Compliance Operator to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/xccdf_result_to_oscal_ar.py)
+- a transformer for your Cloud Service type may already exist, such as: [Tanium to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/tanium_result_to_oscal_ar.py), [OpenShift Compliance Operator to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/xccdf_result_to_oscal_ar.py)
 - once a transformer for a Cloud Service type has been written, it can be open-sourced/re-used
-- writing a transformer is fairly easy: just a few lines of Python code using [trestle](https://oscal-compass.github.io/compliance-trestle/) as a foundation
+- writing a transformer is fairly easy: just a few lines of Python code using [trestle](../../index.md) as a foundation
 
 In the case of Tanium, the [OSCAL](https://pages.nist.gov/OSCAL) compliance data document is a *System Assessment Results* fragment with *Findings* and *Observations*, while in the case of OpenShift Compliance Operator there are *Observations* only.
 
-Tutorials are available to show you: how to [run a transformer](https://oscal-compass.github.io/compliance-trestle/tutorials/task.tanuim-to-oscal/transformation/), how to [write a transformer](https://oscal-compass.github.io/compliance-trestle/tutorials/task.transformer-construction/transformer-construction/).
+Tutorials are available to show you: how to [run a transformer](../task.tanium-result-to-oscal-ar/transformation.md), how to [write a transformer](../task.transformer-construction/transformer-construction.md).
 
 #### Step 2 – Reporting
 
-Coming soon is a [trestle](https://oscal-compass.github.io/compliance-trestle/) tool to assemble the [OSCAL](https://pages.nist.gov/OSCAL) fragments documents together using [OSCAL](https://pages.nist.gov/OSCAL) compliance configuration data (*System Assessment Plan*  and *System Security Plan*) into a complete *System Assessment Results*.
+Coming soon is a [trestle](../../index.md) tool to assemble the [OSCAL](https://pages.nist.gov/OSCAL) fragments documents together using [OSCAL](https://pages.nist.gov/OSCAL) compliance configuration data (*System Assessment Plan*  and *System Security Plan*) into a complete *System Assessment Results*.

docs/tutorials/task.ocp4-cis-profile-to-oscal-catalog/transformation.md

@@ -1,6 +1,6 @@
 # Tutorial: Setup for and use of ComplianceAsCode profile to OSCAL Catalog transformer
 
-Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Catalog](https://pages.nist.gov/OSCAL/reference/latest/catalog/json-outline/) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool.
+Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Catalog](https://pages.nist.gov/OSCAL-Reference/models/latest/catalog/json-outline/) using the [compliance-trestle](../../index.md) tool.
 
 ## *Objective*
 
@@ -12,7 +12,7 @@ The second is a one-command transformation from `.profile` to `OSCAL.json`.
 
 ## *Step 1: Install trestle in a Python virtual environment*
 
-Follow the instructions [here](https://oscal-compass.github.io/compliance-trestle/python_trestle_setup/) to install trestle in a virtual environment.
+Follow the instructions [here](../../python_trestle_setup.md) to install trestle in a virtual environment.
 
 ## *Step 2: Transform profile data (CIS benchmarks)*
 

docs/tutorials/task.ocp4-cis-profile-to-oscal-cd/transformation.md

@@ -1,6 +1,6 @@
 # Tutorial: Setup for and use of ComplianceAsCode profile to OSCAL Component Definition transformer
 
-Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Component Definition](https://pages.nist.gov/OSCAL/reference/latest/component-definition/json-outline/) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool.
+Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Component Definition](https://pages.nist.gov/OSCAL-Reference/models/v1.1.2/complete/json-reference/#/component-definition) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool.
 
 ## *Objective*
 

docs/tutorials/task.transformer-construction/transformer-construction.md

@@ -18,7 +18,7 @@ The objective here is to transform your compliance data into valid OSCAL, in par
 
 Examples of existing transformers  included with trestle are for the
 OpenShift Compliance Operator [OSCO](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/xccdf_result_to_oscal_ar.py) and
-[Tanium](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium-result-to-oscal-ar.py).
+[Tanium](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium_result_to_oscal_ar.py).
 
 ## *Overview*
 
@@ -407,7 +407,7 @@ There are 2 transformers in trestle.
 The [xccdf-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/xccdf_result_to_oscal_ar.py)
 transformer emits OSCAL Observations, the simplest partial result.
 
-The [tanium-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium-result-to-oscal-ar.py)
+The [tanium-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium_result_to_oscal_ar.py)
 transformer emits OSCAL Findings, a more complex partial result.
 
 Table of approximate lines of code.

mkdocs.yml

@@ -44,6 +44,7 @@ nav:
   - Trestle's object model: contributing/trestle_oscal_object_model.md
   - Developer Certificate of Originality: contributing/DCO.md
   - Trestle plugin mechanism: contributing/plugins.md
+  - GitHub actions setup: contributing/github_actions_setup.md
 - Known limitations: errors.md
 - Demos: demonstrations-content.md
 - Plugins:
@@ -192,11 +193,25 @@ nav:
       - transformer_helper: api_reference/trestle.transforms.transformer_helper.md
       - transformer_singleton: api_reference/trestle.transforms.transformer_singleton.md
 plugins:
+# warning don't use `macros`
 - search
+- autorefs
+- htmlproofer:
+    enabled: true
+    validate_rendered_template: false
+    validate_external_urls: true
+    raise_error_after_finish: false
+    raise_error_excludes:
+      # This is to remove some false positives for links which work.
+      # Anchors are validated again by core mkdocs
+      404: ['#trestle.*']
 - mkdocstrings:
+    default_handler: python
     handlers:
       python:
         options:
+          show_inheritance_diagram: true
+          inherited_members: false
           group_by_category: true
           show_category_heading: true
           show_if_no_docstring: true
@@ -206,8 +221,6 @@ plugins:
           - '!^__json'
           - '!^__config__'
           new_path_syntax: true
-    watch:
-    - trestle
 repo_name: oscal-compass/compliance-trestle
 repo_url: https://github.com/oscal-compass/compliance-trestle
 site_description: Documentation for compliance-trestle package.
@@ -222,3 +235,8 @@ theme:
     accent: purple
     primary: teal
     scheme: slate
+validation:
+  omitted_files: warn
+  absolute_links: warn # Or 'relative_to_docs' - new in MkDocs 1.6
+  unrecognized_links: warn
+  anchors: warn # New in MkDocs 1.6

setup.cfg

@@ -79,8 +79,9 @@ dev =
     types-requests
     types-setuptools
     # # Docs website
-    mkdocs==1.5.0
-    mkdocstrings[python-legacy]==0.19.0
+    mkdocs>=1.6.0
+    mkdocstrings[python]>=0.25.2
+    mkdocs-htmlproofer-plugin
     mkdocs-material
     markdown-include
     pymdown-extensions

trestle/common/model_utils.py

@@ -68,7 +68,7 @@ def load_distributed(
             Return a tuple of Model Type (e.g. class 'trestle.oscal.catalog.Catalog'),
             Model Alias (e.g. 'catalog.metadata') and Instance of the Model.
             If the model is decomposed/split/distributed, the instance of the model contains
-            the decomposed models loaded recursively.
+                the decomposed models loaded recursively.
 
         Note:
             This does not validate the model.  You must either validate the model separately or use the load_validate
@@ -171,7 +171,8 @@ def load_model_for_class(
         If you need to load an existing model but its content type may not be known, use this method.
         But the file content type should be specified if it is somehow known.
 
-        Note:  This does not validate the model.  If you want to validate the model use the load_validate utilities.
+        Note:
+            This does not validate the model.  If you want to validate the model use the load_validate utilities.
         """
         root_model_path = ModelUtils._root_path_for_top_level_model(
             trestle_root, model_name, model_class