diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4dc441786..2757a4db9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -260,4 +260,4 @@ ______________________________________________________________________ ##### Overview of process to take OSCAL models and upgrade trestle Python code - +![](images/trestle-OSCAL-upgrade.png) diff --git a/Makefile b/Makefile index 0482664ff..362e6e68d 100644 --- a/Makefile +++ b/Makefile @@ -86,7 +86,7 @@ docs-automation:: python ./scripts/website_automation.py docs-validate:: docs-automation - mkdocs build -c -s + mkdocs build -v -c -s rm -rf site docs-serve: docs-automation @@ -117,4 +117,4 @@ pylint-test: pylint tests --rcfile=.pylintrc_tests check-for-changes: - python scripts/have_files_changed.py -u \ No newline at end of file + python scripts/have_files_changed.py -u diff --git a/docs/cli.md b/docs/cli.md index f7ef55b9e..bd44ac92e 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -113,11 +113,11 @@ Users can query the contents of files using `trestle describe`, and probe the co OSCAL models are rich and contain multiple nested data structures. Given this, a mechanism is required to address _elements_ /_attributes_ within an oscal object. -This accessing method is called 'element path' and is similar to _jsonPath_. Commands provide element path by a `-e` argument where available, e.g. trestle split -f catalog.json -e 'catalog.metadata.\*'. This path is used whenever specifying an attribute or model, rather than exposing trestle's underlying object model name. Users can refer to [NIST's json outline](https://pages.nist.gov/OSCAL/reference/latest/complete/json-outline/) to understand object names in trestle. +This accessing method is called 'element path' and is similar to _jsonPath_. Commands provide element path by a `-e` argument where available, e.g. trestle split -f catalog.json -e 'catalog.metadata.\*'. This path is used whenever specifying an attribute or model, rather than exposing trestle's underlying object model name. Users can refer to [NIST's json outline](https://pages.nist.gov/OSCAL-Reference/models/latest/complete/json-outline/) to understand object names in trestle. ### Rules for element path -1. Element path is an expression of the attribute names, [in json form](https://pages.nist.gov/OSCAL/reference/latest/complete/json-outline/) , concatenated by a period (`.`). +1. Element path is an expression of the attribute names, [in json form](https://pages.nist.gov/OSCAL-Reference/models/latest/complete/json-outline/) , concatenated by a period (`.`). 1. E.g. The metadata in a catalog is referred to as `catalog.metadata` 1. Element paths are relative to the file. 1. e.g. For `metadata.json` roles would be referred to as `metadata.roles`, from the catalog file that would be `catalog.metadata.roles` diff --git a/docs/contributing/images/trestle-OSCAL-upgrade.png b/docs/contributing/images/trestle-OSCAL-upgrade.png new file mode 100644 index 000000000..63e73799f Binary files /dev/null and b/docs/contributing/images/trestle-OSCAL-upgrade.png differ diff --git a/docs/contributing/trestle_oscal_object_model.md b/docs/contributing/trestle_oscal_object_model.md index 71083d6d2..a77b47a4f 100644 --- a/docs/contributing/trestle_oscal_object_model.md +++ b/docs/contributing/trestle_oscal_object_model.md @@ -9,7 +9,7 @@ This functionality, which is built on [pydantic](https://pydantic-docs.helpmanua ## Mapping and variance with OSCAL names. -The underlying object model that trestle relies on is the json schema published by NIST [here](https://github.com/usnistgov/OSCAL/tree/main/json/schema). In understanding these models the [model reference page](https://pages.nist.gov/OSCAL/reference/1.0.0/) is an indispensable source. +The underlying object model that trestle relies on is the json schema published by NIST [here](https://github.com/usnistgov/OSCAL/releases/latest). In understanding these models the [model reference page](https://pages.nist.gov/OSCAL-Reference/models/) is an indispensable source. When generating the python data class based models we have tried to be as faithful as we can to the naming convention provided by OSCAL. This is the hierarchy of rules that we have used: diff --git a/docs/index.md b/docs/index.md index 56ef2dfe4..3759cecaf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -65,7 +65,7 @@ natively supports only `json` and `yaml` formats at this time. Future roadmap anticipates that support for xml [import](https://github.com/oscal-compass/compliance-trestle/issues/177) and [upstream references](https://github.com/oscal-compass/compliance-trestle/issues/178) will be enabled. However, it is expected that full support will remain only for `json` and `yaml`. -Users needing to import XML OSCAL artifacts are recommended to look at NIST's XML to json conversion page [here](https://github.com/usnistgov/OSCAL/tree/master/json#oscal-xml-to-json-converters). +Users needing to import XML OSCAL artifacts are recommended to look at NIST's OSCAL converters page [here](https://github.com/usnistgov/OSCAL/blob/main/build/README.md#converters). ## Python codebase, easy installation via pip @@ -77,12 +77,12 @@ Compliance trestle is currently stable and is based on NIST OSCAL version 1.0.4, ## Contributing to Trestle -Our project welcomes external contributions. Please consult [contributing](contributing/mkdocs_contributing/) to get started. +Our project welcomes external contributions. Please consult [contributing](contributing/mkdocs_contributing.md) to get started. ## License & Authors -If you would like to see the detailed LICENSE click [here](license/). -Consult [contributors](https://github.com/oscal-compass/compliance-trestle/graphs/contributors) for a list of authors and [maintainers](maintainers/) for the core team. +If you would like to see the detailed LICENSE click [here](license.md). +Consult [contributors](https://github.com/oscal-compass/compliance-trestle/graphs/contributors) for a list of authors and [maintainers](maintainers.md) for the core team. ```text # Copyright (c) 2024 The OSCAL Compass Authors. All rights reserved. diff --git a/docs/tutorials/continuous-compliance/continuous-compliance.md b/docs/tutorials/continuous-compliance/continuous-compliance.md index 0bd7e0d3e..a8f5b0c8c 100644 --- a/docs/tutorials/continuous-compliance/continuous-compliance.md +++ b/docs/tutorials/continuous-compliance/continuous-compliance.md @@ -17,7 +17,7 @@ Moreover, assuring continuous compliance across multiple cloud vendors can compl Common sense dictates that standardization would simplify matters. The National Institute of Standards and Technologies (NIST) is developing the Open Security Controls Assessment Language ([OSCAL](https://pages.nist.gov/OSCAL)). -The compliance-[trestle](https://oscal-compass.github.io/compliance-trestle/) open source github project is an effort to employ [OSCAL](https://pages.nist.gov/OSCAL) for compliance standardization and automation. Of great utility is the [trestle](https://oscal-compass.github.io/compliance-trestle/) oscal module that facilitates transformation of data to/from Python object representations in accordance with the [OSCAL](https://pages.nist.gov/OSCAL) schemas. +The compliance-[trestle](../../index.md) open source github project is an effort to employ [OSCAL](https://pages.nist.gov/OSCAL) for compliance standardization and automation. Of great utility is the [trestle](../../index.md) oscal module that facilitates transformation of data to/from Python object representations in accordance with the [OSCAL](https://pages.nist.gov/OSCAL) schemas. #### Simple Continuous Compliance Architecture @@ -25,7 +25,7 @@ The compliance-[trestle](https://oscal-compass.github.io/compliance-trestle/) op Cloud Services can often be configured to monitor (and sometimes enforce) policies. Examples include OpenShift Compliance Operator and Tanium. However, the compliance reporting “raw” data produced is unique to each. -Two steps are needed to ascertain your compliance posture. Step 1 is to transform available compliance “raw” data into standardized form ([OSCAL](https://pages.nist.gov/OSCAL)). Step 2 is to examine the [OSCAL](https://pages.nist.gov/OSCAL) data and assemble a compliance posture for the controls and components of interest. And [trestle](https://oscal-compass.github.io/compliance-trestle/) is the go-to solution. +Two steps are needed to ascertain your compliance posture. Step 1 is to transform available compliance “raw” data into standardized form ([OSCAL](https://pages.nist.gov/OSCAL)). Step 2 is to examine the [OSCAL](https://pages.nist.gov/OSCAL) data and assemble a compliance posture for the controls and components of interest. And [trestle](../../index.md) is the go-to solution. #### Step 1 – Transformation @@ -33,14 +33,14 @@ The bad news is that a transformer to [OSCAL](https://pages.nist.gov/OSCAL) is n However, there is plenty of good news: -- a transformer for your Cloud Service type may already exist, such as: [Tanium to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/tanium-result-to-oscal-ar.py), [OpenShift Compliance Operator to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/xccdf_result_to_oscal_ar.py) +- a transformer for your Cloud Service type may already exist, such as: [Tanium to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/tanium_result_to_oscal_ar.py), [OpenShift Compliance Operator to OSCAL](https://github.com/oscal-compass/compliance-trestle/blob/main/trestle/tasks/xccdf_result_to_oscal_ar.py) - once a transformer for a Cloud Service type has been written, it can be open-sourced/re-used -- writing a transformer is fairly easy: just a few lines of Python code using [trestle](https://oscal-compass.github.io/compliance-trestle/) as a foundation +- writing a transformer is fairly easy: just a few lines of Python code using [trestle](../../index.md) as a foundation In the case of Tanium, the [OSCAL](https://pages.nist.gov/OSCAL) compliance data document is a *System Assessment Results* fragment with *Findings* and *Observations*, while in the case of OpenShift Compliance Operator there are *Observations* only. -Tutorials are available to show you: how to [run a transformer](https://oscal-compass.github.io/compliance-trestle/tutorials/task.tanuim-to-oscal/transformation/), how to [write a transformer](https://oscal-compass.github.io/compliance-trestle/tutorials/task.transformer-construction/transformer-construction/). +Tutorials are available to show you: how to [run a transformer](../task.tanium-result-to-oscal-ar/transformation.md), how to [write a transformer](../task.transformer-construction/transformer-construction.md). #### Step 2 – Reporting -Coming soon is a [trestle](https://oscal-compass.github.io/compliance-trestle/) tool to assemble the [OSCAL](https://pages.nist.gov/OSCAL) fragments documents together using [OSCAL](https://pages.nist.gov/OSCAL) compliance configuration data (*System Assessment Plan* and *System Security Plan*) into a complete *System Assessment Results*. +Coming soon is a [trestle](../../index.md) tool to assemble the [OSCAL](https://pages.nist.gov/OSCAL) fragments documents together using [OSCAL](https://pages.nist.gov/OSCAL) compliance configuration data (*System Assessment Plan* and *System Security Plan*) into a complete *System Assessment Results*. diff --git a/docs/tutorials/task.ocp4-cis-profile-to-oscal-catalog/transformation.md b/docs/tutorials/task.ocp4-cis-profile-to-oscal-catalog/transformation.md index 5b36df015..00a669a2e 100644 --- a/docs/tutorials/task.ocp4-cis-profile-to-oscal-catalog/transformation.md +++ b/docs/tutorials/task.ocp4-cis-profile-to-oscal-catalog/transformation.md @@ -1,6 +1,6 @@ # Tutorial: Setup for and use of ComplianceAsCode profile to OSCAL Catalog transformer -Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Catalog](https://pages.nist.gov/OSCAL/reference/latest/catalog/json-outline/) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool. +Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Catalog](https://pages.nist.gov/OSCAL-Reference/models/latest/catalog/json-outline/) using the [compliance-trestle](../../index.md) tool. ## *Objective* @@ -12,7 +12,7 @@ The second is a one-command transformation from `.profile` to `OSCAL.json`. ## *Step 1: Install trestle in a Python virtual environment* -Follow the instructions [here](https://oscal-compass.github.io/compliance-trestle/python_trestle_setup/) to install trestle in a virtual environment. +Follow the instructions [here](../../python_trestle_setup.md) to install trestle in a virtual environment. ## *Step 2: Transform profile data (CIS benchmarks)* diff --git a/docs/tutorials/task.ocp4-cis-profile-to-oscal-cd/transformation.md b/docs/tutorials/task.ocp4-cis-profile-to-oscal-cd/transformation.md index 23184aea1..58dcef079 100644 --- a/docs/tutorials/task.ocp4-cis-profile-to-oscal-cd/transformation.md +++ b/docs/tutorials/task.ocp4-cis-profile-to-oscal-cd/transformation.md @@ -1,6 +1,6 @@ # Tutorial: Setup for and use of ComplianceAsCode profile to OSCAL Component Definition transformer -Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Component Definition](https://pages.nist.gov/OSCAL/reference/latest/component-definition/json-outline/) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool. +Here are step by step instructions for setup and transformation of [ComplianceAsCode](https://github.com/ComplianceAsCode/content) profile data files into [NIST](https://www.nist.gov/) standard [OSCAL](https://pages.nist.gov/OSCAL/) [Component Definition](https://pages.nist.gov/OSCAL-Reference/models/v1.1.2/complete/json-reference/#/component-definition) using the [compliance-trestle](https://oscal-compass.github.io/compliance-trestle/) tool. ## *Objective* diff --git a/docs/tutorials/task.transformer-construction/transformer-construction.md b/docs/tutorials/task.transformer-construction/transformer-construction.md index b6924437c..e5573cd5e 100644 --- a/docs/tutorials/task.transformer-construction/transformer-construction.md +++ b/docs/tutorials/task.transformer-construction/transformer-construction.md @@ -18,7 +18,7 @@ The objective here is to transform your compliance data into valid OSCAL, in par Examples of existing transformers included with trestle are for the OpenShift Compliance Operator [OSCO](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/xccdf_result_to_oscal_ar.py) and -[Tanium](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium-result-to-oscal-ar.py). +[Tanium](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium_result_to_oscal_ar.py). ## *Overview* @@ -407,7 +407,7 @@ There are 2 transformers in trestle. The [xccdf-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/xccdf_result_to_oscal_ar.py) transformer emits OSCAL Observations, the simplest partial result. -The [tanium-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium-result-to-oscal-ar.py) +The [tanium-result-to-oscal-ar](https://github.com/oscal-compass/compliance-trestle/blob/develop/trestle/tasks/tanium_result_to_oscal_ar.py) transformer emits OSCAL Findings, a more complex partial result. Table of approximate lines of code. diff --git a/images/trestle-OSCAL-upgrade.png b/images/trestle-OSCAL-upgrade.png index eae3bb6ef..63e73799f 100644 Binary files a/images/trestle-OSCAL-upgrade.png and b/images/trestle-OSCAL-upgrade.png differ diff --git a/mkdocs.yml b/mkdocs.yml index 26b7a9821..c011c6728 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -44,6 +44,7 @@ nav: - Trestle's object model: contributing/trestle_oscal_object_model.md - Developer Certificate of Originality: contributing/DCO.md - Trestle plugin mechanism: contributing/plugins.md + - GitHub actions setup: contributing/github_actions_setup.md - Known limitations: errors.md - Demos: demonstrations-content.md - Plugins: @@ -192,11 +193,25 @@ nav: - transformer_helper: api_reference/trestle.transforms.transformer_helper.md - transformer_singleton: api_reference/trestle.transforms.transformer_singleton.md plugins: +# warning don't use `macros` - search +- autorefs +- htmlproofer: + enabled: true + validate_rendered_template: false + validate_external_urls: true + raise_error_after_finish: false + raise_error_excludes: + # This is to remove some false positives for links which work. + # Anchors are validated again by core mkdocs + 404: ['#trestle.*'] - mkdocstrings: + default_handler: python handlers: python: options: + show_inheritance_diagram: true + inherited_members: false group_by_category: true show_category_heading: true show_if_no_docstring: true @@ -206,8 +221,6 @@ plugins: - '!^__json' - '!^__config__' new_path_syntax: true - watch: - - trestle repo_name: oscal-compass/compliance-trestle repo_url: https://github.com/oscal-compass/compliance-trestle site_description: Documentation for compliance-trestle package. @@ -222,3 +235,8 @@ theme: accent: purple primary: teal scheme: slate +validation: + omitted_files: warn + absolute_links: warn # Or 'relative_to_docs' - new in MkDocs 1.6 + unrecognized_links: warn + anchors: warn # New in MkDocs 1.6 diff --git a/setup.cfg b/setup.cfg index 116f70029..c1b26c678 100644 --- a/setup.cfg +++ b/setup.cfg @@ -79,8 +79,9 @@ dev = types-requests types-setuptools # # Docs website - mkdocs==1.5.0 - mkdocstrings[python-legacy]==0.19.0 + mkdocs>=1.6.0 + mkdocstrings[python]>=0.25.2 + mkdocs-htmlproofer-plugin mkdocs-material markdown-include pymdown-extensions diff --git a/trestle/common/model_utils.py b/trestle/common/model_utils.py index d24991abd..47e901b7c 100644 --- a/trestle/common/model_utils.py +++ b/trestle/common/model_utils.py @@ -68,7 +68,7 @@ def load_distributed( Return a tuple of Model Type (e.g. class 'trestle.oscal.catalog.Catalog'), Model Alias (e.g. 'catalog.metadata') and Instance of the Model. If the model is decomposed/split/distributed, the instance of the model contains - the decomposed models loaded recursively. + the decomposed models loaded recursively. Note: This does not validate the model. You must either validate the model separately or use the load_validate @@ -171,7 +171,8 @@ def load_model_for_class( If you need to load an existing model but its content type may not be known, use this method. But the file content type should be specified if it is somehow known. - Note: This does not validate the model. If you want to validate the model use the load_validate utilities. + Note: + This does not validate the model. If you want to validate the model use the load_validate utilities. """ root_model_path = ModelUtils._root_path_for_top_level_model( trestle_root, model_name, model_class