Add compliance config fetcher and check (#3)

CHANGES.md

@@ -1,3 +1,7 @@
+# 0.1.0
+
+- [NEW] Add compliance configuration fetcher and check.
+
 # 0.0.1
 
 - [NEW] Add abandoned evidence fetcher and check.

README.md

@@ -86,7 +86,7 @@ Follow these steps to integrate auditree-arboretum fetchers and checks into your
   For example to use the Abandoned Evidence fetcher from the `auditree` technology, add
   the following to your `fetch_auditree_technology.py`:
 
-  ```
+  ```python
   from arboretum.technology.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
   ```
 

arboretum/__init__.py

@@ -14,4 +14,4 @@
 # limitations under the License.
 """Arboretum - Checking your compliance & security posture, continuously."""
 
-__version__ = '0.0.1'
+__version__ = '0.1.0'

arboretum/technology/auditree/README.md

@@ -58,6 +58,23 @@ updated in over that `threshold` value otherwise the default is 30 days.  TTL is
    ```python
    from arboretum.technology.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
    ```
+### Compliance Configuration
+
+* Class: [ComplianceConfigFetcher][fetch-compliance-config]
+* Purpose: Writes the current configuration stored in the ComplianceConfig object to the evidence locker.
+* Behavior: Stores the configuration used to execute the compliance fetchers to the evidence locker and sets a time to
+live (TTL) to 2 hours.  This fetcher will refresh the configuration evidence on every execution of the fetchers.
+* Expected configuration elements:
+   * None
+* Expected configuration:
+   * None
+* Expected credentials:
+   * None
+* Import statement:
+
+   ```python
+   from arboretum.technology.auditree.fetchers.fetch_compliance_config import ComplianceConfigFetcher
+   ```
 
 ## Checks
 
@@ -121,8 +138,30 @@ the current check execution.  The default threshold is 30 days beyond the time t
    ```python
    from arboretum.technology.auditree.checks.test_abandoned_evidence import AbandonedEvidenceCheck
    ```
+### Compliance Configuration
 
+* Class: [ComplianceConfigCheck][check-compliance-config]
+* Purpose: Compare the configuration captured as evidence with the current configuration in the ComplianceConfig object
+being used to execute the checks.
+* Behavior: For every difference found between the evidence and the current configuration a failure is generated and
+reported on.
+* Evidence depended upon:
+   * Compliance tooling configuration settings
+      * `raw/auditree/compliance_config.json`
+      * Gathered by the `auditree` provider [ComplianceConfigFetcher][fetch-compliance-config]
+* Expected configuration elements:
+   * None
+* Expected configuration (optional):
+   * None
+
+* Import statement:
+
+   ```python
+   from auditree_central.provider.auditree.checks.test_compliance_config import ComplianceConfigCheck
+   ```
 
 [usage]: https://github.com/ComplianceAsCode/auditree-arboretum#usage
 [fetch-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/fetchers/fetch_abandoned_evidence.py
+[fetch-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/fetchers/fetch_compliance_config.py
 [check-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/checks/test_abandoned_evidence.py
+[check-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/checks/test_compliance_config.py

arboretum/technology/auditree/checks/test_compliance_config.py

@@ -0,0 +1,85 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Compliance execution configuration check."""
+
+import json
+
+from compliance.check import ComplianceCheck
+from compliance.evidence import DAY, ReportEvidence, with_raw_evidences
+
+
+class ComplianceConfigCheck(ComplianceCheck):
+    """
+    Compare running & historical configuration.
+
+    This check compares configuration captured by the fetcher as evidence with
+    the current configuration being used by the checks.
+    """
+
+    @property
+    def title(self):
+        """
+        Provide the title of the checks.
+
+        :returns: the title of the checks
+        """
+        return 'Compliance Configuration'
+
+    @classmethod
+    def setUpClass(cls):
+        """Initialize the check object with configuration settings."""
+        cls.config.add_evidences(
+            [
+                ReportEvidence(
+                    'compliance_config.md',
+                    'auditree',
+                    DAY,
+                    'Compliance repository configuration settings report.'
+                )
+            ]
+        )
+
+        return cls
+
+    @with_raw_evidences('auditree/compliance_config.json')
+    def test_compliance_configuration(self, evidence):
+        """Check that current config matches with config evidence."""
+        evidence_config = json.loads(evidence.content)
+        if evidence_config != self.config.raw_config:
+            evidence = json.dumps(evidence_config, indent=2).split('\n')
+            config = json.dumps(self.config.raw_config, indent=2).split('\n')
+            self.add_failures(
+                'Differences found',
+                {
+                    'Fetcher Configuration': evidence,
+                    'Check Configuration': config
+                }
+            )
+
+    def get_reports(self):
+        """
+        Provide the check report name.
+
+        :returns: the report(s) generated for this check
+        """
+        return ['auditree/compliance_config.md']
+
+    def get_notification_message(self):
+        """
+        Compliance configuration check notifier.
+
+        :returns: notification dictionary.
+        """
+        return {'body': None}

arboretum/technology/auditree/fetchers/fetch_compliance_config.py

@@ -0,0 +1,39 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Compliance execution configuration fetcher."""
+
+import json
+
+from compliance.evidence import HOUR, RawEvidence
+from compliance.fetch import ComplianceFetcher
+
+
+class ComplianceConfigFetcher(ComplianceFetcher):
+    """Fetch the current current compliance tooling configuration."""
+
+    def fetch_compliance_configuration(self):
+        """Fetch the compliance tooling configuration.
+
+        - Evidence is refreshed to the locker regardless of TTL
+        - Evidence is valid for the next two hours (for the check)
+        """
+        evidence = RawEvidence(
+            'compliance_config.json',
+            'auditree',
+            2 * HOUR,
+            'Compliance Configuration'
+        )
+        evidence.set_content(json.dumps(self.config.raw_config))
+        self.locker.add_evidence(evidence)

arboretum/technology/auditree/templates/reports/auditree/compliance_config.md.tmpl

@@ -0,0 +1,57 @@
+{#- -*- mode:jinja2; coding: utf-8 -*- -#}
+{#
+Copyright (c) 2020 IBM Corp. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+#}
+
+# {{ test.title }} Report {{ now.strftime('%Y-%m-%d') }}
+
+This report displays differences between the configuration used by the fetcher
+execution and the configuration used by the check execution.  Fetchers and checks
+are executed as separate processes but the expectation is that both processes
+are executed using the same configuration settings as defined by a project's
+configuration JSON file.  If discrepancies are found and reported here then a
+serious problem occurred during execution.
+
+## Remediation
+
+All discrepancies reported here need to be fully investigated and accounted for.
+Since the official execution of fetchers and checks is handled by a CI/CD tool
+like Travis, the execution environment should be a closed "eco-system" and there
+should be no reason for a difference in execution configuration between the
+fetchers and the checks.  In the event of a problem reported here, check your
+CI/CD build logs for more details.
+
+{% if test.total_issues_count(results) == 0 %}
+**No execution configuration discrepancies to report**
+{% else %}
+
+{% if test.failures_for_check_count(results) > 0 %}
+{% for k in all_failures.keys() -%}
+## {{ k }}
+
+{% for config_name, config_content in all_failures[k][0].items() %}
+### {{ config_name }}
+
+```
+{% for line in config_content %}
+{{ line|safe }}
+{%- endfor %}
+```
+
+{% endfor %}
+
+{%- endfor -%}
+{% endif -%}
+{% endif %}

controls.json

@@ -3,5 +3,10 @@
     "auditree_evidence": {
       "auditree_control": ["tech.auditree"]
     }
+  },
+  "arboretum.technology.auditree.checks.test_compliance_config.ComplianceConfigCheck": {
+    "auditree_evidence": {
+      "auditree_control": ["tech.auditree"]
+    }
   }
 }