Re-org repo folder structure (#8)

CHANGES.md

@@ -1,3 +1,14 @@
+# 0.3.0
+
+- [BREAKING] Move `auditree` fetchers and checks up to arboretum.auditree.
+- [NEW] Add folder hierarchy for Ansible fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for Chef fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for IBM Cloud fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for Kubernetes fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for Object Storage fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for Pager Duty fetchers, checks, and harvest reports.
+- [NEW] Add folder hierarchy for Splunk fetchers, checks, and harvest reports.
+
 # 0.2.0
 
 - [NEW] Add Python packages fetcher and check.

MANIFEST.in

@@ -1,4 +1,2 @@
-graft arboretum/provider/*/templates
-graft arboretum/technology/*/templates
-graft arboretum/provider/*/reports/report_templates
-graft arboretum/technology/*/reports/report_templates
+graft arboretum/*/templates
+graft arboretum/*/reports/report_templates

README.md

@@ -15,6 +15,22 @@ reports built upon the [Auditree compliance automation framework][auditree-frame
 
 ## Repo content
 
+### Functionality categorization
+
+Arboretum fetchers, checks, and Harvest reports are organized into functional
+grouping categories.  The following categories have either been contributed to
+or will be contributed to in the near future.  We anticipate that this list will
+grow as arboretum matures.
+
+- [Ansible](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/ansible)
+- [Auditree](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/auditree)
+- [Chef](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/chef)
+- [IBM Cloud](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/ibm_cloud)
+- [Kubernetes](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/kubernetes)
+- [Object Storage](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/object_storage)
+- [Pager Duty](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/pager_duty)
+- [Splunk](https://github.com/ComplianceAsCode/auditree-arboretum/tree/main/arboretum/splunk)
+
 ### Fetchers
 
 Please read the framework documentation for [fetcher design principles][] before
@@ -53,8 +69,8 @@ identified by checks.  These violations are in the form of failures and warnings
 ### Harvest Reports
 
 Harvest reports are hosted with the fetchers/checks that collect the evidence for
-the reports process. Within `auditree-arboretum` this means the code lives in the
-appropriate provider or technology directory. For more details check out
+the reports process. Within `auditree-arboretum` this means the harvest report code
+lives in `reports` folders throughout this repository. For more details check out
 [harvest report development][harvest-rpt-dev] in the [harvest][harvest] README.
 
 ## Usage
@@ -72,30 +88,30 @@ Follow these steps to integrate auditree-arboretum fetchers and checks into your
 
 * Add this `auditree-arboretum` package as a dependency in your Python project.
 * The following steps can be taken to import individual arboretum fetchers and checks.
-  * For a fetcher, add a `fetch_<category>_<technology|provider>.py` module, if one
-  does not already exist, in your project's `fetchers` path where the `<category>` is
-  either the provider or technology of that fetcher.  Having a separate common "category"
-  module guards against name collisions across providers and technologies.
-  * For a check, add a `test_<category>_<technology|provider>.py` module, if one
-  does not already exist, in your project's `checks` path where the `<category>` is
-  either the provider or technology of that check.  Having a separate common "category"
-  module guards against name collisions across providers and technologies.
+  * For a fetcher, add a `fetch_<category>_common.py` module, if one does not already
+  exist, in your project's `fetchers` path where the `<category>` is
+  the respective category folder within this repo of that fetcher.  Having a separate
+  common "category" module guards against name collisions across categories.
+  * For a check, add a `test_<category>_common.py` module, if one does not already exist,
+  in your project's `checks` path where the `<category>` is the respective category folder
+  within this repo of that check.  Having a separate common "category" module guards
+  against name collisions across providers and technologies.
   * Import the desired fetcher or check class and the `auditree-framework` will handle
   the rest.
 
-  For example to use the Abandoned Evidence fetcher from the `auditree` technology, add
-  the following to your `fetch_auditree_technology.py`:
+  For example to use the Abandoned Evidence fetcher from the `auditree` category, add
+  the following to your `fetch_auditree_common.py`:
 
   ```python
-  from arboretum.technology.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
+  from arboretum.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
   ```
 
 * `auditree-arboretum` fetchers and checks are designed to execute as part of a downstream
 Python project, so you may need to setup your project's configuration in order for the
-fetchers and checks to execute as desired.  Each provider and technology folder in this
-repository includes a README.md that documents each fetcher's and check's configuration.
+fetchers and checks to execute as desired.  Each category folder in this repository
+includes a README.md that documents each fetcher's and check's configuration.
   * In general `auditree-arboretum` fetchers and checks expect an `org` field with content
-  that capture each fetcher's and check's configuration settings.
+  that captures each fetcher's and check's configuration settings.
 
   For example:
 
@@ -123,9 +139,9 @@ set which is useful for organizing check notifications and targeted check execut
 
   ```json
   {
-    "arboretum.technology.auditree.checks.test_abandoned_evidence.AbandonedEvidenceCheck": {
+    "arboretum.auditree.checks.test_abandoned_evidence.AbandonedEvidenceCheck": {
       "auditree_evidence": {
-        "auditree_control": ["tech.auditree"]
+        "auditree_control": ["arboretum.auditree"]
       }
     }
   }

arboretum/__init__.py

@@ -14,4 +14,4 @@
 # limitations under the License.
 """Arboretum - Checking your compliance & security posture, continuously."""
 
-__version__ = '0.2.0'
+__version__ = '0.3.0'

arboretum/ansible/README.md

@@ -0,0 +1,27 @@
+# Ansible library
+
+The fetchers and checks contained within this `ansible` category folder are
+common tests that can be configured and executed for the purpose of generating
+compliance reports and notifications using the [auditree-framework][].  They
+validate the configuration and ensure smooth execution of an auditree instance.
+See [auditree-framework documentation](https://complianceascode.github.io/auditree-framework/)
+for more details.
+
+These tests are normally executed by a CI/CD system like
+[Travis CI](https://travis-ci.com/) as part of another project that uses this
+library package as a dependency.
+
+## Usage as a library
+
+See [usage][usage] for specifics on including this library as a dependency and
+how to include the fetchers and checks from this library in your downstream project.
+
+## Fetchers
+
+Fetchers coming soon...
+
+## Checks
+
+Checks coming soon...
+
+[usage]: https://github.com/ComplianceAsCode/auditree-arboretum#usage

arboretum/technology/auditree/checks/__init__.py → arboretum/ansible/__init__.py

@@ -12,4 +12,4 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""arboretum.technology.auditree validation checks."""
+"""Ansible fetchers, checks, and harvest reports."""

arboretum/provider/__init__.py → arboretum/ansible/checks/__init__.py

@@ -12,4 +12,4 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""Fetchers & checks associated to a specific vendor."""
+"""Ansible validation checks."""

arboretum/technology/auditree/fetchers/__init__.py → arboretum/ansible/evidences/__init__.py

@@ -12,4 +12,4 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""arboretum.technology.auditree evidence gathering fetchers."""
+"""Ansible evidence helper modules/classes."""

arboretum/technology/auditree/reports/__init__.py → arboretum/ansible/fetchers/__init__.py

@@ -12,4 +12,4 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""arboretum.technology.auditree harvest reports and templates."""
+"""Ansible evidence gathering fetchers."""

arboretum/ansible/reports/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Ansible harvest reports and templates."""

arboretum/technology/auditree/README.md → arboretum/auditree/README.md

@@ -1,6 +1,6 @@
-# Auditree technology library
+# Auditree library
 
-The fetchers and checks contained within this `auditree` technology folder are
+The fetchers and checks contained within this `auditree` category folder are
 common tests that can be configured and executed for the purpose of generating
 compliance reports and notifications using the [auditree-framework][].  They
 validate the configuration and ensure smooth execution of an auditree instance.
@@ -63,7 +63,7 @@ over that `threshold` value otherwise the default is 30 days.  TTL is set to 1 d
 * Import statement:
 
    ```python
-   from arboretum.technology.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
+   from arboretum.auditree.fetchers.fetch_abandoned_evidence import AbandonedEvidenceFetcher
    ```
 ### Compliance Configuration
 
@@ -83,7 +83,7 @@ the fetchers.
 * Import statement:
 
    ```python
-   from arboretum.technology.auditree.fetchers.fetch_compliance_config import ComplianceConfigFetcher
+   from arboretum.auditree.fetchers.fetch_compliance_config import ComplianceConfigFetcher
    ```
 
 ### Python Packages
@@ -103,7 +103,7 @@ live (TTL) is set to 1 day for all evidences.
 * Import statement:
 
    ```python
-   from arboretum.technology.auditree.fetchers.fetch_python_packages import PythonPackageFetcher
+   from arboretum.auditree.fetchers.fetch_python_packages import PythonPackageFetcher
    ```
 
 ## Checks
@@ -171,7 +171,7 @@ execution.  The default threshold is 30 days beyond the time to live (TTL) setti
 * Import statement:
 
    ```python
-   from arboretum.technology.auditree.checks.test_abandoned_evidence import AbandonedEvidenceCheck
+   from arboretum.auditree.checks.test_abandoned_evidence import AbandonedEvidenceCheck
    ```
 ### Compliance Configuration
 
@@ -192,7 +192,7 @@ configuration a failure is generated and reported on.
 * Import statement:
 
    ```python
-   from auditree_central.provider.auditree.checks.test_compliance_config import ComplianceConfigCheck
+   from arboretum.auditree.checks.test_compliance_config import ComplianceConfigCheck
    ```
 
 ### Python Packages
@@ -221,13 +221,13 @@ used are not at the current release version.
 * Import statement:
 
    ```python
-   from arboretum.technology.auditree.checks.test_python_packages import PythonPackageCheck
+   from arboretum.auditree.checks.test_python_packages import PythonPackageCheck
    ```
 
 [usage]: https://github.com/ComplianceAsCode/auditree-arboretum#usage
-[fetch-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/fetchers/fetch_abandoned_evidence.py
-[fetch-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/fetchers/fetch_compliance_config.py
-[fetch-python-packages]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/fetchers/fetch_python_packages.py
-[check-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/checks/test_abandoned_evidence.py
-[check-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/checks/test_compliance_config.py
-[check-python-packages]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/technology/auditree/checks/test_python_packages.py
+[fetch-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/fetchers/fetch_abandoned_evidence.py
+[fetch-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/fetchers/fetch_compliance_config.py
+[fetch-python-packages]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/fetchers/fetch_python_packages.py
+[check-abandoned-evidence]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_abandoned_evidence.py
+[check-compliance-config]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_compliance_config.py
+[check-python-packages]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/auditree/checks/test_python_packages.py

arboretum/auditree/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Auditree fetchers, checks, and harvest reports."""

arboretum/auditree/checks/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Auditree validation checks."""

arboretum/technology/auditree/checks/test_python_packages.py → arboretum/auditree/checks/test_python_packages.py

@@ -17,11 +17,13 @@
 import json
 from datetime import datetime, timedelta
 
+from arboretum.auditree.evidences.python_package_release import (
+    PackageReleaseEvidence
+)
+
 from compliance.check import ComplianceCheck
 from compliance.evidence import DAY, ReportEvidence, with_raw_evidences
 
-from ..evidences.python_package_release import PackageReleaseEvidence
-
 
 class PythonPackageCheck(ComplianceCheck):
     """Compare the software versions used in Auditree execution."""
@@ -105,6 +107,10 @@ def test_auditree_framework_version(self, packages, releases):
         """Check auditree-framework version matches latest release."""
         self._test_versions(packages, releases, 'auditree-framework')
 
+    @with_raw_evidences(
+        'auditree/python_packages.json',
+        'auditree/auditree_harvest_releases.xml'
+    )
     def test_auditree_harvest_version(self, packages, releases):
         """Check auditree-harvest version matches latest release."""
         self._test_versions(packages, releases, 'auditree-harvest')

arboretum/auditree/evidences/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Auditree evidence helper modules/classes."""

arboretum/auditree/fetchers/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Auditree evidence gathering fetchers."""

arboretum/technology/auditree/fetchers/fetch_python_packages.py → arboretum/auditree/fetchers/fetch_python_packages.py

@@ -16,10 +16,10 @@
 
 import json
 
-from arboretum.common.constants import PYPI_RSS_BASE_URL
-from arboretum.technology.auditree.evidences.python_package_release import (
+from arboretum.auditree.evidences.python_package_release import (
     PackageReleaseEvidence
 )
+from arboretum.common.constants import PYPI_RSS_BASE_URL
 
 from compliance.evidence import DAY, RawEvidence, store_raw_evidence
 from compliance.fetch import ComplianceFetcher

arboretum/auditree/reports/__init__.py

@@ -0,0 +1,15 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2020 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Auditree harvest reports and templates."""