Merge pull request #105 from openedx/jill/plugin-structure

ADR 8. Project structure

README.md

@@ -8,8 +8,12 @@ Aspects is an optional implementation of analytics for the Open edX LMS. It is t
 
 ## What _isn't_ Aspects?
 
-A deployable application in-and-of itself, it helps deploy other applications together using Tutor plugins and customizations to the Open edX platform to connect well-supported, existing, third party applications.
+A deployable application in-and-of itself, it helps deploy other applications together using a Tutor plugin and customizations to the Open edX platform to connect well-supported, existing, third party applications.
 
 ## Status
 
-Aspects is in development by efforts from Axim Collaborative and Open edX community contributors. You can follow design, development, and implementation at the [Data Working Group issue board](https://github.com/orgs/openedx/projects/5/views/1).
+Aspects is in development by efforts from Axim Collaborative and Open edX community contributors. For more information, see:
+
+* [Data Working Group issue board](https://github.com/orgs/openedx/projects/5/views/1): project design, development, and implementation
+* [Aspects Documentation](https://docs.openedx.org/projects/openedx-aspects): project documentation
+* [tutor-contrib-aspects](https://github.com/openedx/tutor-contrib-aspects): deploys Aspects using Tutor.

docs/decisions/0006_areas_of_responsibility.rst

@@ -4,7 +4,7 @@
 Status
 ******
 
-Accepted
+Superseded by `ADR 8 Project structure`_.
 
 Context
 *******
@@ -106,3 +106,6 @@ ClickHouse schema which Ralph writes to part of, but the rest is managed elsewhe
 
 This would also leak Aspects implementation details into all of the plugins, reducing their
 flexibility for other use cases.
+
+
+.. _ADR 8 Project structure: 0008_project_structure.rst

docs/decisions/0008_project_structure.rst

@@ -0,0 +1,111 @@
+8. Project Structure
+####################
+
+Status
+******
+
+Accepted
+
+Context
+*******
+
+The Aspects Analytics system (Aspects) consists of several pieces of technology working together via
+some configuration and scripting. Decisions around where to store data and configuration need to be
+made so that developers understand where to make changes and look for the causes of issues. These
+decisions can have wide ranging impact on things such as extensibility and configurability of the
+system as a whole.
+
+This ADR offers guiding principles to address these issues:
+
+* The number of plugins and systems in play can make it difficult to know where to look for
+  configuration or add new features.
+* Initialization steps for the third-party systems are sometimes intricate and dependent on other
+  systems, so Aspects deployment needs to control exactly what happens when.
+* In the future, Aspects will offer a non-Tutor deployment option for experienced operators.
+
+Decisions
+*********
+
+**Aspects repo**
+
+This repo (`openedx-aspects`_) remains the primary Aspects repository where the project decisions
+and documentation are stored.
+
+This repo should also be used to store any common code or configuration between Tutor and non-Tutor
+Aspects deployments.
+
+**Single Aspects Tutor plugin**
+
+All of the third-party services that Aspects requires can be deployed and configured by a single
+`tutor-contrib-aspects`_ plugin.
+
+This plugin employs the following structure to delineate the different areas of responsibility::
+
+    tutoraspects/
+    ├── patches
+    └── templates
+        ├── aspects
+        │   ├── apps
+        │   │   ├── aspects
+        │   │   ├── clickhouse
+        │   │   ├── ralph
+        │   │   ├── superset
+        │   │   └── vector
+        │   ├── build
+        │   │   ├── aspects
+        │   │   └── aspects-superset
+        │   │       └── openedx-assets
+        │   └── jobs
+        │       └── init
+        │           ├── aspects
+        │           ├── clickhouse
+        │           ├── lms
+        │           ├── mysql
+        │           └── superset
+        └── openedx-assets
+
+This structure is guided by the following principles:
+
+* Configuration or functionality specific to a single service should live under a directory named
+  for that service.
+
+  E.g. `templates/aspects/apps/clickhouse` contains deployment templates related to Clickhouse,
+  and `templates/aspects/jobs/init/clickhouse` contains the Clickhouse initialization scripts.
+
+* Configuration or functionality that spans multiple services should live under an `aspects` directory.
+
+  E.g. `templates/aspects/apps/aspects` contains Aspects-specific database migrations (Clickhouse
+  migrations handled by Alembic) and dbt profiles (data transformation layer).
+
+* Templates for Superset assets (dashboards, charts, etc.) live under the
+  `templates/aspects/build/aspects-superset/openedx-assets` directory to facilitate localization of
+  this user-facing content.
+
+  These asset templates utilize shared, non-localized files stored under `templates/openedx-assets`.
+
+**LMS plugins**
+
+Aspects depends on specific versions of a few Django plugins to move events from the LMS into its
+processing pipeline (currently `openedx-event-sink-clickhouse`_ and `event-routing-backends`_).
+These plugins should live in their own repositories, and be installed to the LMS/CMS as "extra
+requirements", using the chosen deployment's best practices. For example, the Tutor plugin installs
+these dependencies to the LMS+CMS using the `OPENEDX_EXTRA_PIP_REQUIREMENTS` variable.
+
+Consequences
+************
+
+* The service-based Tutor plugins (``tutor-contrib-clickhouse``, ``tutor-contrib-ralph``, ``tutor-contrib-superset``) are consolidated into `tutor-contrib-aspects`_ and archived under `openedx-unsupported`_.
+
+Rejected Alternatives
+*********************
+
+**Separate plugins for each third-party service**
+
+Supercedes `ADR 6 Areas of responsibility`_.
+
+.. _ADR 6 Areas of responsibility: 0006_areas_of_responsibility.rst
+.. _event-routing-backends: https://github.com/openedx/event-routing-backends
+.. _openedx-aspects: https://github.com/openedx/openedx-aspects
+.. _openedx-event-sink-clickhouse: https://github.com/openedx/openedx-event-sink-clickhouse
+.. _openedx-unsupported: https://github.com/openedx-unsupported
+.. _tutor-contrib-aspects: https://github.com/openedx/tutor-contrib-aspects