Proposed roadmap towards a unified well documented API and best practices template for study add-on engineers

[motin]

This is a meta-issue and proposed roadmap towards the following objective and key result:

As a shield study add-on engineer, I want to be able to deliver QA:ed non-legacy add-ons using one single well documented study utils API using a well documented re-usable best practices template/boilerplate - regardless of the data processing pipeline (pioneer or not).

Today, we have two different study utils APIs (https://github.com/mozilla/shield-studies-addon-utils/ and https://github.com/mozilla/pioneer-utils/).
Study add-on engineers needs to choose one or the other depending on the characteristics of the data processing pipeline (Pioneer or not). Neither are very well documented and the best practices template is based solely on the API of shield-studies-addon-utils.

To make a studies experimental API that would expose all the necessary functionality would likely be the best course of action given that we would like to transition studies away from legacy add-ons and towards web extensions. (Bug 1419884 - [meta] Remove use of legacy add-ons within Mozilla)

Proposed roadmap:

Phase 1 - "documented re-usable best practices template"

This is ongoing and should be taken care of the following issues/PRs:

• Improvements sourced from several shield add-on repositories #49
• Basic doc improvements #113

UPDATE: These were merged 2018-03-10

Phase 2 - "non-legacy add-ons"

Make a studies experimental API in shield-studies-addon-utils that would expose all the necessary functionality directly to web extensions. Tracking issues:

• Expose the shield-utils API directly to web extensions using Web Extension Experiments
• Support Web Extension Experiments workflows in this template #53

By having this experimental API developed on the sidelines of the legacy study utils APIs, we ensure that stable development on both shield-studies-addon-utils and pioneer-utils can continue as normal and legacy add-ons can be developed using those APIs. There is no point in time where there will only be volatile APIs available.

Original status:

• @biancadanforth is setting up a prototype at https://github.com/biancadanforth/shield-study-webextension-experiment-template that will give us a good idea of what works and what does not work with web extension experiments
• We both are planning to experiment with this in upcoming shield study add-ons, implementing both the legacy and new experimental APIs - the legacy one for stability and the experimental one to progress towards moving away from legacy.

Timeline-related notes:
Bundled experiments are available to try out in development from Firefox 59 onwards.

Currently (as of 2018-03-24, 01:45), work is underway to convert the template, the example study therein and study-utils to WebExtension Experiments:

• mozilla/shield-studies-addon-utils - [develop] Start of version 5 - WebExtension Experiments #120
• https://github.com/mozilla/shield-studies-addon-template - [develop] Convert template and example study to use WebExtension Experiments #55

Phase 3 - "regardless of the data processing pipeline (pioneer or not)"

• Add pioneer-utils as a dependency to shield-studies-addon-utils and let the studies experimental API take care of choosing the appropriate legacy API methods of shield-studies-addon-utils and/or pioneer-utils depending on add-on configuration (Pioneer or not)
  AND/OR
• Merge pioneer-utils functionality into shield-studies-addon-utils - Request to merge / unfork: Shield-Studies-Addon-Utils #25

(From the perspective of an add-on engineer, any of the above approaches will work since the API will be the same from the web extension perspective. The first option will probably be necessary step to start with in case the merge is likely to happen in a more distant future)

Tracking issues:

• [v5] [develop] Support different data pipelines #142
• Merge ideas from Pioneer Utils #128

UPDATE 2018-09-10: A PR to solve both linked issues above is available at #263

Phase 4 - "deliver QA:ed"

We can begin with QA:ing a few add-ons that has implemented both the legacy and experimental APIs and have already been through QA with the legacy implementation. This would ensure that QA can follow the same test plan and expected functionality / telemetry to ensure that using the experimental APIs does not lead to any regressions.

Dependencies / blocking bugs:
UPDATE 2018-03-20: No longer blocking bugs, since we use webpack to bundle all modules instead:

• Bug 1442900 - Importing embedded JavaScript modules yields NS_NOINTERFACE when loading built web extensions including bundled experiments
  UPDATE 2018-08-28: No longer blocking bugs, since we have successfully launched studies using the experimental APIs:
• The ability to update add-ons/extensions for studies that are live (TODO: Link to issue/bug) - because if a breaking change needs to be made to the API we need a way to be able to roll out updates to the studies that rely on it.
• Mozilla signing routines to allow for bundled experiments to be deployed in Release. (TODO: Link to issue/bug)

Phase 5 - "well documented study utils API"

Once we have put a few web extension study add-ons through QA and know more about what the necessary functionality may be we can document the API properly and consider further actions, such as extending it and moving parts of it into tree - Bug 1414409 - Proposed list of "locked down" API's to allow Shield, Pioneer, Test Pilot to move to webExtensions only

UPDATE 2018-07-31: The experimental API:s have auto-generated API docs, and a PR with a v5-related clean-up of the documentation is available at #255

Related:
Bug 1280234 - Allow Telemetry to be sent to Mozilla from a WebExtension

[motin]

Ping @gregglind @rehandalal @biancadanforth

[biancadanforth]

This all sounds great! We should also think about what dependencies we have on other teams (thinking about the state of WebExtension Experiments, privileged API development, ...), and where that fits into these milestones. That may take some experimentation and collaboration.

Also one notable new requirement of Shield study development that is not listed here is in-tree testing. That process (which I have found a suboptimal way of doing thus far) needs to be streamlined and baked into the template with better automation and a lot of documentation.

Regarding mlopatka's request for timelines, I think a v1.0 of Phases 1 and 2 could be complete by Q2 or Q3 this year.

[motin]

Phase 2 is well underway: mozilla/shield-studies-addon-template#55

[motin]

We should also think about what dependencies we have on other teams (thinking about the state of WebExtension Experiments, privileged API development, ...), and where that fits into these milestones. That may take some experimentation and collaboration.

Added links to blocking / dependent bugs in each phase.

Also one notable new requirement of Shield study development that is not listed here is in-tree testing. That process (which I have found a suboptimal way of doing thus far) needs to be streamlined and baked into the template with better automation and a lot of documentation.

@biancadanforth Very true, is there a tracking bug for the in-tree testing?

Regarding mlopatka's request for timelines, I think a v1.0 of Phases 1 and 2 could be complete by Q2 or Q3 this year.

I believe that phases 1-2 can be done for a couple of first extensions in a week, and a good enough implementation of phase 3 within two weeks, but Phase 4 may take a lot of time to get together.

[motin]

Work is underway to convert the template, the example study therein and study-utils to WebExtension Experiments:

• mozilla/shield-studies-addon-utils - [develop] Start of version 5 - WebExtension Experiments #120
• https://github.com/mozilla/shield-studies-addon-template - [develop] Convert template and example study to use WebExtension Experiments #55

[motin]

Quick update: Phase 2 is under active development.
Tracking issues for Phase 3 created:

• [v5] [develop] Support different data pipelines #142
• Merge ideas from Pioneer Utils #128

[motin]

The utils-part of Phase 2 is now complete with utils v5 landing in master (#197)

[motin]

This meta-issue is very close to completion:

UPDATE 2018-07-31: The experimental API:s have auto-generated API docs, and a PR with a v5-related clean-up of the documentation is available at #255, so Phase 5 is close to be considered executed.

UPDATE 2018-08-28: Phase 2 and 4 are now executed, since we have successfully QA:ed and launched non-legacy studies using the experimental APIs.

UPDATE 2018-09-10: Pioneer-utils is on it's way into shield-studies-addon-utils #263, which when approved will mark Phase 3 as executed.

[motin]

Success! Closing this since Pioneer-utils is merged and published (#263), documentation has been cleaned up somewhat (#255), the auto-generated documentation describes the v5 API rather well and we are collecting relevant notes in the following document related to moving utils to tree: https://docs.google.com/spreadsheets/d/1w6zZKCxn68gTDDViO_s7ei8ITIlr1teEUi7rP_0ehAc/edit#gid=0