[v5] Finalize v5 api

[gregglind]

Yes

• studyConfig (with modifications)
• reading firefox prefs, but probably not setting them. (permissions, testing branches)
• make sure we send correct ids to surveys. (see investigate if for every participant we should have a survey reponse #36)
• builds a dist/ directory that you put in your addon.
• telemetryEnvironment.setExperimentActive (3 field version)
• deterministicVariation make sure we really catch errors, maybe rename to magic or usual as prefix.
• better endStudy api and config options. (see nit in endStudy if reason,fullname #91)

Maybe:

• have cli to lint study config / jsonschema for such more tools.
• tie and document with SG. (ask RRayborn)
• catch addon startup errors using @Osmose error collector. (v3 of shield packet #55, point 13)
• support queries for 'clients alive in study today'

NO:

• know other studies running enh - addons should check if another study is running, opt out if so. #16

Notes, unfinished.

glossary:

• ssau shield-studies-addon-utils
• ssat shield-studies-addon-template

shield-utils (browser.study.)

• chooseBranch (weights, optional rng?)
• configure // alternative is to new an object
  • shield or pioneer
  • name
  • weights?
  • variation?
  • ...
  • rules: pioneer, or isPioneer: true, or something!
• permissions // STUDY IMPLEMENTER DON'T HAVE TO REMEMBER ALL THE PREFS
  • OK? "some boolean is summary for basic isEligble"
  • details
    • auto pb mode?
    • I can shield (my shield pref is true)
    • I can telemetry
• telemetry (must handle pioneer as well)
  • send
  • configure?
  • setActive
  • unsetActive
• userInfo?? // alternative is to get it from normandy modules
  • (needed? for survey urls, pings)
  • get
    • fx version
    • addons
    • ...
• endStudy(...anEnding)
• utils
  • surveyQueryArgs. => (get the args for survey?) openUrl? construct url?
• sampling
  • hashing
  • unbalanced weighted selection
• (proposed)
  • expiration
• prefs
  • get
  • set

Example MVP Bootstrap.js

// alt extension prefs manager?
const studyConfig = {
    weights: 

}

studySetup.expired = await Study.hasExpired();  // per-session, but should be an observer / timer 

const theBranchPref = `shield.${addonWidgetId}.testingBranch`;
mybranch = await browser.shield.prefs.getStringPref(thePref) || await browser.shield.chooseVariation(weights);
studySetup.eligible = await Study.isEligible();
const { variation } = await browser.shieldUtils.info();
await browser.shield.configure(...studySetup)  // implies one study per addon

// handle install vs. regular run. 
await browser.shield.attemptRun()

await browser.shield.telemetry(aPing)
await browser.shield.endStudy(...anEnding, uninstall)
// endSTudy, ps has scary side effects, describe them.

NOT

// claim:  this seems non-idiomatic
const aStudy = browser.shield.createStudy(...options);
// benefit: cant accidentally call methods that need config

Big picture for template and tools

• WebExtensionExperiment NOT Legacy addon
• v5 for SSAU, SSAT -
• GITHUB:
  • SSAU - develop branch issue [develop] Start of version 5 - WebExtension Experiments #120
    • milestone v5 tbd
  • SSAT - develop branch issue Fixes two small typos in the docs #53
    • milestone v5 wee tbd
• Linting Exists
  • PdH does both (What's a good way to force expiry for test purposes (e.g. test end of study survey) #61)
• USE THE MOZILLA INFRA for CI and testing if possible (except Github)
  • Unit Tests for sample study
  • CI on CircleCI (unit tests) / Task Cluster ? ???? (jmaher@)
    • SSAU
    • SSAT
  • TryServer
    • SSAT
• merge with pioneer (what is that?)
  • Send Telemetry
  • Different methods for sample
• Team: glind, bdanforth, pdh, motin, rhelmer, pdahiya, raymak, rdalal,
• Does endStudy uninstall the addon? Probably
• branch choice persistence preferred mechanism
  • restart, get same branch
  • testers can choose a branch (reasonably)
  • mechanism for branch is mostly not magic
    • chooseBranch(weights, unit test optional unityFractions) = R cut

[Osmose]

catch addon startup erros using @Osmose error collector. (#55, point 13)

There's not really anything you need to do for getting these collected; as long as they're errors that get logged to the browser console they'll get picked up (I was mixing up a different feature when I was saying you need to register something).

The bigger problem is that getting access and noticing your errors. Error collection is still just a prototype and will be removed from Firefox in Q2 or Q3. Access is limited to a subset of approved employees, and Sentry has nothing for setting up alerts for errors related only to your code. Also, errors are only collected from users on the Nightly channel and at a sample rate of 0.1% of total errors that occur. So unless your study is going to 100% of Nightly users and has a common error, the current prototype collection won't help you.

[gregglind]

Proposed approach:

1. Build this is in yaml, not json, for the sake of easy diff.
2. Convert with npm install -g yamljs (yaml2json the file)

IN the PR, let's work on the yaml version, until we finalize.

I am open to deciding if the yaml is canonical, or the json.

[gregglind]

secondary note; I somehow borked / dont understand tracking branches and made it on MOzilla instead of Gregglind :)

[gregglind]

installed / startup

• alternative solution
  • during startup, have a localstorage with a 'installed' key
  • check if there
  • if not there, check eligible or die

This seems non idiomatic

Should 'configure' setActiveExperiment? I claim yes. Assumes all these ARE active experiments!

Claim: timers are best handled in the class not the utils directly.