Merge web extension experiments support to master #73

motin · 2018-06-15T21:06:24Z

Fixes #53

Bonus - fixes or makes these irrelevant:
Fixes #69
Fixes #60
Fixes #27
Fixes #10
Fixes #61
Fixes #38
Fixes #19

…f running npm run lint

…adme lean and easy to overview

…ension to pure webextension with bundled web extension experiments

motin · 2018-07-28T10:52:48Z

The CI tests fail with "Uncaught WebDriverError: Failed to decode response from marionette" on Nightly. The tests pass locally using latest nightly. No idea what the underlying issue in CI is.
EDIT: When running circleci locally via Docker, the tests fail (again only on Nightly) a bit further down, with "WebDriverError: Failed to extract Web Element from response"

…ugging failing functional tests in CI

…be downloaded/inspected after the tests has run

motin · 2018-07-30T06:49:57Z

The CI tests fail with "Uncaught WebDriverError: Failed to decode response from marionette" on Nightly. The tests pass locally using latest nightly. No idea what the underlying issue in CI is.

I enabled geckodriver trace logs and made sure that CI stores them under the Artifacts tab for inspection. Comparing the working and non-working logs showed that there seemed to be an issue with expiration timings. A study had expired before the test was able to finish, so I increased the expiration time, and now the tests pass again.

EDIT: They started failing again on a subsequent CI run...

motin · 2018-07-30T07:13:41Z

Nightly is crashing somewhere in relation to onShutdown / onEndStudy when an expired study is encountered. This is what causes the cryptic WebDriver errors.

Geckodriver logs reports A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down, see below:

1532933762828	webdriver::server	DEBUG	-> POST /session/687d4340-59ad-464c-8d4a-017f8f94c2c4/moz/addon/install {"path":"/home/circleci/checkout/dist/button_icon_preference_-_shield_study_example-2.0.0.zip","temporary":true}
1532933762828	webdriver::server	DEBUG	-> POST /session/687d4340-59ad-464c-8d4a-017f8f94c2c4/moz/context {"context":"chrome"}
1532933762831	Marionette	TRACE	0 -> [0,2,"Addon:Install",{"path":"/home/circleci/checkout/dist/button_icon_preference_-_shield_study_example-2.0.0.zip","temporary":true}]
console.debug: shield-study-utils: 
  loading web extension experiment study/api.js
console.debug: shield-study-utils: 
  constructed!
console.debug: shield-study-utils: 
  wanting guard
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  adding schemas [object Object]
console.debug: shield-study-utils: 
  Ajv schemas
Array
    - 0 = http://json-schema.org/draft-07/schema
    - 1 = http://json-schema.org/draft-04/schema
    - 2 = NullableString
    - 3 = NullableInteger
    - 4 = NullableNumber
    - 5 = studyTypesEnum
    - 6 = weightedVariationObject
    - 7 = weightedVariationsArray
    - 8 = anEndingRequest
    - 9 = onEndStudyResponse
    - 10 = studyInfoObject
    - 11 = studySetup
    - 12 = telemetryPayload
    - 13 = searchTelemetryQuery
    - 14 = anEndingAnswer
    - length = 15
1532933763166	Marionette	TRACE	0 <- [1,2,null,{"value":"[email protected]"}]
1532933763176	webdriver::server	DEBUG	<- 200 OK {"value":"[email protected]"}
1532933763181	Marionette	TRACE	0 -> [0,3,"Marionette:SetContext",{"value":"chrome"}]
1532933763182	Marionette	TRACE	0 <- [1,3,null,{"value":null}]
1532933763184	webdriver::server	DEBUG	<- 200 OK {"value": null}
console.debug: shield-study-utils: 
  setting up! -- {"activeExperimentName":"[email protected]","endings":{"user-disable":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=user-disable"],"category":null,"exacturls":null,"fullname":null},"ineligible":{"baseUrls":[],"category":null,"exacturls":null,"fullname":null},"expired":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=expired"],"category":null,"exacturls":null,"fullname":null},"used-often":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=used-often"],"category":"ended-positive","exacturls":null,"fullname":null},"a-non-url-opening-ending":{"baseUrls":[],"category":"ended-neutral","exacturls":null,"fullname":null},"introduction-leave-study":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=introduction-leave-study"],"category":"ended-negative","exacturls":null,"fullname":null}},"expire":{"days":14},"studyType":"shield","telemetry":{"removeTestingFlag":false,"send":true},"testing":{"expired":null,"firstRunTimestamp":0,"variationName":"kittens"},"weightedVariations":[{"name":"kittens","weight":1.5},{"name":"puppers","weight":1.5},{"name":"lizard","weight":1}],"allowEnroll":true}
console.debug: shield-study-utils: 
  about to guard
  studySetup
Object
    - activeExperimentName = [email protected]
    - endings = {"user-disable":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=user-disable"],"category":null,"exacturls":null,"fullname":null},"ineligible":{"baseUrls":[],"category":null,"exacturls":null,"fullname":null},"expired":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=expired"],"category":null,"exacturls":null,"fullname":null},"used-often":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=used-often"],"category":"ended-positive","exacturls":null,"fullname":null},"a-non-url-opening-ending":{"baseUrls":[],"category":"ended-neutral","exacturls":null,"fullname":null},"introduction-leave-study":{"baseUrls":["https://qsurvey.mozilla.com/s3/Shield-Study-Example-Survey/?reason=introduction-leave-study"],"category":"ended-negative","exacturls":null,"fullname":null}}
    - expire = {"days":14}
    - studyType = shield
    - telemetry = {"removeTestingFlag":false,"send":true}
    - testing = {"expired":null,"firstRunTimestamp":0,"variationName":"kittens"}
    - weightedVariations = Array [{"name":"kittens","weight":1.5},{"name":"puppers","weight":1.5},{"name":"lizard","weight":1}]
    - allowEnroll = true
console.debug: shield-study-utils: 
  setting up: variation kittens
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
console.debug: shield-study-utils: 
  encountered already expired study
console.debug: shield-study-utils: 
  called endStudy anEndingAlias
console.debug: shield-study-utils: 
  endStudy expired
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
console.debug: shield-study-utils: 
  unmarking TelemetryEnvironment
  [email protected]
  kittens
console.debug: shield-study-utils: 
  telemetry in:  shield-study {"study_state":"expired","study_state_fullname":"expired"}
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
console.debug: shield-study-utils: 
  telemetry INFO: {"activeExperimentName":"[email protected]","isFirstRun":false,"firstRunTimestamp":0,"variation":{"name":"kittens","weight":1.5},"shieldId":"","delayInMinutes":0}
console.debug: shield-study-utils: 
  telemetry: {"version":3,"study_name":"[email protected]","branch":"kittens","addon_version":"2.0.0","shield_version":"5.0.4","type":"shield-study","data":{"study_state":"expired","study_state_fullname":"expired"},"testing":true}
console.debug: shield-study-utils: 
  telemetry in:  shield-study {"study_state":"exit"}
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
console.debug: shield-study-utils: 
  telemetry INFO: {"activeExperimentName":"[email protected]","isFirstRun":false,"firstRunTimestamp":0,"variation":{"name":"kittens","weight":1.5},"shieldId":"","delayInMinutes":0}
console.debug: shield-study-utils: 
  telemetry: {"version":3,"study_name":"[email protected]","branch":"kittens","addon_version":"2.0.0","shield_version":"5.0.4","type":"shield-study","data":{"study_state":"exit"},"testing":true}
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
console.debug: shield-study-utils: 
  getting info
console.debug: shield-study-utils: 
  getVariation: {"name":"kittens","weight":1.5}
console.debug: shield-study-utils: 
  about to guard
  studyInfoObject
Object
    - activeExperimentName = [email protected]
    - isFirstRun = false
    - firstRunTimestamp = 0
    - variation = {"name":"kittens","weight":1.5}
    - shieldId = 
    - delayInMinutes = 0
1532933763403	Marionette	DEBUG	[4294967300] Frame script loaded
1532933763414	Marionette	DEBUG	[4294967300] Frame script registered
console.debug: shield-study-utils: 
  possible uninstalling
  ADDON_UNINSTALL
console.debug: shield-study-utils: 
  definitely uninstall | disable
  ADDON_UNINSTALL
console.debug: shield-study-utils: 
  endStudy, already ending!
1532933763470	Marionette	DEBUG	[6442450945] Frame script loaded
1532933763476	Marionette	DEBUG	[6442450945] Frame script registered
1532933764419	Marionette	DEBUG	[21] Frame script loaded
1532933764420	Marionette	DEBUG	[21] Frame script registered
A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
1532933764568	Marionette	DEBUG	Closed connection 0
1532933765179	webdriver::server	DEBUG	-> POST /session/687d4340-59ad-464c-8d4a-017f8f94c2c4/moz/context {"context":"chrome"}
1532933765180	webdriver::server	DEBUG	Deleting session
1532933765395	Marionette	DEBUG	Received observer notification xpcom-will-shutdown
1532933765396	Marionette	DEBUG	Resetting recommended pref app.update.disabledForTesting
1532933765396	Marionette	DEBUG	Resetting recommended pref security.turn_off_all_security_so_that_viruses_can_take_over_this_computer
1532933765396	Marionette	DEBUG	Resetting recommended pref apz.content_response_timeout
1532933765396	Marionette	DEBUG	Resetting recommended pref browser.download.panel.shown
1532933765396	Marionette	DEBUG	Resetting recommended pref browser.pagethumbnails.capturing_disabled
1532933765396	Marionette	DEBUG	Resetting recommended pref browser.search.update
1532933765396	Marionette	DEBUG	Resetting recommended pref toolkit.cosmeticAnimations.enabled
1532933765397	Marionette	DEBUG	Resetting recommended pref browser.tabs.disableBackgroundZombification
1532933765397	Marionette	DEBUG	Resetting recommended pref browser.tabs.warnOnCloseOtherTabs
1532933765397	Marionette	DEBUG	Resetting recommended pref browser.tabs.warnOnOpen
1532933765398	Marionette	DEBUG	Resetting recommended pref browser.usedOnWindows10.introURL
1532933765398	Marionette	DEBUG	Resetting recommended pref browser.urlbar.suggest.searches
1532933765398	Marionette	DEBUG	Resetting recommended pref datareporting.policy.dataSubmissionPolicyAccepted
1532933765399	Marionette	DEBUG	Resetting recommended pref dom.disable_beforeunload
1532933765399	Marionette	DEBUG	Resetting recommended pref dom.disable_open_during_load
1532933765399	Marionette	DEBUG	Resetting recommended pref dom.file.createInChild
1532933765399	Marionette	DEBUG	Resetting recommended pref dom.max_chrome_script_run_time
1532933765399	Marionette	DEBUG	Resetting recommended pref dom.max_script_run_time
1532933765399	Marionette	DEBUG	Resetting recommended pref extensions.getAddons.cache.enabled
1532933765399	Marionette	DEBUG	Resetting recommended pref extensions.webservice.discoverURL
1532933765399	Marionette	DEBUG	Resetting recommended pref network.http.prompt-temp-redirect
1532933765400	Marionette	DEBUG	Resetting recommended pref network.http.speculative-parallel-limit
1532933765400	Marionette	DEBUG	Resetting recommended pref security.fileuri.strict_origin_policy
1532933765400	Marionette	DEBUG	Resetting recommended pref security.notification_enable_delay
1532933765400	Marionette	DEBUG	Resetting recommended pref signon.autofillForms
1532933765400	Marionette	DEBUG	Resetting recommended pref signon.rememberSignons
1532933765400	Marionette	INFO	Stopped listening on port 41148
1532933765400	Marionette	DEBUG	Remote service is inactive
1532933765581	geckodriver::marionette	DEBUG	Browser process stopped: exit code: 0
1532933765589	webdriver::server	DEBUG	<- 500 Internal Server Error {"value":{"error":"unknown error","message":"Failed to decode response from marionette","stacktrace":""}}
1532933765597	webdriver::server	DEBUG	-> DELETE /session/687d4340-59ad-464c-8d4a-017f8f94c2c4 
1532933765597	webdriver::server	DEBUG	<- 500 Internal Server Error {"value":{"error":"session not created","message":"Tried to run command without establishing a connection","stacktrace":""}}

motin · 2018-08-30T14:32:27Z

@gregglind @biancadanforth Merging this PR will make it more likely that add-on developers actually use the v5 template. Let's try to review this and merge it so that we can get more eyes on it and less duplicated efforts?

@gregglind @biancadanforth Any update on this? It pains me to see that the v4 version of the template is still the master branch, and thus the "official" one. See mozilla/shield-studies-addon-utils#256

biancadanforth · 2018-08-30T16:07:42Z

Thanks motin! I defer to Gregg, as I do not have the bandwidth right now (or any time soon) to look at this. I know Gregg has been traveling lately too. We are a little short-handed on the Weird Science team at the moment.

chenba · 2018-09-12T20:46:13Z

@motin Hey sorry, I kept this tab open and did manage to look through about half of the patch. But since I don't have much bandwidth and am not familiar with the project, I don't know much value my review would've provided. Glad to see it finally merged. 🥂

motin added 30 commits April 25, 2018 09:36

Two nits

c20531e

Use npm run lint instead of npm run eslint + committing the effects o…

0529c28

…f running npm run lint

Moved majority of docs from readme to DEV and TESTPLAN, making the re…

2b738d1

…adme lean and easy to overview

Changed structure of template from hybrid add-on with embedded webext…

5749e6b

…ension to pure webextension with bundled web extension experiments

Restored most of the button icon example study

89c9d2e

Let readme reflect the current conversion to web extension experiments

21ea464

Npm commands restored (using web-ext)

b6c2f4f

Renamed webextension source directory to src

3671078

Using web-ext-config.js simplifies commands drastically

aefba05

Using dist/ as build output directory

3ba5224

Updated docs and commands to reflect web extensions

e0d2ae5

Restored initiation of test runs

4ac9f97

Improved dev docs

1511701

Tweaks, formatting, linting

d460d0c

Removed unused example study icons

0fe681f

Minor tweaks while debugging the broken browserAction

42a4a03

Updated package-lock.json

e18cbd2

Specified study utils as a runtime dependency since it is

9ba41eb

Example study ui works as advertised

6f63fe7

Linting of manifest.json

cd0f4ed

Fixpack results

73cb290

Fixed incorrect path in npm run lint:addons-linter

ee5feb0

Removed likely-to-be-obsolete dependencies

c47af61

Updated package-lock.json

055d652

Removed unused icons from file tree docs (again)

6050333

Removed some stale files and doc references due to rebase

74460a9

Updated path to add-on zip in npm commands

ff69cd4

Nits in DEV.md

9a8521c

Excluded verbose output for web-ext lint

838e813

Removed addons-linter since we use web-ext lint instead

be41ea6

motin added 4 commits July 28, 2018 13:15

Using nightly as default target for npm start and functional tests

19ff770

Updated circle ci cachebuster

8031abb

Using npm version of shield utils 5.0.3

31efc21

Eslint fix

a2ec1b3

motin added 6 commits July 29, 2018 11:02

Upgraded geckodriver from 0.20.0 to 0.21.0

fddc2d0

Updated selenium-webdriver from 3.5.0 to 3.6.0 (latest available on npm)

3fc9ac3

Producing trace-level geckodriver logs in test/results/logs - for deb…

cdee692

…ugging failing functional tests in CI

Storing geckodriver logs in circle ci as artifacts, so that they can …

20c2999

…be downloaded/inspected after the tests has run

Modified timings in expiration tests

e349697

Avoid EISGIT error on subsequent npm installs

351ab42

motin added 2 commits July 30, 2018 09:54

Added note about functional tests ADDON_ZIP env var

4ece663

Restored and ran npm run format

0267866

This was referenced Jul 30, 2018

find ways to remind people to install nightly / run with nightly #78

Closed

Nightly is crashing (possible only on Linux) somewhere in relation to onShutdown / onEndStudy when an expired study is encountered mozilla/shield-studies-addon-utils#254

Closed

motin added 2 commits August 31, 2018 18:52

Geckodriver npm package is now fixed upstream

b307f14

Updated ci cachebusters to ensure that latest nightly is used

75be261

gregglind merged commit cf220bd into master Sep 12, 2018

motin deleted the develop branch October 26, 2018 07:00

This was referenced Nov 16, 2018

Bootstrap should also shutdown the webextension component #69

Closed

linked-addon.xpi fails to load in Firefox Developer Edition (outdated README?) #43

Closed

Add prettier? #38

Closed

motin mentioned this pull request Jan 30, 2019

Convert bin/xpi.sh into cross-env Node type script? #60

Closed

This was referenced Mar 2, 2019

What Template SHOULD be able to do #10

Closed

Consider adding build artifact support for CI or provide easy process for QA to build XPIs locally #58

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge web extension experiments support to master #73

Merge web extension experiments support to master #73

motin commented Jun 15, 2018 •

edited

Loading

motin commented Jul 28, 2018 •

edited

Loading

motin commented Jul 30, 2018 •

edited

Loading

motin commented Jul 30, 2018 •

edited

Loading

motin commented Aug 30, 2018

biancadanforth commented Aug 30, 2018

chenba commented Sep 12, 2018

Merge web extension experiments support to master #73

Merge web extension experiments support to master #73

Conversation

motin commented Jun 15, 2018 • edited Loading

motin commented Jul 28, 2018 • edited Loading

motin commented Jul 30, 2018 • edited Loading

motin commented Jul 30, 2018 • edited Loading

motin commented Aug 30, 2018

biancadanforth commented Aug 30, 2018

chenba commented Sep 12, 2018

motin commented Jun 15, 2018 •

edited

Loading

motin commented Jul 28, 2018 •

edited

Loading

motin commented Jul 30, 2018 •

edited

Loading

motin commented Jul 30, 2018 •

edited

Loading