experiment design

[gkovacs]

hypothesis

we hypothesize that novelty has benefits to effectiveness and/or attrition. we want to see whether this is true (this experiment) and measure its degree.

if this turns out to be true, in follow up studies we hope to be able to utilize this information for intelligently picking interventions / crowdsourcing them taking into account novelty

research questions

effectiveness: comparing the high novelty condition weeks to the low novelty condition weeks, is more time saved relative to baseline (shorter sessions)

attrition: on weeks where the high novelty condition is shown, is there a difference in uninstall rates (hopefully lower) relative to low novelty weeks

experiment itself

within subjects design

2 weeks in duration

each week users are either in a high novelty or low novelty condition

high novelty = show randomly chosen message from a large pool

low novelty = show same message always. (just chosen at install time randomly)

in all weeks, half the time we don't show any message. this is the control so we know how much the baseline time would have been without any interventions (hence we know how much time is being saved)

timeline

hopefully should launch before the end of this quarter