Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(experiments): Apply new Trends continuous stats methods #26835

Merged
merged 40 commits into from
Dec 20, 2024
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
53000ec
Rename in prep for continuous implementation
danielbachhuber Dec 11, 2024
7a5ffc5
First pass at continuous stats methods
danielbachhuber Dec 11, 2024
2394003
Drop assertion context because it's too noisy
danielbachhuber Dec 11, 2024
f5ec622
Add a test case for near zero means
danielbachhuber Dec 11, 2024
2501c1c
Handle zero mean scenario
danielbachhuber Dec 11, 2024
43735d2
Update query snapshots
github-actions[bot] Dec 11, 2024
af09286
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 12, 2024
8a670dd
Update query snapshots
github-actions[bot] Dec 12, 2024
280cd48
Update query snapshots
github-actions[bot] Dec 12, 2024
883750f
Adjust test values
danielbachhuber Dec 12, 2024
4be08a0
Use `absolute_exposure`, not relative exposure
danielbachhuber Dec 12, 2024
f619d41
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 13, 2024
c6c58b4
Introduce a new `assertRange()` method
danielbachhuber Dec 13, 2024
2a27a31
Increase log variance to increase uncertainty for small sample sizes
danielbachhuber Dec 13, 2024
d3b8c7e
Use correct exposure values
danielbachhuber Dec 13, 2024
61bfc39
Use correct absolute exposure for these tests
danielbachhuber Dec 13, 2024
4c14b48
Use correct exposure values
danielbachhuber Dec 13, 2024
6c0ab9c
Adjust range values for `test_many_variants_not_significant`
danielbachhuber Dec 13, 2024
4aa1fc6
Update values for `test_large_sample_two_variants_significant`
danielbachhuber Dec 13, 2024
6962e97
Update values for `test_different_relative_and_absolute_exposure`
danielbachhuber Dec 13, 2024
61f118c
Replace some use of assertRange
danielbachhuber Dec 13, 2024
dc8fceb
Replace use of `assertRange`
danielbachhuber Dec 13, 2024
7dfc17b
This test is no longer necessary
danielbachhuber Dec 13, 2024
9f43ba7
Use correct `absolute_exposure` values
danielbachhuber Dec 13, 2024
772208f
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 13, 2024
3a6d407
Drop duplicative test
danielbachhuber Dec 13, 2024
3d2e24c
Add a test case with real world data
danielbachhuber Dec 13, 2024
2ea703d
Use a standard `LOG_VARIANCE` between methods
danielbachhuber Dec 13, 2024
858d7b4
Provide a helpful message when the tests fail
danielbachhuber Dec 16, 2024
dc9b7bb
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 16, 2024
00a4d64
Revert "Provide a helpful message when the tests fail"
danielbachhuber Dec 16, 2024
838067a
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 17, 2024
4e81ea4
Use consistent constant names
danielbachhuber Dec 17, 2024
f34bea5
Use the log variance provided by Anders
danielbachhuber Dec 17, 2024
bcf3259
Merge branch 'master' into experiments/trends-continuous
danielbachhuber Dec 19, 2024
ecaed9a
Implement expected loss for trend count
danielbachhuber Dec 19, 2024
17ea023
Apply expected loss to trends continuous
danielbachhuber Dec 19, 2024
78af92d
More informative description
danielbachhuber Dec 19, 2024
54f989e
Incorporate continuous methods into TrendsQueryRunner
danielbachhuber Dec 19, 2024
340ed68
Add a test for 54f989ebd8b8fd9a7ae96a62292d3fcde05006a6
danielbachhuber Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
calculate_credible_intervals,
calculate_probabilities,
)
from posthog.hogql_queries.experiments.trends_statistics_v2 import (
are_results_significant_v2,
calculate_credible_intervals_v2,
calculate_probabilities_v2,
from posthog.hogql_queries.experiments.trends_statistics_v2_count import (
are_results_significant_v2_count,
calculate_credible_intervals_v2_count,
calculate_probabilities_v2_count,
)
from posthog.hogql_queries.insights.trends.trends_query_runner import TrendsQueryRunner
from posthog.hogql_queries.query_runner import QueryRunner
Expand Down Expand Up @@ -316,9 +316,9 @@ def run(query_runner: TrendsQueryRunner, result_key: str, is_parallel: bool):
# Statistical analysis
control_variant, test_variants = self._get_variants_with_base_stats(count_result, exposure_result)
if self.stats_version == 2:
probabilities = calculate_probabilities_v2(control_variant, test_variants)
significance_code, p_value = are_results_significant_v2(control_variant, test_variants, probabilities)
credible_intervals = calculate_credible_intervals_v2([control_variant, *test_variants])
probabilities = calculate_probabilities_v2_count(control_variant, test_variants)
significance_code, p_value = are_results_significant_v2_count(control_variant, test_variants, probabilities)
credible_intervals = calculate_credible_intervals_v2_count([control_variant, *test_variants])
else:
probabilities = calculate_probabilities(control_variant, test_variants)
significance_code, p_value = are_results_significant(control_variant, test_variants, probabilities)
Expand Down
Loading
Loading