Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(experiments): detailed results for primary metrics #27046

Merged
merged 16 commits into from
Dec 20, 2024

Conversation

jurajmajerik
Copy link
Contributor

@jurajmajerik jurajmajerik commented Dec 19, 2024

Problem

Part of #27014

Changes

Feature flag: experiments-multiple-metrics

  1. Add a button to open a modal which displays detailed results for primary metrics.
  2. Add a significance/non-significance highlight
image image image

How did you test this code?

👀

@jurajmajerik jurajmajerik requested a review from a team December 19, 2024 11:00
Copy link
Contributor

github-actions bot commented Dec 19, 2024

Size Change: 0 B

Total Size: 1.11 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 1.11 MB

compressed-size-action

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

  • chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

3 snapshot changes in total. 0 added, 3 modified, 0 deleted:

  • chromium: 0 added, 3 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@danielbachhuber
Copy link
Contributor

😱

CleanShot 2024-12-19 at 07 18 09@2x

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

  • chromium: 0 added, 2 modified, 0 deleted (wasn't pushed!)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

  • chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@@ -96,6 +96,7 @@ describe('Experiments', () => {
cy.get('[data-attr="experiment-creation-date"]').contains('a few seconds ago').should('be.visible')
cy.get('[data-attr="experiment-start-date"]').should('not.exist')

cy.wait(1000)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some re-rendering glitch causes the click to fail. Waiting for a bit solves this; no issues when testing manually.

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

Copy link
Contributor

@andehen andehen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Starting to look like something! Great to be able to quickly click into a metric and see the timeline of it 👍 Maybe add a button to explore the metric further in the product analytics tool?

Not for later: Something we can easily support here as well is pre-exposure / post-exposure analysis (by allowing the user to change the date filter). We got this feature request a lot in my previous gig. And it makes sense, as it allows users to see if there was any bias in the groups prior to the experiment.

Unrelated to this change maybe: It looks a bit weird in fullscreen, maybe limit the width of the metrics component?
Screenshot 2024-12-20 at 14 00 15

@jurajmajerik jurajmajerik merged commit d8f79bf into master Dec 20, 2024
104 of 105 checks passed
@jurajmajerik jurajmajerik deleted the primary-metrics-detail branch December 20, 2024 13:13
@jurajmajerik
Copy link
Contributor Author

jurajmajerik commented Dec 20, 2024

@andehen

Maybe add a button to explore the metric further

Good catch, will add in a follow up 👍

maybe limit the width of the metrics component

Will make the responsiveness better in a follow up too

pre-exposure / post-exposure analysis

I'll create an issue for this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants