feat: exp for nontechnical users in prod. analytics #26993

surbhi-posthog · 2024-12-17T22:03:01Z

Problem

We noticed that many users entering our funnel are categorized as "Low ICP." To help these users convert through the onboarding flow in product analytics that has the most number of users, this PR introduces a few product recommendations:

For non-technical users in the experiment ONLY ~
Show HTML first – This is likely to be more useful for non-technical users.
Show Invite banner on top of the page, so its obvious they can use this feature if needed.

I also considered creating two different experiments for this but I'd rather keep it under one since these changes are relatively small

For more details on the hypothesis please refer to this notebook: https://us.posthog.com/project/2/notebooks/p9AvDIAK

Changes

Reordered the SDK list.
Added a feature flag (Note: I wasn't able to test this properly, as the feature flag didn't appear in the values).
Moved invite banner to the top of the page for non-technical users only

Before	After

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

Cloud: yes

How did you test this code?

Test Plan

Screen.Recording.2024-12-20.at.9.43.06.AM.mov

User Signup
- Sign up as a new user.
- Verify that the process completes successfully and that the role_at_organization data is captured in the model for future use.
Invite New User Signup
- Invite a new user to sign up via email.
- Verify that the invited user successfully signs up and that their role_at_organization data is recorded.
Product Analytics Installation Test
- Walk through the product analytics installation process with both technical and non-technical users.
- Verify that both user types can complete the installation without issues.
Impact on Other Products
- Verify that other products are unaffected by the changes made during this process. Ensure no disruptions to existing features or functionality.
[NOTE] I added role_at_organization changes for when the user signs up as a demo. I haven't tested this case. I need to verify what the UX looks for this case. TODO @surbhi-posthog

github-actions · 2024-12-17T22:14:15Z

Size Change: +74 B (+0.01%)

Total Size: 1.11 MB

ℹ️ View Unchanged

Filename	Size	Change
`frontend/dist/toolbar.js`	1.11 MB	+74 B (+0.01%)

_{compressed-size-action}

rafaeelaudibert

I'm just reviewing a couple of random PRs to get myself familiarized with the rest of the codebase, so take my review with a grain of salt, I don't know anything about current goals in the team :)

Wont approve nor request changes because Im just passing by

rafaeelaudibert · 2024-12-18T17:37:41Z

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

@@ -39,10 +41,25 @@ Products that will often be installed in multiple places, eg. web and mobile
 */
 export const multiInstallProducts = [ProductKey.PRODUCT_ANALYTICS, ProductKey.FEATURE_FLAGS]

+const getOrderedSDKs = (sdks: SDK[]): SDK[] => {


This is only "ordered" in terms of product analytics, so I'd be more semantic in this function name

Suggested change

const getOrderedSDKs = (sdks: SDK[]): SDK[] => {

const getOrderedProductAnalyticsSdks = (sdks: SDK[]): SDK[] => {

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

zlwaterfield

Great work here :). Added a few requests.

zlwaterfield · 2024-12-18T19:02:30Z

frontend/src/lib/constants.tsx

@@ -156,6 +156,7 @@ export const FEATURE_FLAGS = {
    FF_DASHBOARD_TEMPLATES: 'ff-dashboard-templates', // owner: @EDsCODE
    ARTIFICIAL_HOG: 'artificial-hog', // owner: @Twixes
    CS_DASHBOARDS: 'cs-dashboards', // owner: @pauldambra
+    PRODUCT_ANALYTICS_ONBOARDING: 'product-analytics-onboarding', // owner: @surbhi


Can we make this experiment key more specific to these changes incase we want to add more in onboarding.

zlwaterfield · 2024-12-18T19:03:35Z

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

+        '/onboarding/:productKey': (_product_key, { sdk }) => {
+            if (
+                values.productKey === ProductKey.PRODUCT_ANALYTICS &&
+                values.featureFlags[FEATURE_FLAGS.PRODUCT_ANALYTICS_ONBOARDING]


For an experiment we can do values.featureFlags[FEATURE_FLAGS.PRODUCT_ANALYTICS_ONBOARDING] === "test" where "test" is the experiment variant.

zlwaterfield · 2024-12-18T19:04:02Z

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

+            // Set default source filter for Product Analytics
+            if (
+                productKey === ProductKey.PRODUCT_ANALYTICS &&
+                values.featureFlags[FEATURE_FLAGS.PRODUCT_ANALYTICS_ONBOARDING]


Sam as comment below.

zlwaterfield · 2024-12-18T19:04:23Z

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

                .filter((sdk) => {
                    if (!values.sourceFilter || !sdk) {
                        return true
                    }
                    return sdk.tags.includes(values.sourceFilter)
                })
                .filter((sdk) => Object.keys(values.availableSDKInstructionsMap).includes(sdk.key))
+
+            if (
+                values.featureFlags[FEATURE_FLAGS.PRODUCT_ANALYTICS_ONBOARDING] &&


Same as comment below.

surbhi-posthog · 2024-12-18T23:02:14Z

I've made suggested changes. However, I'm still not able to see the feature flag values show up locally. This is currently undefined:values.featureFlags[FEATURE_FLAGS.PRODUCT_ANALYTICS_MODIFIED_SDK_LIST]. Perhaps, it only shows up once I enable the feature flag, but I didn't want to do that yet since I haven't completed the experiment setup

zlwaterfield · 2024-12-18T23:43:53Z

frontend/src/scenes/onboarding/sdks/sdksLogic.tsx

@@ -198,7 +222,6 @@ export const sdksLogic = kea<sdksLogicType>([
        },
        [onboardingLogic.actionTypes.setProductKey]: () => {
            // TODO: This doesn't seem to run when the setProductKey action is called in onboardingLogic...
-            actions.resetSDKs()


Any reason this was removed?

raquelmsmith · 2024-12-19T00:51:57Z

frontend/src/lib/constants.tsx

Commenting on this file so we can thread the discussion...

I'm generally hesitant about this change because it effectively hides the rest of the SDKs we offer from the page.

The null hypothesis is that people won't notice the "recommended" filter, won't click the "X" to clear it, and won't see all the other SDKs we offer.

Why don't we just put the recommended ones at the top in a separate section, with some way to show that they are recommended?

raquelmsmith · 2024-12-19T00:56:22Z

frontend/src/lib/constants.tsx

Another thing - I think with this we are maybe prioritizing the wrong set of users?

High ICP users - engineers - will want to see all the SDK options we have so they can find the one they need. Maybe the experience there needs to change for those users if their conversion through this step isn't great, but I don't see why we'd change the onboarding for our most popular product to cater to low ICP users.

If anything, we should only apply this change to people who we know are low-ICP (using role data is easy easy easy and required on signup for this exact reason). We can even skip this SDKs screen entirely for these people and do a custom invite thing where it says "you're all set up, now invite your technical team to implement" and in the email we send them we give them context about what needs to happen.

raquelmsmith

I feel like my comments should be addressed before this is merged

surbhi-posthog · 2024-12-20T15:30:34Z

After consolidating the feedback above, I've made a few significant changes to the experiment:

Role-Based Targeting: The new features will now only be shown to users with a low ICP (Ideal Customer Profile) score, which corresponds to our non-technical users.
Invite Help Card Placement: Instead of using a recommended list, I moved the invite help card to the top of the page for these users.

I’m hoping to get some signal from this test, and if anything learn how to collect signal correctly for this flow.

This got me thinking about other ideas, some of which I know we’re already discussing:

When the user selects products they want to use, we still show a list of all the product available in the side bar, I was wondering if adding an optimization there could help with user journey
Allowing users to skip the SDK step entirely, if needed.
Showing onboarding steps for all products selected at the beginning.
Allowing users to revisit the onboarding flow from different situations.

zlwaterfield · 2024-12-20T15:53:35Z

frontend/src/scenes/userLogic.ts

@@ -250,6 +250,14 @@ export const userLogic = kea<userLogicType>([
                return user?.theme_mode || 'light'
            },
        ],
+
+        isUserTechnical: [


I think we should flip this so it defaults to technical for all users.

It's important to note we won't always have this data because it's only via email/password not social SSO (Google, Github, etc.)

zlwaterfield · 2024-12-20T15:55:24Z

posthog/models/user.py

+        ("other", "Other"),
+    )
+
+    role_at_organization = models.CharField(max_length=64, choices=ROLE_CHOICES, null=True, blank=True)


Can you break this PR out into multiple PRs? Database migrations are always important to be methodical about:

Add new column w/ migration

Add changes to signup API to support the new attribute

Update the frontend to send the new attribute (and make any other changes)

Also we can move this row up to be with the other user details.

zlwaterfield · 2024-12-20T15:56:41Z

posthog/models/user.py

@@ -172,6 +172,19 @@ class User(AbstractUser, UUIDClassicModel):
    # Remove unused attributes from `AbstractUser`
    username = None

+    ROLE_CHOICES = (


Move this up to the top with the others.

surbhi-posthog requested review from zlwaterfield, raquelmsmith and patricio-posthog December 17, 2024 22:03

surbhi-posthog marked this pull request as ready for review December 17, 2024 22:08

surbhi-posthog force-pushed the onboarding branch 2 times, most recently from 7e87466 to 274ada4 Compare December 18, 2024 01:02

onboarding product logic:

d639bc6

surbhi-posthog force-pushed the onboarding branch from 274ada4 to d639bc6 Compare December 18, 2024 01:03

rafaeelaudibert reviewed Dec 18, 2024

View reviewed changes

zlwaterfield requested changes Dec 18, 2024

View reviewed changes

updated experiment name and addressed other comments

5ab27a5

surbhi-posthog requested a review from zlwaterfield December 18, 2024 23:41

zlwaterfield reviewed Dec 18, 2024

View reviewed changes

placing back resetSDK on setProductKey action

c2dc08d

surbhi-posthog requested a review from zlwaterfield December 19, 2024 00:12

raquelmsmith reviewed Dec 19, 2024

View reviewed changes

raquelmsmith requested changes Dec 19, 2024

View reviewed changes

surbhi-posthog added 3 commits December 19, 2024 16:45

role_at_org model changes and ui updates

4b8948b

user serializer udpates

dd59b7d

ux updates

7e5c788

surbhi-posthog marked this pull request as draft December 20, 2024 00:07

moving logic to selectors

a80c01c

surbhi-posthog marked this pull request as ready for review December 20, 2024 15:36

surbhi-posthog requested a review from raquelmsmith December 20, 2024 15:36

surbhi-posthog changed the title ~~feat: show modified recommended list as default for analytics~~ feat: exp for nontechnical users in prod. analytics Dec 20, 2024

surbhi-posthog requested a review from rafaeelaudibert December 20, 2024 15:48

surbhi-posthog requested a review from annaszell December 20, 2024 15:53

zlwaterfield reviewed Dec 20, 2024

View reviewed changes

surbhi-posthog mentioned this pull request Dec 20, 2024

feat: [1/n][role experiement] user model changes #27113

Closed

rafaeelaudibert removed their request for review December 25, 2024 18:51

surbhi-posthog mentioned this pull request Dec 30, 2024

feat: [1/n][role experiement] user model changes #27190

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: exp for nontechnical users in prod. analytics #26993

feat: exp for nontechnical users in prod. analytics #26993

surbhi-posthog commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

rafaeelaudibert left a comment

rafaeelaudibert Dec 18, 2024

zlwaterfield left a comment

zlwaterfield Dec 18, 2024

zlwaterfield Dec 18, 2024

zlwaterfield Dec 18, 2024

zlwaterfield Dec 18, 2024

surbhi-posthog commented Dec 18, 2024 •

edited

Loading

zlwaterfield Dec 18, 2024

raquelmsmith Dec 19, 2024

raquelmsmith Dec 19, 2024

raquelmsmith left a comment

surbhi-posthog commented Dec 20, 2024 •

edited

Loading

zlwaterfield Dec 20, 2024

zlwaterfield Dec 20, 2024

zlwaterfield Dec 20, 2024

zlwaterfield Dec 20, 2024

	const getOrderedSDKs = (sdks: SDK[]): SDK[] => {
	const getOrderedProductAnalyticsSdks = (sdks: SDK[]): SDK[] => {

feat: exp for nontechnical users in prod. analytics #26993

Are you sure you want to change the base?

feat: exp for nontechnical users in prod. analytics #26993

Conversation

surbhi-posthog commented Dec 17, 2024 • edited Loading

Problem

Changes

Does this work well for both Cloud and self-hosted?

How did you test this code?

Test Plan

github-actions bot commented Dec 17, 2024 • edited Loading

rafaeelaudibert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zlwaterfield left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

surbhi-posthog commented Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raquelmsmith left a comment

Choose a reason for hiding this comment

surbhi-posthog commented Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

surbhi-posthog commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

surbhi-posthog commented Dec 18, 2024 •

edited

Loading

surbhi-posthog commented Dec 20, 2024 •

edited

Loading