From 53cdaa757cb7a9351f4776af7eac6e6ed06d6ef3 Mon Sep 17 00:00:00 2001 From: Isaiah Robinson <95643215+internetisaiah@users.noreply.github.com> Date: Tue, 11 Jun 2024 21:11:56 -0700 Subject: [PATCH 1/4] BD-3161: Feature flag experiments --- .../platform_wide/feature_flags/canvas.md | 2 +- .../feature_flags/experiments.md | 49 +++++++++++++------ 2 files changed, 34 insertions(+), 17 deletions(-) diff --git a/_docs/_developer_guide/platform_wide/feature_flags/canvas.md b/_docs/_developer_guide/platform_wide/feature_flags/canvas.md index f5d6821f09a..4d087ee9d45 100644 --- a/_docs/_developer_guide/platform_wide/feature_flags/canvas.md +++ b/_docs/_developer_guide/platform_wide/feature_flags/canvas.md @@ -1,6 +1,6 @@ --- nav_title: Feature Flags in Canvas -page_order: 30 +page_order: 40 noindex: true tool: Feature Flags platform: diff --git a/_docs/_developer_guide/platform_wide/feature_flags/experiments.md b/_docs/_developer_guide/platform_wide/feature_flags/experiments.md index fe55c09789e..75220d0919a 100644 --- a/_docs/_developer_guide/platform_wide/feature_flags/experiments.md +++ b/_docs/_developer_guide/platform_wide/feature_flags/experiments.md @@ -1,7 +1,7 @@ --- nav_title: Feature Flag Experiments article_title: Feature Flag Experiments -page_order: 40 +page_order: 30 description: "Feature flag experiments let you A/B test changes to your applications to optimize conversion rates." tool: Feature Flags platform: @@ -11,7 +11,7 @@ platform: --- -# Creating a feature flag experiment +# Feature flag experiments > Feature flag experiments let you A/B test changes to your applications to optimize conversion rates. Marketers can use feature flags to determine whether a new feature positively or negatively impacts conversion rates, or which set of feature flag properties is most optimal. @@ -32,27 +32,25 @@ if (featureFlag?.enabled) { ``` -## Step 1: Create an experiment +## Creating a feature flag experiment + +### Step 1: Create an experiment 1. Go to **Messaging** > **Campaigns** and click **+ Create Campaign**. 2. Select **Feature Flag Experiment**. 3. Name your campaign something clear and meaningful. -## Step 2: Add experiment variants - -Next, create variations. For each variant, choose the feature flag you want to turn on or off and review the assigned properties. +### Step 2: Add experiment variants -To test the impact of your feature, use variants to split traffic into two or more groups. Name one group "My control group" and turn its feature flags off. +Next, create variations. For each variant, choose the feature flag you want to turn on or off and review the assigned properties. To test the impact of your feature, use variants to split traffic into two or more groups. Name one group "My control group" and turn its feature flags off. -### Overwriting properties +#### Overwriting properties -Though you specified default properties when you originally set up your feature flag, you can choose to overwrite those values for users who receive a specific campaign variant. +Though you specified default properties when you originally set up your feature flag, you can choose to overwrite those values for users who receive a specific campaign variant. To edit, add, or remove additional default properties, edit the feature flag itself from **Messaging** > **Feature Flags**. ![][image1]{: style="max-width:80%"} -To edit, add, or remove additional default properties, edit the feature flag itself from **Messaging** > **Feature Flags**. - -## Step 3: Choose users to target +### Step 3: Choose users to target Next, you need to [target users][4] by choosing segments or filters to narrow down your audience. Segment membership is calculated when feature flags are refreshed for a given user. @@ -60,18 +58,37 @@ Next, you need to [target users][4] by choosing segments or filters to narrow do Your target audience will be eligible for the feature flag as soon as you save a rollout greater than 0%. Changes are made available once your app refreshes feature flags, or when a new session is started. {% endalert %} -## Step 4: Distribute variants +### Step 4: Distribute variants Choose the percentage distribution for your experiment. As a best practice, you should not change the distribution once your experiment has been launched. -## Step 5: Assign conversions +### Step 5: Assign conversions Braze lets you to track how often users perform specific actions, [conversion events][5], after receiving a campaign. Specify up to a 30-day window during which a conversion will be counted if the user takes the specified action. -## Step 6: Review and launch +### Step 6: Review and launch + +After you’ve finished building the last of your experiment, review its details, then click **Launch Experiment**. When your experiment is finished, you can [analyze the results](#analyzing-your-experiment). + +## Reviewing the results + +After your feature flag experiment is finished, you can review your results for... + +Go to **Messaging** > **Campaigns** and select the campaign containing your feature flag experiment. + +### Campaign analytics + +Campaign analytics offers a high-level overview of your experiment's performance. + +Review this panel to see overall metrics such as the total number of impressions logged from this experiment, the unique impressions logged from the experiment, the primary conversion rate, the total revenue generated by this message, and the estimated audience. You can also review delivery, audience, and conversion settings from this page. + +![ALT_TEXT]() + +### Feature flag experiment performance -After you’ve finished building the last of your experiment, review its details, then click **Launch Experiment**. +The Feature Flags Experiments Performance panel outlines how well your message has performed across various dimensions. The metrics in this panel vary depending on your chosen messaging channel, and whether or not you are running a multivariate test. You can click on the Preview icon to see the feature flag values that pertain to each variant. +![ALT_TEXT]() [1]: {{site.baseurl}}/user_guide/administrative/manage_your_braze_users/teams/ [2]: {{site.baseurl}}/user_guide/administrative/app_settings/manage_app_group/tags/ From d43a0f9c94a5958beb8b5d1a2e892c532c9ad71e Mon Sep 17 00:00:00 2001 From: internetisaiah <95643215+internetisaiah@users.noreply.github.com> Date: Wed, 21 Aug 2024 08:46:25 -0700 Subject: [PATCH 2/4] Second round --- .../platform_wide/feature_flags/experiments.md | 14 ++++++++++---- .../data_and_analytics/report_metrics.md | 4 ++++ .../reporting/campaign_analytics.md | 3 +++ 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/_docs/_developer_guide/platform_wide/feature_flags/experiments.md b/_docs/_developer_guide/platform_wide/feature_flags/experiments.md index 75220d0919a..cfa4ad34f0e 100644 --- a/_docs/_developer_guide/platform_wide/feature_flags/experiments.md +++ b/_docs/_developer_guide/platform_wide/feature_flags/experiments.md @@ -74,19 +74,25 @@ After you’ve finished building the last of your experiment, review its details After your feature flag experiment is finished, you can review your results for... -Go to **Messaging** > **Campaigns** and select the campaign containing your feature flag experiment. +Go to **Messaging** > **Campaigns** and select the campaign with your feature flag experiment. ### Campaign analytics -Campaign analytics offers a high-level overview of your experiment's performance. +**Campaign Analytics** offers a high-level overview of your experiment's performance, such as: -Review this panel to see overall metrics such as the total number of impressions logged from this experiment, the unique impressions logged from the experiment, the primary conversion rate, the total revenue generated by this message, and the estimated audience. You can also review delivery, audience, and conversion settings from this page. +- The total number of impressions +- The number of unique impressions +- The primary conversion rate +- The total revenue generated by the message +- The estimated audience + +You can also view the experiment's settings for delivery, audience, and conversion. ![ALT_TEXT]() ### Feature flag experiment performance -The Feature Flags Experiments Performance panel outlines how well your message has performed across various dimensions. The metrics in this panel vary depending on your chosen messaging channel, and whether or not you are running a multivariate test. You can click on the Preview icon to see the feature flag values that pertain to each variant. +**Feature Flags Experiments Performance** shows how well your message performed across various dimensions. The specific metrics you see will vary depending on your chosen messaging channel, and whether you're running a multivariate test. To see the feature flag values associated with each variant, select **Preview**. ![ALT_TEXT]() diff --git a/_docs/_user_guide/data_and_analytics/report_metrics.md b/_docs/_user_guide/data_and_analytics/report_metrics.md index 458febee797..7c351d17de7 100644 --- a/_docs/_user_guide/data_and_analytics/report_metrics.md +++ b/_docs/_user_guide/data_and_analytics/report_metrics.md @@ -115,6 +115,10 @@ Total number of clicks on Button 2 of the message. {% api %} +### Campaign analytics + +The performance of the message across various dimensions. The metrics shown depend on the selected messaging channel, and whether the [Feature Flag experiment]({{site.baseurl}}/developer_guide/platform_wide/feature_flags/experiments/#campaign-analytics) is a multivariate test. + ### Choices Submitted {% apitags %} diff --git a/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md b/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md index c72c4967286..3a07b0657df 100644 --- a/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md +++ b/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md @@ -18,6 +18,9 @@ guide_featured_list: - name: Email link: /docs/user_guide/message_building_by_channel/email/reporting_and_analytics/email_reporting/ image: /assets/img/braze_icons/mail-01.svg + - name: Feature Flags + link: docs/developer_guide/platform_wide/feature_flags/experiments/ + image: /assets/img/braze_icons/whatsapp.svg - name: In-App Messages link: /docs/user_guide/message_building_by_channel/in-app_messages/reporting/ image: /assets/img/braze_icons/message-text-circle-01.svg From e46caa202a7219d120845e531167a9d69489f550 Mon Sep 17 00:00:00 2001 From: internetisaiah <95643215+internetisaiah@users.noreply.github.com> Date: Wed, 21 Aug 2024 08:53:03 -0700 Subject: [PATCH 3/4] replacing placeholder icon --- .../data_and_analytics/reporting/campaign_analytics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md b/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md index 3a07b0657df..79661173f1f 100644 --- a/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md +++ b/_docs/_user_guide/data_and_analytics/reporting/campaign_analytics.md @@ -20,7 +20,7 @@ guide_featured_list: image: /assets/img/braze_icons/mail-01.svg - name: Feature Flags link: docs/developer_guide/platform_wide/feature_flags/experiments/ - image: /assets/img/braze_icons/whatsapp.svg + image: /assets/img/braze_icons/flag-06.svg - name: In-App Messages link: /docs/user_guide/message_building_by_channel/in-app_messages/reporting/ image: /assets/img/braze_icons/message-text-circle-01.svg From cd6e9f6dc021cf4e22e3d5b6251cddb5dd9a3b81 Mon Sep 17 00:00:00 2001 From: internetisaiah <95643215+internetisaiah@users.noreply.github.com> Date: Thu, 22 Aug 2024 17:55:22 -0700 Subject: [PATCH 4/4] finishing report_metrics.md --- .../data_and_analytics/report_metrics.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/_docs/_user_guide/data_and_analytics/report_metrics.md b/_docs/_user_guide/data_and_analytics/report_metrics.md index 7c351d17de7..385cc8e63cb 100644 --- a/_docs/_user_guide/data_and_analytics/report_metrics.md +++ b/_docs/_user_guide/data_and_analytics/report_metrics.md @@ -117,8 +117,16 @@ Total number of clicks on Button 2 of the message. ### Campaign analytics +{% apitags %} +Feature Flags +{% endapitags %} + The performance of the message across various dimensions. The metrics shown depend on the selected messaging channel, and whether the [Feature Flag experiment]({{site.baseurl}}/developer_guide/platform_wide/feature_flags/experiments/#campaign-analytics) is a multivariate test. +{% endapi %} + +{% api %} + ### Choices Submitted {% apitags %} @@ -299,6 +307,18 @@ The WhatsApp message could not send because the Internet Service Provider return {% api %} +### Feature flag experiment performance + +{% apitags %} +Feature Flags +{% endapitags %} + +Performance metrics for the message in a Feature Flag experiment. The specific metrics shown will vary depending on the messaging channel, and whether or not the experiment was a multivariate test. + +{% endapi %} + +{% api %} + ### Influenced Opens {% apitags %}