You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. Run the application, [see step 4 of Use variant feature flags](./howto-variant-feature-flags-javascript.md#run-the-application).
95
-
96
-
1. Create 10 different users and log into the application. As you log in with each user, you get a different message variant for some of them. ~50% of the time you get no message. 25% of the time you get the message "Hello!" and 25% of the time you get "I hope this makes your day!"
97
-
98
-
1. With some of the users click the **Like** button to trigger the telemetry event.
99
-
100
-
> [!div class="mx-imgBorder"]
101
-
>
102
-
103
-
1. Open your Application Insights resource in the Azure portal and select**Logs** under **Monitoring**. In the query window, run the following query to see the telemetry events:
104
-
105
-
```kusto
106
-
// Step 1: Get distinct users and their Variant from FeatureEvaluation
>
158
-
159
-
You see one "FeatureEvaluation" event for each time the quote page was loaded and one "Liked" event for each time the like button was clicked. The "FeatureEvaluation" event has a custom property called `FeatureName` with the name of the feature flag that was evaluated. Both events have a custom property called `TargetingId` with the name of the user that liked the quote.
94
+
## Collect telemetry
160
95
161
-
For more information about the "FeatureEvaluation" event, go to the [Feature flag telemetry reference](./feature-flag-telemetry-reference.md)
96
+
Deploy your application to begin collecting telemetry from your users. To test its functionality, you can simulate user activity by creating many test users. Each user will experience a different variant of greeting messages, and they can interact with the application by clicking the heart button to like a quote. As your user base grows, you can monitor the increasing volume of telemetry data collected in Azure App Configuration. Additionally, you can drill down into the data to analyze how each variant of the feature flag influences user behavior.
97
+
- [Review telemetry results in App Configuration](./howto-telemetry.md#review-telemetry-results-in-azure-app-configuration).
162
98
163
99
## Additional resources
164
100
165
101
- [Quote of the Day sample](https://github.com/Azure-Samples/quote-of-the-day-javascript)
166
-
167
-
## Next steps
168
-
169
-
For the full feature rundown of the JavaScript feature management library, refer to the following document.
- For the full feature rundown of the JavaScript feature management library you can refer to the [JavaScript Feature Management reference documentation](./feature-management-javascript-reference.md)
1. Run the application, [see step 2 of Use variant feature flags](./howto-variant-feature-flags-python.md#build-and-run-the-app).
109
-
110
-
1. Create 10 different users and log into the application. As you log in with each user, you get a different message variant for some of them. ~50% of the time you get no message. 25% of the time you get the message "Hello!" and 25% of the time you get "I hope this makes your day!".
111
-
112
-
1. With some of the users select the **Like** button to trigger the telemetry event.
113
-
114
-
1. Open your Application Insights resource in the Azure portal and select **Logs** under **Monitoring**. In the query window, run the following query to see the telemetry events:
115
-
116
-
```kusto
117
-
// Step 1: Get distinct users and their Variant from FeatureEvaluation
> 
170
-
171
-
You see one "FeatureEvaluation" for each time the quote page was loaded and one "Liked" event for each time the like button was clicked. The "FeatureEvaluation" event have a custom property called `FeatureName` with the name of the feature flag that was evaluated. Both events have a custom property called `TargetingId` with the name of the user that liked the quote.
110
+
Deploy your application to begin collecting telemetry from your users. To test its functionality, you can simulate user activity by creating many test users. Each user will experience a different variant of greeting messages, and they can interact with the application by clicking the heart button to like a quote. As your user base grows, you can monitor the increasing volume of telemetry data collected in Azure App Configuration. Additionally, you can drill down into the data to analyze how each variant of the feature flag influences user behavior.
111
+
- [Review telemetry results in App Configuration](./howto-telemetry.md#review-telemetry-results-in-azure-app-configuration).
172
112
173
113
## Additional resources
174
114
- [Flask Quote of the Day sample](https://github.com/Azure-Samples/quote-of-the-day-python)
Telemetry is the automated process of collecting, transmitting, and analyzing data about how your application and its features are used. Enabling telemetry for feature flags offers valuable insights into the behavior and impact of feature rollouts, helping teams iterate faster, enhance user experience, detect issues early, and validate the effectiveness of new features. With telemetry, teams can answer critical questions such as:
15
+
16
+
- Is a feature enabled or disabled as expected?
17
+
- Are specific users or groups accessing the new feature?
18
+
- Is a feature causing performance regressions or errors?
19
+
- What is the impact of a feature on key metrics like engagement or conversion?
20
+
21
+
By leveraging telemetry data, organizations can make informed, data-driven decisions, quickly identify and resolve issues, and optimize feature delivery for better business and user outcomes.
22
+
14
23
Telemetry is the process of collecting, transmitting, and analyzing data about the usage and performance of your application. It helps you monitor feature flag behavior and make data-driven decisions. When a feature flag change is deployed, it's often important to analyze its effect on an application. For example, here are a few questions that may arise:
15
24
16
25
- Are my flags enabled/disabled as expected?
@@ -51,3 +60,122 @@ These types of questions can be answered through the emission and analysis of fe
51
60
52
61
*[Python](./howto-telemetry-python.md)
53
62
*[JavaScript](./howto-telemetry-javascript.md)
63
+
64
+
## Review telemetry results in Azure App Configuration
65
+
66
+
1. Navigate to the **Application Insights** blade in the App Configuration portal. You should see a graph displaying all events from your application. This graph provides an initial overview of activity patterns.
67
+
> [!div class="mx-imgBorder"]
68
+
> 
69
+
70
+
1. Use the time range selector to focus on specific periods to identify trends or investigate particular timeframes of interest.
71
+
72
+
1. Filter by feature Flag
73
+
1. Click on the dropdown menu above the event graph
74
+
1. Under **Feature flags with events** select your feature flag
75
+
1. The graph will now display only events related to the feature flag's evaluations
76
+
> [!div class="mx-imgBorder"]
77
+
> 
78
+
79
+
1. To access more detailed telemetry, click **View details** to open the telemetry tab.
80
+
> [!div class="mx-imgBorder"]
81
+
> 
82
+
83
+
> [!NOTE]
84
+
> You can also access this tab by going to the **Feature manager** and clicking **View events** in the telemetry column for the feature flag of interest.
85
+
> 
86
+
87
+
88
+
### Verify variant assignments
89
+
90
+
In the telemetry tab, you can view:
91
+
92
+
-**Total events**: Total number of evaluation events emitted by your application
93
+
-**Unique users**: Number of distinct users who were targeted and for whom events were emitted.
94
+
95
+
In order to show the distribution of users and number of evaluations across Simple, Long, and None variants, group the metrics by Variant. This grouping enables you to see whether the configured allocations are working as expected, and that all expected variants are being served to users.
96
+
97
+
> [!div class="mx-imgBorder"]
98
+
> 
99
+
100
+
In this example, we see that the number of users assigned the "None" variant is almost twice that of the "Simple" and "Long" variants given the configured 50-25-25 percentile split between "None", "Simple" and "Long" respectively.
101
+
102
+
103
+
### Confirm overrides and behavior based on flag state
104
+
105
+
Users may receive a variant for different reasons. You want to ensure that your variant assignments aren't only in the right proportion, but also for the right reason. You can group metrics by assignment reason. In this example, we see that the only assignment reason is solely due to Percentile allocations.
106
+
107
+
> [!div class="mx-imgBorder"]
108
+
> 
109
+
110
+
1. Disable the feature flag by going to the feature manager and toggling the feature flag "Enable" switch.
111
+
1. In the telemetry column, click **View events** to go to telemetry tab in read-only mode.
112
+
1. View Unique user count by Variant. You should see that all assignments for Long and Simple go to zero. Only the None variant, which is the default in our case, is assigned to users.
113
+
> [!div class="mx-imgBorder"]
114
+
> 
115
+
116
+
1. Switch to view unique user count by assignment reason.
117
+
Confirm from the graph that the Percentile allocations fall to zero and DefaultWhenDisabled is the only reason for which users are being assigned variants.
118
+
119
+
Other possible reasons include "DefaultWhenEnabled", "Group" or "User" if configured.
120
+
> [!div class="mx-imgBorder"]
121
+
> 
122
+
123
+
124
+
## Analyze telemetry in Application Insights
125
+
126
+
Now that you have confirmed the feature flag allocations are working as expected, you can dive deeper into the telemetry events to see how different variants are performing based on the likes emitted for users.
127
+
128
+
Open your Application Insights resource in the Azure portal and select **Logs** under **Monitoring**. In the query window, run the following query to see the telemetry events:
129
+
130
+
```kusto
131
+
// Step 1: Get distinct users and their Variant from FeatureEvaluation (Replace <store-endpoint> with your store's endpoint)
132
+
let evaluated_users =
133
+
customEvents
134
+
| where name == "FeatureEvaluation"
135
+
| where tostring(customDimensions.FeatureFlagReference) == "https://<store-endpoint>/kv/.appconfig.featureflag/Greeting"
> 
178
+
179
+
You see one "FeatureEvaluation" event for each time the quote page was loaded and one "Liked" event for each time the like button was clicked. The "FeatureEvaluation" events have a custom property called `FeatureName` with the name of the feature flag that was evaluated. Both events have a custom property called `TargetingId` with the name of the user that liked the quote.
180
+
181
+
In this example, we can see that, even though the number of users getting the Long variant vs Simple was roughly the same, the Simple variant appears to be performing better by a margin of 22%.
0 commit comments