Skip to content

Commit 0a1a710

Browse files
committed
App Insights Pre aggregated and log metrics from Dale Update
1 parent 80662a5 commit 0a1a710

File tree

1 file changed

+15
-9
lines changed

1 file changed

+15
-9
lines changed

articles/azure-monitor/app/pre-aggregated-metrics-log-metrics.md

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -11,29 +11,29 @@ ms.reviewer: mbullwin
1111

1212
# Log-based and pre-aggregated metrics in Application Insights
1313

14-
This article explains the difference between traditional Application Insights metrics that are based on logs, and pre-aggregated metrics that are currently in public preview. Both types of metrics are available to the users of Application Insights, and each brings a unique value in monitoring application health, diagnostics and analytics. The developers who are instrumenting applications can decide which type of metric is best suited to a particular scenario, depending on the size of the application, expected volume of telemetry, and business requirements for metrics precision and alerting.
14+
This article explains the difference between "traditional" Application Insights metrics that are based on logs, and pre-aggregated metrics that are currently in public preview. Both types of metrics are available to the users of Application Insights, and each brings a unique value in monitoring application health, diagnostics and analytics. The developers who are instrumenting applications can decide which type of metric is best suited to a particular scenario, depending on the size of the application, expected volume of telemetry, and business requirements for metrics precision and alerting.
1515

1616
## Log-based Metrics
1717

1818
Until recently, the application monitoring telemetry data model in Application Insights was solely based on a small number of predefined types of events, such as requests, exceptions, dependency calls, page views, etc. Developers can use the SDK to either emit these events manually (by writing code that explicitly invokes the SDK) or they can rely on the automatic collection of events from auto-instrumentation. In either case, the Application Insights backend stores all collected events as logs, and the Application Insights blades in the Azure portal act as an analytical and diagnostic tool for visualizing event-based data from logs.
1919

20-
Using logs to retain a complete set of events can bring great analytical and diagnostic value. For example, you can get an exact count of requests to a particular URL with the number of distinct users who made these calls. Or you can get detailed diagnostic traces, including exceptions and dependency calls for any user session. Having this type of information can significantly improve visibility into the application health and usage, allowing to cut down the time necessary to diagnose issues with an app.
20+
Using logs to retain a complete set of events can bring great analytical and diagnostic value. For example, you can get an exact count of requests to a particular URL with the number of distinct users who made these calls. Or you can get detailed diagnostic traces, including exceptions and dependency calls for any user session. Having this type of information can significantly improve visibility into the application health and usage, allowing to cut down the time necessary to diagnose issues with an app.
2121

22-
At the same time, collecting a complete set of events may be impractical (or even impossible) for applications that generate a lot of telemetry. For situations when the volume of events is too high, Application Insights implements several telemetry volume reduction techniques, such as [sampling](https://docs.microsoft.com/azure/application-insights/app-insights-sampling) and [filtering](https://docs.microsoft.com/azure/application-insights/app-insights-api-filtering-sampling) that reduce the number of collected and stored events. Unfortunately, lowering the number of stored events also lowers the accuracy of the metrics that, behind the scenes, must perform query-time aggregations of the events stored in logs.
22+
At the same time, collecting a complete set of events may be impractical (or even impossible) for applications that generate a large volume of telemetry. For situations when the volume of events is too high, Application Insights implements several telemetry volume reduction techniques, such as [sampling](https://docs.microsoft.com/azure/application-insights/app-insights-sampling) and [filtering](https://docs.microsoft.com/azure/application-insights/app-insights-api-filtering-sampling) that reduce the number of collected and stored events. Unfortunately, lowering the number of stored events also lowers the accuracy of the metrics that, behind the scenes, must perform query-time aggregations of the events stored in logs.
2323

2424
> [!NOTE]
2525
> In Application Insights, the metrics that are based on the query-time aggregation of events and measurements stored in logs are called log-based metrics. These metrics typically have many dimensions from the event properties, which makes them superior for analytics, but the accuracy of these metrics is negatively affected by sampling and filtering.
2626
2727
## Pre-aggregated metrics
2828

29-
In addition to log-based metrics, in Fall of 2018, the Application Insights team shipped a public preview of metrics that are stored in a specialized repository that is optimized for time series. The new metrics are no longer kept as individual events with lots of properties. Instead, they are stored as pre-aggregated time series, and only with key dimensions. This makes the new metrics superior at query time: retrieving data happens much faster and requires less compute power. This consequently enables new scenarios such as [near real-time alerting on dimensions of metrics](https://docs.microsoft.com/azure/monitoring-and-diagnostics/monitoring-near-real-time-metric-alerts), more responsive [dashboards](https://docs.microsoft.com/azure/azure-monitor/app/overview-dashboard), and more.
29+
In addition to log-based metrics, in late 2018, the Application Insights team shipped a public preview of metrics that are stored in a specialized repository that is optimized for time series. The new metrics are no longer kept as individual events with lots of properties. Instead, they are stored as pre-aggregated time series, and only with key dimensions. This makes the new metrics superior at query time: retrieving data happens much faster and requires less compute power. This consequently enables new scenarios such as [near real-time alerting on dimensions of metrics](https://docs.microsoft.com/azure/monitoring-and-diagnostics/monitoring-near-real-time-metric-alerts), more responsive [dashboards](https://docs.microsoft.com/azure/azure-monitor/app/overview-dashboard), and more.
3030

3131
> [!IMPORTANT]
32-
> Both, log-based and pre-aggregated metrics coexist in Application Insights. To differentiate the two, in the Application Insights UX the pre-aggregated metrics are now called Standard metrics (preview), while the traditional metrics from the events were renamed to Log-based metrics.
32+
> Both, log-based and pre-aggregated metrics coexist in Application Insights. To differentiate the two, in the Application Insights UX the pre-aggregated metrics are now called "Standard metrics (preview)", while the traditional metrics from the events were renamed to "Log-based metrics".
3333
34-
The newer SDKs ([Application Insights 2.7](https://www.nuget.org/packages/Microsoft.ApplicationInsights/2.7.2) SDK or later for .NET) pre-aggregate metrics during collection before telemetry volume reduction techniques kick in. This means that the accuracy of the new metrics isnt affected by sampling and filtering when using the latest Application Insights SDKs.
34+
The newer SDKs ([Application Insights 2.7](https://www.nuget.org/packages/Microsoft.ApplicationInsights/2.7.2) SDK or later for .NET) pre-aggregate metrics during collection before telemetry volume reduction techniques kick in. This means that the accuracy of the new metrics isn't affected by sampling and filtering when using the latest Application Insights SDKs.
3535

36-
For the SDKs that dont implement pre-aggregation (that is, older versions of Application Insights SDKs or for browser instrumentation) the Application Insights backend still populates the new metrics by aggregating the events received by the Application Insights event collection endpoint. This means that while you dont benefit from the reduced volume of data transmitted over the wire, you can still use the pre-aggregated metrics and experience better performance and support of the near real-time dimensional alerting with SDKs that dont pre-aggregate metrics during collection.
36+
For the SDKs that don't implement pre-aggregation (that is, older versions of Application Insights SDKs or for browser instrumentation) the Application Insights backend still populates the new metrics by aggregating the events received by the Application Insights event collection endpoint. This means that while you don't benefit from the reduced volume of data transmitted over the wire, you can still use the pre-aggregated metrics and experience better performance and support of the near real-time dimensional alerting with SDKs that don't pre-aggregate metrics during collection.
3737

3838
It is worth mentioning that the collection endpoint pre-aggregates events before ingestion sampling, which means that [ingestion sampling](https://docs.microsoft.com/azure/application-insights/app-insights-sampling) will never impact the accuracy of pre-aggregated metrics, regardless of the SDK version you use with your application.
3939

@@ -45,7 +45,7 @@ There are several [ways of sending custom metrics from the Application Insights
4545

4646
## Custom metrics dimensions and pre-aggregation
4747

48-
All metrics that you send using [trackMetric](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#trackmetric) or [GetMetric and TrackValue](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#getmetric) API calls are automatically stored in both logs and metrics stores. However, while the log-based version of your custom metric always retains all dimensions, the pre-aggregated version of the metric is stored by default with no dimensions. You can turn on collection of dimensions of custom metrics on the [usage and estimated cost](https://docs.microsoft.com/azure/application-insights/app-insights-pricing) tab by checking Enable alerting on custom metric dimensions:
48+
All metrics that you send using [trackMetric](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#trackmetric) or [GetMetric and TrackValue](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#getmetric) API calls are automatically stored in both logs and metrics stores. However, while the log-based version of your custom metric always retains all dimensions, the pre-aggregated version of the metric is stored by default with no dimensions. You can turn on collection of dimensions of custom metrics on the [usage and estimated cost](https://docs.microsoft.com/azure/application-insights/app-insights-pricing) tab by checking "Enable alerting on custom metric dimensions":
4949

5050
![Usage and estimated cost](./media/pre-aggregated-metrics-log-metrics/001-cost.png)
5151

@@ -59,7 +59,13 @@ Use [Azure Monitor Metrics Explorer](../platform/metrics-getting-started.md) to
5959

6060
![Metric namespace](./media/pre-aggregated-metrics-log-metrics/002-metric-namespace.png)
6161

62+
## Pricing models for Application Insights metrics
63+
64+
Ingesting metrics into Application Insights, whether log-based or pre-aggregated, will generate costs based on the size of the ingested data, as described [here](https://docs.microsoft.com/azure/azure-monitor/app/pricing#pricing-model). Your custom metrics, including all its dimensions, are always stored in the Application Insights log-store; additionally, a pre-aggregated version of your custom metrics (with no dimensions) is forwarded to the metrics store by default.
65+
66+
Selecting the [Enable alerting on custom metric dimensions](#custom-metrics-dimensions-and-pre-aggregation) option to store all dimensions of the pre-aggregated metrics in the metric store, can generate **additional** costs based on [Custom Metrics pricing](https://azure.microsoft.com/pricing/details/monitor/).
67+
6268
## Next steps
6369

6470
* [Near real-time alerting](https://docs.microsoft.com/azure/monitoring-and-diagnostics/monitoring-near-real-time-metric-alerts)
65-
* [GetMetric and TrackValue](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#getmetric)
71+
* [GetMetric and TrackValue](https://docs.microsoft.com/azure/application-insights/app-insights-api-custom-events-metrics#getmetric)

0 commit comments

Comments
 (0)