Merge pull request #172360 from ralarcon/func-prod-monitoring

Add clarification regarding log level changes and guidance for high volume telemetry

Author: PMEds28

articles/azure-functions/configure-monitoring.md

@@ -11,7 +11,7 @@ ms.custom: "contperf-fy21q2"
 
 Azure Functions integrates with Application Insights to better enable you to monitor your function apps. Application Insights, a feature of Azure Monitor, is an extensible Application Performance Management (APM) service that collects data generated by your function app, including information your app writes to logs. Application Insights integration is typically enabled when your function app is created. If your app doesn't have the instrumentation key set, you must first [enable Application Insights integration](#enable-application-insights-integration). 
 
-You can use Application Insights without any custom configuration. The default configuration can result in high volumes of data. If you're using a Visual Studio Azure subscription, you might hit your data cap for Application Insights. To learn more about Application Insights costs, see [Manage usage and costs for Application Insights](../azure-monitor/app/pricing.md).
+You can use Application Insights without any custom configuration. The default configuration can result in high volumes of data. If you're using a Visual Studio Azure subscription, you might hit your data cap for Application Insights. To learn more about Application Insights costs, see [Manage usage and costs for Application Insights](../azure-monitor/app/pricing.md). For more information, see [Solutions with high-volume of telemetry](#solutions-with-high-volume-of-telemetry).
 
 Later in this article, you learn how to configure and customize the data that your functions send to Application Insights. For a function app, logging is configured in the [host.json] file. 
 
@@ -137,6 +137,16 @@ If [host.json] includes multiple logs that start with the same string, the more
 
 You can use a log level setting of `None` prevent any logs from being written for a category. 
 
+> [!CAUTION]
+> Azure Functions integrates with Application Insights by storing telemetry events in Application Insights tables, setting a category log level to any value different from `Information` will prevent the telemetry to flow to those tables, as outcome, you will not be able to see the related data in Application Insights or Function Monitor tab.
+>
+> From above samples:
+> * If `Host.Results` category is set to `Error` log level, it will only gather host execution telemetry events in the `requests` table for failed function executions, preventing to display host execution details of success executions in both Application Insights and Function Monitor tab.
+> * If `Function` category is set to `Error` log level, it will stop gathering function telemetry data related to `dependencies`, `customMetrics`, and `customEvents` for all the functions, preventing to see any of this data in Application Insights. It will only gather `traces` logged with `Error` level. 
+>
+> In both cases you will continue to collect errors and exceptions data in Application Insights and Function Monitor tab. For more information, see [Solutions with high-volume of telemetry](#solutions-with-high-volume-of-telemetry).
+
+
 ## Configure the aggregator
 
 As noted in the previous section, the runtime aggregates data about function executions over a period of time. The default period is 30 seconds or 1,000 runs, whichever comes first. You can configure this setting in the [host.json] file.  Here's an example:
@@ -276,6 +286,132 @@ When you enable Application Insights, disable the built-in logging that uses Azu
 
 To disable built-in logging, delete the `AzureWebJobsDashboard` app setting. For information about how to delete app settings in the Azure portal, see the **Application settings** section of [How to manage a function app](functions-how-to-use-azure-function-app-settings.md#settings). Before you delete the app setting, make sure no existing functions in the same function app use the setting for Azure Storage triggers or bindings.
 
+## Solutions with high volume of telemetry 
+
+Your function apps can be an essential part of solutions that by nature cause high volumes of telemetry (IoT solutions, event driven based solutions, high load financial systems, integration systems...). In this case, you should consider extra configuration to reduce costs while maintaining observability.
+
+Depending on how the telemetry generated is going to be consumed, real-time dashboards, alerting, detailed diagnostics, and so on, you will need to define a strategy to reduce the volume of data generated. That strategy will allow you to properly monitor, operate, and diagnose your function apps in production. You can consider the following options:
+
+* **Use sampling**: as mentioned [earlier](#configure-sampling), it will help to dramatically reduce the volume of telemetry events ingested while maintaining a statistically correct analysis. It could happen that even using sampling you still get high volume of telemetry. Inspect the options that [Adaptive sampling](../azure-monitor/app/sampling.md#configuring-adaptive-sampling-for-aspnet-applications) provides to you, for example set the `maxTelemetryItemsPerSecond` to a value that balances the volume generated with your monitoring needs. Keep in mind that the telemetry sampling is applied per host executing your function app. 
+
+* **Default log level**: use `Warning` or `Error` as the default value for all telemetry categories. Now you can decide which [categories](#configure-categories) you want to set at `Information` so you can monitor and diagnose your functions properly.
+
+* **Tune your functions telemetry**: with the default log level set to `Error` or `Warning`, no detailed information from each function will be gathered (dependencies, custom metrics, custom events, and traces). For those functions that are key for production monitoring, define an explicit entry for `Function.<YOUR_FUNCTION_NAME>` category and set it to `Information`, so you can gather detailed information. At this point, to avoid gathering [user-generated logs](functions-monitoring.md#writing-to-logs) at `Information` level, set the `Function.<YOUR_FUNCTION_NAME>.User` category to `Error` or `Warning` log level.
+
+* **Host.Aggregator category**: as described in [Configure categories](#configure-categories), this category provides aggregated information of function invocations. The information from this category is gathered in Application Insights `customMetrics` table, and it's shown in the function Overview tab in the Azure portal. Depending on how you configure the aggregator, consider that there will be a delay, determined by the `flushTimeout`, in the telemetry gathered. If you set this category to other value different than `Information`, you will stop gathering the data in the `customMetrics` table and will not display metrics in the function Overview tab.
+
+  The following screenshot shows Host.Aggregator telemetry data displayed in the function Overview tab.
+  :::image type="content" source="media/configure-monitoring/host-aggregator-function-overview.png" alt-text="Screenshot of Host.Aggregator telemetry displayed in function Overview tab." lightbox="media/configure-monitoring/host-aggregator-function-overview-big.png":::
+
+  The following screenshot shows Host.Aggregator telemetry data in Application Insights customMetrics table.
+  :::image type="content" source="media/configure-monitoring/host-aggregator-custom-metrics.png" alt-text="Screenshot of Host.Aggregator telemetry in customMetrics Application Insights table." lightbox="media/configure-monitoring/host-aggregator-custom-metrics-big.png":::
+
+* **Host.Results category**: as described in [Configure categories](#configure-categories), this category provides the runtime-generated logs indicating the success or failure of a function invocation. The information from this category is gathered in the Application Insights `requests` table, and it is shown in the function Monitor tab and in different Application Insights dashboards (Performance, Failures...). If you set this category to other value different than `Information`, you will only gather telemetry generated at the log level defined (or higher), for example, setting it to `error` results in tracking requests data only for failed executions. 
+
+  The following screenshot shows the Host.Results telemetry data displayed in the function Monitor tab.
+  :::image type="content" source="media/configure-monitoring/host-results-function-monitor.png" alt-text="Screenshot of Host.Results telemetry in function Monitor tab." lightbox="media/configure-monitoring/host-results-function-monitor-big.png":::
+
+  The following screenshot shows Host.Results telemetry data displayed in Application Insights Performance dashboard.
+  :::image type="content" source="media/configure-monitoring/host-results-application-insights.png" alt-text="Screenshot of Host.Results telemetry in Application Insights Performance dashboard." lightbox="media/configure-monitoring/host-results-application-insights-big.png":::
+
+* **Host.Aggregator vs Host.Results**: both categories provide good insights about function executions, if needed, you can remove the detailed information from one of these categories, so you can use the other for monitoring and alerting.
+Here's a sample:
+# [v2.x+](#tab/v2)
+
+``` json
+{
+  "version": "2.0",  
+  "logging": {
+    "logLevel": {
+      "default": "Warning",
+      "Function": "Error",
+      "Host.Aggregator": "Error",
+      "Host.Results": "Information", 
+      "Function.Function1": "Information",
+      "Function.Function1.User": "Error"
+    },
+    "applicationInsights": {
+      "samplingSettings": {
+        "isEnabled": true,
+        "maxTelemetryItemsPerSecond": 1,
+        "excludedTypes": "Exception"
+      }
+    }
+  }
+} 
+```
+# [v1.x](#tab/v1) 
+```json
+{
+  "logger": {
+    "categoryFilter": {
+      "defaultLevel": "Warning",
+      "categoryLevels": {
+        "Function": "Error",
+        "Host.Aggregator": "Error",
+        "Host.Results": "Information",
+        "Host.Executor": "Warning"
+      }
+    }
+  },
+  "applicationInsights": {
+    "sampling": {
+      "isEnabled": true,
+      "maxTelemetryItemsPerSecond" : 5
+    }
+  }
+}
+```
+---
+
+With this configuration, you will have:
+
+* The default value for all functions and telemetry categories is set to `Warning` (including Microsoft and Worker categories) so, by default, all errors and warnings generated by both, the runtime and custom logging, are gathered. 
+
+* The `Function` category log level is set to `Error`, so for all functions, by default, only exceptions and error logs will be gathered (dependencies, user-generated metrics, and user-generated events will be skipped).
+
+* For the `Host.Aggregator` category, as it is set to `Error` log level, no aggregated information from function invocations will be gathered in the `customMetrics` Application Insights table, and no information about executions counts (total, successful, failed...) will be shown in the function overview dashboard.
+
+* For the `Host.Results` category, all the host execution information is gathered in the `requests` Application Insights table. All the invocations results will be shown in the function Monitor dashboard and in Application Insights dashboards.
+
+* For the function called `Function1`, we have set the log level to `Information` so, for this concrete function, all the telemetry is gathered (dependency, custom metrics, custom events). For the same function, the `Function1.User` category (user-generated traces) is set to `Error`, so only custom error logging will be gathered. Note that per function configuration is not supported in v1.x. 
+
+* Sampling is configured to send one telemetry item per second per type, excluding the exceptions. This sampling will happen for each server host running our function app, so if we have four instances, this configuration will emit four telemetry items per second per type and all the exceptions that might occur. Note that, for metrics counts such as request rate and exception rate are adjusted to compensate for the sampling rate, so that they show approximately correct values in Metric Explorer.  
+
+> [!TIP]
+> Experiment with different configurations to ensure you cover your requirements for logging, monitoring and alerting. Ensure you have detailed diagnostics in case of unexpected errors or malfunctioning.
+
+### Overriding monitoring configuration at runtime
+Finally, there could be situations where you need to quickly change the logging behavior of a certain category in production, and you don't want to make a whole deployment just for a change in the `host.json` file. For such as cases, you can override the [host json values](functions-host-json.md#override-hostjson-values).
+
+
+To configure these values at App settings level (and avoid redeployment on just host.json changes), you should override specific `host.json` values by creating an equivalent value as an application setting. When the runtime finds an application setting in the format `AzureFunctionsJobHost__path__to__setting`, it overrides the equivalent `host.json` setting located at `path.to.setting` in the JSON. When expressed as an application setting, the dot (`.`) used to indicate JSON hierarchy is replaced by a double underscore (`__`). For example, you can use the below app settings to configure individual function log levels as in `host.json` above.
+
+
+| Host.json path | App setting |
+|----------------|-------------|
+| logging.logLevel.default  | AzureFunctionsJobHost__logging__logLevel__default  |
+| logging.logLeve.Host.Aggregator | AzureFunctionsJobHost__logging__logLevel__Host__Aggregator |
+| logging.logLevel.Function | AzureFunctionsJobHost__logging__logLevel__Function |
+| logging.logLevel.Function.Function1 | AzureFunctionsJobHost__logging__logLevel__Function1 |
+| logging.logLevel.Function.Function1.User | AzureFunctionsJobHost__logging__logLevel__Function1.User |
+
+
+You can override the settings directly at the Azure portal Function App Configuration blade or by using an Azure CLI or PowerShell script.
+
+# [az cli](#tab/v2)
+```azurecli-interactive
+az functionapp config appsettings set --name MyFunctionApp --resource-group MyResourceGroup --settings "AzureFunctionsJobHost__logging__logLevel__Host__Aggregator=Information"
+```
+# [PowerShell](#tab/v1) 
+```powershell
+Update-AzFunctionAppSetting -Name MyAppName -ResourceGroupName MyResourceGroupName -AppSetting @{"AzureFunctionsJobHost__logging__logLevel__Host__Aggregator" = "Information"}
+```
+---
+
+> [!NOTE]
+> Overriding the `host.json` through changing app settings will restart your function app.
+ 
 ## Next steps
 
 To learn more about monitoring, see: