Merge pull request #290558 from jlian/release-aio-ga

Diagnostics and metrics scrub

Author: Jill Grant

articles/iot-operations/.openpublishing.redirection.iot-operations.json

@@ -534,6 +534,11 @@
             "source_path_from_root": "/articles/iot-operations/reference/glossary.md",
             "redirect_url": "/azure/iot/iot-glossary?toc=/azure/iot-operations/toc.json&bc=/azure/iot-operations/breadcrumb/toc.json",
             "redirect_document_id": false
+        },
+        {
+            "source_path_from_root": "/articles/iot-operations/reference/observability-metrics-mq.md",
+            "redirect_url": "/azure/iot-operations/reference/observability-metrics-mqtt-broker",
+            "redirect_document_id": false
         }
     ]
 }

articles/iot-operations/manage-mqtt-broker/howto-broker-diagnostics.md

@@ -1,48 +1,64 @@
 ---
-title: Configure Azure IoT Operations MQTT broker diagnostic settings
-description: Learn how to configure diagnostic settings for the Azure IoT Operations MQTT broker, like logs, metrics, self-check, and tracing.
+title: Configure Azure IoT Operations MQTT broker diagnostics settings
+description: Learn how to configure diagnostics settings for the Azure IoT Operations MQTT broker, like logs, metrics, self-check, and tracing.
 author: PatAltimore
 ms.author: patricka
 ms.topic: how-to
 ms.service: azure-iot-operations
 ms.subservice: azure-mqtt-broker
-ms.date: 11/07/2024
+ms.date: 11/14/2024
 
 #CustomerIntent: As an operator, I want to configure diagnostics so that I can monitor MQTT broker communications.
 ---
 
-# Configure MQTT broker diagnostic settings
+# Configure MQTT broker diagnostics settings
 
 > [!IMPORTANT]
 > This setting requires modifying the Broker resource and can only be configured at initial deployment time using the Azure CLI or Azure Portal. A new deployment is required if Broker configuration changes are needed. To learn more, see [Customize default Broker](./overview-broker.md#customize-default-broker).
 
-Diagnostic settings allow you to enable metrics and tracing for MQTT broker.
+Diagnostic settings allow you to configure metrics, logs, and self-check for the MQTT broker.
 
-- Logs provide information about the operations performed by MQTT broker.
-- Metrics provide information about the resource utilization and throughput of MQTT broker.
-- Self-check periodically checks the health of MQTT broker by running a set of diagnostic tests.
-- Tracing provides detailed information about the requests and responses handled by MQTT broker.
+## Metrics
 
-## Change diagnostic settings
+Metrics provide information about the current and past health and status of the MQTT broker. These metrics are emitted in OpenTelemetry format (OTLP). They can be converted to Prometheus format using an OpenTelemetry Collector and routed to Azure Managed Grafana Dashboards using Azure Monitor Managed Service for Prometheus. To learn more, see [Configure observability and monitoring](../configure-observability-monitoring/howto-configure-observability.md).
 
-In most scenarios, the default diagnostic settings are sufficient. To override default diagnostic settings for MQTT broker, edit the `diagnostics` section in the Broker resource. Currently, changing settings is only supported using the `--broker-config-file` flag when you deploy the Azure IoT Operations using the `az iot ops create` command. 
+For the full list of metrics available, see [MQTT broker metrics](../reference/observability-metrics-mqtt-broker.md).
+
+## Logs
+
+Logs provide information about the operations performed by MQTT broker. These logs are available in the Kubernetes cluster as container logs. They can be configured to be sent to Azure Monitor Logs with Container Insights.
+
+To learn more, see [Configure observability and monitoring](../configure-observability-monitoring/howto-configure-observability.md).
+
+## Self-check
+
+The MQTT broker's self-check mechanism is enabled by default. It uses a diagnostics probe and OpenTelemetry (OTel) traces to monitor the broker. The probe sends test messages to check the system's behavior and timing.
+
+The validation process checks if the system works correctly by comparing the test results with expected outcomes. These outcomes include:
+
+1. The paths messages take through the system.
+2. The system's timing behavior.
+
+The diagnostics probe periodically executes MQTT operations (PING, CONNECT, PUBLISH, SUBSCRIBE, UNSUBSCRIBE) on the MQTT broker and monitors the corresponding ACKs and traces to check for latency, message loss, and correctness of the replication protocol.
+
+> [!IMPORTANT]
+> The self-check diagnostics probe publishes messages to the `azedge/dmqtt/selftest` topic. Don't publish or subscribe to diagnostic probe topics that start with `azedge/dmqtt/selftest`. Publishing or subscribing to these topics might affect the probe or self-test checks resulting in invalid results. Invalid results might be listed in diagnostic probe logs, metrics, or dashboards. For example, you might see the issue *Path verification failed for probe event with operation type 'Publish'* in the diagnostics-probe logs. For more information, see [Known Issues](../troubleshoot/known-issues.md#mqtt-broker).
+
+## Change diagnostics settings
+
+In most scenarios, the default diagnostics settings are sufficient. To override default diagnostics settings for MQTT broker, edit the `diagnostics` section in the Broker resource. Currently, changing settings is only supported using the `--broker-config-file` flag when you deploy the Azure IoT Operations using the `az iot ops create` command. 
 
 To override, first prepare a Broker config file following the [BrokerDiagnostics](/rest/api/iotoperations/broker/create-or-update#brokerdiagnostics) API reference. For example:
 
 ```json
 {
   "diagnostics": {
-    "logs": {
-      "level": "debug"
-    },
     "metrics": {
       "prometheusPort": 9600
     },
-    "selfCheck": {
-      "mode": "Enabled",
-      "intervalSeconds": 30,
-      "timeoutSeconds": 15
-    },
+      "logs": {
+        "level": "debug"
+      },
     "traces": {
       "mode": "Enabled",
       "cacheSizeMegabytes": 16,
@@ -51,6 +67,11 @@ To override, first prepare a Broker config file following the [BrokerDiagnostics
         "intervalSeconds": 30
       },
       "spanChannelCapacity": 1000
+    },
+    "selfCheck": {
+      "mode": "Enabled",
+      "intervalSeconds": 30,
+      "timeoutSeconds": 15
     }
   }
 }
@@ -62,8 +83,6 @@ Then, deploy Azure IoT Operations using the `az iot ops create` command with the
 az iot ops create ... --broker-config-file <FILE>.json
 ```
 
-The self-test diagnostics probe publishes messages to the `azedge/dmqtt/selftest` topic. You shouldn't publish or subscribe to this topic as it's used for internal diagnostics. For more information, see [Known Issues](../troubleshoot/known-issues.md#mqtt-broker).
-
 To learn more, see [Azure CLI support for advanced MQTT broker configuration](https://aka.ms/aziotops-broker-config) and [Broker examples](/rest/api/iotoperations/broker/create-or-update#examples).
 
 ## Next steps

articles/iot-operations/manage-mqtt-broker/overview-broker.md

@@ -181,8 +181,9 @@ Customizing the default Broker resource isn't required for most setups. The sett
 - [**Cardinality**](./howto-configure-availability-scale.md#configure-scaling-settings): Determines the broker's capacity to handle more connections and messages, and it enhances high availability if there are pod or node failures.
 - [**Memory profile**](./howto-configure-availability-scale.md#configure-memory-profile): Sets the maximum memory usage of the broker and how to handle memory usage as the broker scales up.
 - [**Disk-backed message buffer**](./howto-disk-backed-message-buffer.md): Configuration for buffering messages to disk as RAM fills up.
-- [**Encryption of internal traffic**](./howto-encrypt-internal-traffic.md): Configuration for encrypting internal traffic between broker frontend and backend pods.
+- [**Diagnostics settings**](./howto-broker-diagnostics.md): Configuration for diagnostic settings like log level and metrics endpoint.
 - [**Advanced MQTT client options**](./howto-broker-mqtt-client-options.md): Configuration for advanced MQTT client options like session expiry, message expiry, and keep-alive settings.
+- [**Encryption of internal traffic**](./howto-encrypt-internal-traffic.md): Configuration for encrypting internal traffic between broker frontend and backend pods.
 
 Customizing the default broker must be done during initial deployment time using Azure CLI or Azure portal. A new deployment is required if different Broker configuration settings are needed.