Diagnostics and metrics scrub

Author: jlian

articles/iot-operations/.openpublishing.redirection.iot-operations.json

@@ -534,6 +534,11 @@
             "source_path_from_root": "/articles/iot-operations/reference/glossary.md",
             "redirect_url": "/azure/iot/iot-glossary?toc=/azure/iot-operations/toc.json&bc=/azure/iot-operations/breadcrumb/toc.json",
             "redirect_document_id": false
+        },
+        {
+            "source_path_from_root": "/articles/iot-operations/reference/observability-metrics-mq.md",
+            "redirect_url": "/azure/iot-operations/reference/observability-metrics-mqtt-broker",
+            "redirect_document_id": false
         }
     ]
 }

articles/iot-operations/manage-mqtt-broker/howto-broker-diagnostics.md

@@ -1,48 +1,66 @@
 ---
-title: Configure Azure IoT Operations MQTT broker diagnostic settings
-description: Learn how to configure diagnostic settings for the Azure IoT Operations MQTT broker, like logs, metrics, self-check, and tracing.
+title: Configure Azure IoT Operations MQTT broker diagnostics settings
+description: Learn how to configure diagnostics settings for the Azure IoT Operations MQTT broker, like logs, metrics, self-check, and tracing.
 author: PatAltimore
 ms.author: patricka
 ms.topic: how-to
 ms.service: azure-iot-operations
 ms.subservice: azure-mqtt-broker
-ms.date: 11/07/2024
+ms.date: 11/14/2024
 
 #CustomerIntent: As an operator, I want to configure diagnostics so that I can monitor MQTT broker communications.
 ---
 
-# Configure MQTT broker diagnostic settings
+# Configure MQTT broker diagnostics settings
 
 > [!IMPORTANT]
 > This setting requires modifying the Broker resource and can only be configured at initial deployment time using the Azure CLI or Azure Portal. A new deployment is required if Broker configuration changes are needed. To learn more, see [Customize default Broker](./overview-broker.md#customize-default-broker).
 
-Diagnostic settings allow you to enable metrics and tracing for MQTT broker.
+Diagnostic settings allow you to configure metrics, tracing, logging, and self-check for the MQTT broker.
 
-- Logs provide information about the operations performed by MQTT broker.
-- Metrics provide information about the resource utilization and throughput of MQTT broker.
-- Self-check periodically checks the health of MQTT broker by running a set of diagnostic tests.
-- Tracing provides detailed information about the requests and responses handled by MQTT broker.
+## Metrics
 
-## Change diagnostic settings
+Metrics provide information about the resource utilization and throughput of MQTT broker. These metrics are available in Prometheus format and can be scraped by Prometheus or Azure Monitor through OpenTelemetry Collector. To learn more, see [Configure observability and monitoring](../configure-observability-monitoring/howto-configure-observability.md).
 
-In most scenarios, the default diagnostic settings are sufficient. To override default diagnostic settings for MQTT broker, edit the `diagnostics` section in the Broker resource. Currently, changing settings is only supported using the `--broker-config-file` flag when you deploy the Azure IoT Operations using the `az iot ops create` command. 
+For the full list of metrics available, see [MQTT broker metrics](../reference/observability-metrics-mqtt-broker.md).
+
+## Logs and traces
+
+Logs provide information about the operations performed by MQTT broker. These logs are available in the Kubernetes cluster as container logs. They can be configured to be sent to Azure Monitor Logs with Container Insights.
+
+Traces are for [distributed tracing](https://opentelemetry.io/docs/concepts/signals/traces/) and provide detailed information about the requests and responses handled by MQTT broker. These traces can be sent to Azure Monitor through OpenTelemetry Collector.
+
+To learn more, see [Configure observability and monitoring](../configure-observability-monitoring/howto-configure-observability.md).
+
+## Self-check
+
+The self-check probe is a part of the MQTT broker and is enabled by default. It uses OpenTelemetry (OTel) events to monitor the system. The probe sends test messages to check the system's behavior and timing.
+
+The validation process checks if the system works correctly by comparing the test results with expected outcomes. These outcomes include:
+
+1. The paths messages take through the system.
+2. The system's timing behavior.
+
+The system uses MQTT for messaging, including the replication protocol. The probe generates test traffic for all MQTT operations (CONNECT, PUBLISH, SUBSCRIBE, PING) and monitors them to ensure they follow the correct paths.
+
+> [!IMPORTANT]
+> The self-check probe publishes messages to the `azedge/dmqtt/selftest` topic. Don't publish or subscribe to diagnostic probe topics that start with `azedge/dmqtt/selftest`. Publishing or subscribing to these topics might affect the probe or self-test checks resulting in invalid results. Invalid results might be listed in diagnostic probe logs, metrics, or dashboards. For example, you might see the issue *Path verification failed for probe event with operation type 'Publish'* in the diagnostics-probe logs. For more information, see [Known Issues](../troubleshoot/known-issues.md#mqtt-broker).
+
+## Change diagnostics settings
+
+In most scenarios, the default diagnostics settings are sufficient. To override default diagnostics settings for MQTT broker, edit the `diagnostics` section in the Broker resource. Currently, changing settings is only supported using the `--broker-config-file` flag when you deploy the Azure IoT Operations using the `az iot ops create` command. 
 
 To override, first prepare a Broker config file following the [BrokerDiagnostics](/rest/api/iotoperations/broker/create-or-update#brokerdiagnostics) API reference. For example:
 
 ```json
 {
   "diagnostics": {
-    "logs": {
-      "level": "debug"
-    },
     "metrics": {
       "prometheusPort": 9600
     },
-    "selfCheck": {
-      "mode": "Enabled",
-      "intervalSeconds": 30,
-      "timeoutSeconds": 15
-    },
+      "logs": {
+        "level": "debug"
+      },
     "traces": {
       "mode": "Enabled",
       "cacheSizeMegabytes": 16,
@@ -51,6 +69,11 @@ To override, first prepare a Broker config file following the [BrokerDiagnostics
         "intervalSeconds": 30
       },
       "spanChannelCapacity": 1000
+    },
+    "selfCheck": {
+      "mode": "Enabled",
+      "intervalSeconds": 30,
+      "timeoutSeconds": 15
     }
   }
 }
@@ -62,8 +85,6 @@ Then, deploy Azure IoT Operations using the `az iot ops create` command with the
 az iot ops create ... --broker-config-file <FILE>.json
 ```
 
-The self-test diagnostics probe publishes messages to the `azedge/dmqtt/selftest` topic. You shouldn't publish or subscribe to this topic as it's used for internal diagnostics. For more information, see [Known Issues](../troubleshoot/known-issues.md#mqtt-broker).
-
 To learn more, see [Azure CLI support for advanced MQTT broker configuration](https://aka.ms/aziotops-broker-config) and [Broker examples](/rest/api/iotoperations/broker/create-or-update#examples).
 
 ## Next steps