Merge pull request #280660 from PatAltimore/patricka-mq-diagnostics-release-aio-july-updates

MQTT broker diagnostics changes

Author: ShannonLeavitt

articles/iot-operations/.openpublishing.redirection.iot-operations.json

@@ -12,7 +12,7 @@
         },
         {
             "source_path_from_root": "/articles/iot-operations/manage-mqtt-connectivity/howto-configure-diagnostics.md",
-            "redirect_url": "/azure/iot-operations/configure-observability-monitoring/howto-configure-diagnostics",
+            "redirect_url": "/azure/iot-operations/manage-mqtt-broker/howto-configure-availability-scale",
             "redirect_document_id": false
         },
         {
@@ -202,8 +202,8 @@
         },
         {
             "source_path_from_root": "/articles/iot-operations/monitor/howto-configure-diagnostics.md",
-            "redirect_url": "/azure/iot-operations/configure-observability-monitoring/howto-configure-diagnostics",
-            "redirect_document_id": true
+            "redirect_url": "/azure/iot-operations/manage-mqtt-broker/howto-configure-availability-scale",
+            "redirect_document_id": false
         },
         {
             "source_path_from_root": "/articles/iot-operations/monitor/howto-clean-up-observability-resources.md",

articles/iot-operations/configure-observability-monitoring/howto-configure-diagnostics.md

@@ -7,7 +7,7 @@ ms.topic: how-to
 ms.subservice: azure-mqtt-broker
 ms.custom:
   - ignite-2023
-ms.date: 07/02/2024
+ms.date: 07/11/2024
 
 #CustomerIntent: As an operator, I want to understand the settings for the MQTT broker so that I can configure it for high availability and scale.
 ---
@@ -133,41 +133,43 @@ Then, run the following command to deploy the broker:
 kubectl apply -f <path-to-yaml-file>
 ```
 
-## Configure MQ broker diagnostic settings
+## Configure MQTT broker diagnostic settings
+
+MQTT broker includes diagnostics that provides metrics for MQTT broker components. Operators can use these metrics to monitor the health of the system. A Prometheus endpoint is provided for metrics from all MQTT broker components including self-test metrics.
+
+Diagnostic settings allow you to enable metrics and tracing for MQTT broker.
+
+- Metrics provide information about the resource utilization and throughput of MQTT broker.
+- Tracing provides detailed information about the requests and responses handled by MQTT broker.
+- Adjust the log level of MQTT broker to control the amount and detail of information that is logged. The log level can be set for different components of MQTT broker. The default log level is `info`.
+
+You can configure diagnostics using the *Broker* custom resource definition (CRD). The following table shows the properties of the broker diagnostic settings and all default values.
+
+| Name                                 | Format           | Default | Description                                                     |
+| ------------------------------------ | ---------------- | ------- | --------------------------------------------------------------- |
+| logs.exportIntervalSeconds           | integer          | 30      | How often to export the logs to the open telemetry collector    |
+| logs.exportLogLevel                  | string           | error   | The level of logs to export                                     |
+| logs.level                           | string           | info    | The log level. For example, `debug`, `info`, `warn`, `error`, `trace` |
+| logs.openTelemetryCollectorAddress   | string           |         | The open telemetry collector endpoint where to export           |
+| metrics.exportIntervalSeconds        | integer          | 30      | How often to export the metrics to the open telemetry collector |
+| metrics.mode                         | MetricsEnabled   | Enabled | The toggle to enable/disable metrics.                           |
+| metrics.openTelemetryCollectorAddress| string           |         | The open telemetry collector endpoint where to export           |
+| metrics.prometheusPort               | integer          | 9600    | The prometheus port to expose the metrics                       |
+| metrics.stalenessTimeSeconds         | integer          | 600     | The time used to determine if a metric is stale and drop from the metrics cache |
+| metrics.updateIntervalSeconds        | integer          | 30      | How often to refresh the metrics                                |
+| selfcheck.intervalSeconds            | integer          | 30      | The self check interval                                         |
+| selfcheck.mode                       | SelfCheckMode    | Enabled | The toggle to enable/disable self check                         |
+| selfcheck.timeoutSeconds             | integer          | 15      | The timeout for self check                                      |
+| traces.cacheSizeMegabytes            | integer          | 16      | The cache size in megabytes                                     |
+| traces.exportIntervalSeconds         | integer          | 30      | How often to export the metrics to the open telemetry collector |
+| traces.mode                          | TracesMode       | Enabled | The toggle to enable/disable traces                             |
+| traces.openTelemetryCollectorAddress | string           |         | The open telemetry collector endpoint where to export           |
+| traces.selfTracing                   | SelfTracing      |         | The self tracing properties                                     |
+| traces.spanChannelCapacity           | integer          | 1000    | The span channel capacity                                       |
+
+Here's an example of a *Broker* custom resource with diagnostics settings enabled:
 
-Diagnostic settings allow you to enable metrics and tracing for MQ broker.
-
-- Metrics provide information about the resource utilization and throughput of MQ broker.
-- Tracing provides detailed information about the requests and responses handled by MQ broker.
-
-To enable these features, first [Configure the MQ diagnostic service settings](../configure-observability-monitoring/howto-configure-diagnostics.md).
-
-To override default diagnostic settings for MQ broker, update the `spec.diagnostics` section in  the Broker CR. You also need to specify the diagnostic service endpoint, which is the address of the service that collects and stores the metrics and traces. The default endpoint is `aio-mq-diagnostics-service :9700`.
-
-You can also adjust the log level of MQ broker to control the amount and detail of information that is logged. The log level can be set for different components of MQ broker. The default log level is `info`.
-
-If you don't specify settings, default values are used. The following table shows the properties of the broker diagnostic settings and all default values.
-
-| Name                                       | Required | Format           | Default| Description                                              |
-| ------------------------------------------ | -------- | ---------------- | -------|----------------------------------------------------------|
-| `diagnosticServiceEndpoint`                | true    | String            | N/A    |An endpoint to send metrics/ traces to                    |
-| `enableMetrics`                            | false   | Boolean           | true   |Enable or disable broker metrics                          |
-| `enableTracing`                            | false   | Boolean           | true   |Enable or disable tracing                                 |
-| `logLevel`                                 | false   | String            |`info`  |Log level. `trace`, `debug`, `info`, `warn`, or `error`   |
-| `enableSelfCheck`                          | false   | Boolean           | true   |Component that periodically probes the health of broker   |
-| `enableSelfTracing`                        | false   | Boolean           | true   |Automatically traces incoming messages at a frequency of 1 every `selfTraceFrequencySeconds`  |
-| `logFormat`                                | false   | String            |`text`  |Log format in `json` or `text`                         |
-| `metricUpdateFrequencySeconds`             | false   | Integer           | 30     |The frequency to send metrics to diagnostics service endpoint, in seconds |
-| `selfCheckFrequencySeconds`                | false   | Integer           | 30     |How often the probe sends test messages|
-| `selfCheckTimeoutSeconds`                  | false   | Integer           | 15     |Timeout interval for probe messages  |
-| `selfTraceFrequencySeconds`                | false   | Integer           | 30     |How often to automatically trace external messages if `enableSelfTracing` is true |  
-| `spanChannelCapacity`                      | false   | Integer           | 1000   |Maximum number of spans that selftest can store before sending to the diagnostics service     |  
-| `probeImage`                               | true    | String            |mcr.microsoft.com/azureiotoperations/diagnostics-probe:0.4.0-preview | Image used for self check |
-
-
-Here's an example of a Broker CR with metrics and tracing enabled and self-check disabled:
-
-```yml
+```yaml
 apiVersion: mq.iotoperations.azure.com/v1beta1
 kind: Broker
 metadata:
@@ -176,11 +178,31 @@ metadata:
 spec:
   mode: auto
   diagnostics:
-    diagnosticServiceEndpoint: diagnosticservices.mq.iotoperations:9700
-    enableMetrics: true  
-    enableTracing: true
-    enableSelfCheck: false
-    logLevel: debug,hyper=off,kube_client=off,tower=off,conhash=off,h2=off
+    logs:
+      exportIntervalSeconds: 220
+      exportLogLevel: nym
+      level: debug
+      openTelemetryCollectorAddress: acfqqatmodusdbzgomgcrtulvjy
+    metrics:
+      stalenessTimeSeconds: 463
+      mode: Enabled
+      exportIntervalSeconds: 246
+      openTelemetryCollectorAddress: vyasdzsemxfckcorfbfx
+      prometheusPort: 60607
+      updateIntervalSeconds: 15
+    selfCheck:
+      mode: Enabled
+      intervalSeconds: 106
+      timeoutSeconds: 70
+    traces:
+      cacheSizeMegabytes: 97
+      mode: Enabled
+      exportIntervalSeconds: 114
+      openTelemetryCollectorAddress: oyujxiemzlqlcsdamytj
+      selfTracing:
+        mode: Enabled
+        intervalSeconds: 179
+      spanChannelCapacity: 47152
 ```
 
 ## Configure encryption of internal traffic

articles/iot-operations/manage-mqtt-broker/howto-configure-availability-scale.md

@@ -185,8 +185,6 @@ items:
         href: configure-observability-monitoring/howto-add-cluster.md
       - name: Clean up observability resources
         href: configure-observability-monitoring/howto-clean-up-observability-resources.md
-      - name: Configure MQ diagnostic service settings
-        href: configure-observability-monitoring/howto-configure-diagnostics.md
     - name: Create edge apps
       items:
       - name: Edge apps overview

articles/iot-operations/toc.yml

@@ -6,7 +6,7 @@ ms.author: dobett
 ms.topic: troubleshooting-known-issue
 ms.custom:
   - ignite-2023
-ms.date: 05/03/2024
+ms.date: 07/11/2024
 ---
 
 # Known issues: Azure IoT Operations Preview
@@ -36,7 +36,7 @@ This article lists the known issues for Azure IoT Operations Preview.
 
 - You can't configure the size of a disk-backed buffer unless your chosen storage class supports it.
 
-- Even though the MQTT broker's [diagnostic service](../configure-observability-monitoring/howto-configure-diagnostics.md) produces telemetry on its own topic, you might still get messages from the self-test when you subscribe to `#` topic.
+- Even though the MQTT broker's [diagnostics](../manage-mqtt-broker/howto-configure-availability-scale.md#configure-mqtt-broker-diagnostic-settings) produces telemetry on its own topic, you might still get messages from the self-test when you subscribe to `#` topic.
 
 - Some clusters that have slow Kubernetes API calls may result in selftest ping failures: `Status {Failed}. Probe failed: Ping: 1/2` from running `az iot ops check` command.