Docs scrub

Author: jlian

articles/iot-operations/connect-to-cloud/overview-dataflow.md

@@ -6,7 +6,7 @@ ms.author: patricka
 ms.service: azure-iot-operations
 ms.subservice: azure-data-flows
 ms.topic: conceptual
-ms.date: 10/22/2024
+ms.date: 11/11/2024
 
 #CustomerIntent: As an operator, I want to understand how I can use dataflows to connect data sources.
 ---
@@ -62,7 +62,7 @@ By using dataflows, you can efficiently manage your data paths. You can ensure t
 
 Schema registry, a feature provided by Azure Device Registry, is a synchronized repository in the cloud and at the edge. The schema registry stores the definitions of messages coming from edge assets, and then exposes an API to access those schemas at the edge. Southbound connectors like the OPC UA connector can create message schemas and add them to the schema registry or customers can upload schemas to the operations experience web UI.
 
-Dataflows uses messages schemas at both the source and destination points. For sources, message schemas can work as filters to identify the specific messages that you want to capture for a dataflow. For destinations, message schemas help to transform the message into the format expected by the destination endpoint.
+Dataflows uses messages schemas to transform the message into the format expected by the destination endpoint.
 
 For more information, see [Understand message schemas](./concept-schema-registry.md).
 

articles/iot-operations/manage-mqtt-broker/howto-broker-mqtt-client-options.md

@@ -6,7 +6,7 @@ ms.author: patricka
 ms.topic: how-to
 ms.service: azure-iot-operations
 ms.subservice: azure-mqtt-broker
-ms.date: 11/04/2024
+ms.date: 11/11/2024
 
 #CustomerIntent: As an operator, I want to configure MQTT broker so that I can control MQTT client interactions.
 ---
@@ -62,22 +62,27 @@ The subscriber queue limit has two settings:
 
 - **Strategy**: The strategy to use when the queue is full. The two strategies are:
 
-  <!-- TODO: check for accuracy -->
-  - **None**: Messages aren't dropped [unless they expire](#message-expiry), and the queue can grow indefinitely. This is the default behavior.
+  - **None**: Messages aren't dropped [unless the session expires](#session-expiry), and the queue can grow indefinitely. This is the default behavior.
 
   - **DropOldest**: The oldest message in the queue is dropped.
 
 The limit only applies to the subscriber's outgoing queue, which holds messages that haven't been assigned packet IDs because the in-flight queue is full. This limit doesn't apply to the in-flight queue.
 
-Since the limit is applied per [backend chain](./howto-configure-availability-scale.md#backend-chain), the broker can't guarantee the total number of outgoing messages for a subscriber across the entire cluster. For example, setting the length to 10,000 doesn't mean the subscriber will receive at most 10,000 messages. Instead, it could receive up to `10,000 * number of chains * number of backend workers` messages.
+Since the limit is applied per [backend partition](./howto-configure-availability-scale.md#backend-chain), the broker can't guarantee the total number of outgoing messages for a subscriber across the entire cluster. For example, setting the length to 10,000 doesn't mean the subscriber will receive at most 10,000 messages. Instead, it could receive up to `10,000 * number of partitions * number of backend workers` messages.
 
 ### Slow subscribers
 
 A slow subscriber is one that can't keep up with the incoming message rate. This can occur if the subscriber processes messages slowly, is disconnected, or is offline. The subscriber queue limit helps prevent a slow subscriber from consuming too much memory.
 
 ### Message expiry
 
-The subscriber queue limit works with the `maxSessionExpirySeconds` setting to ensure messages aren't kept in the queue indefinitely. If a message stays in the queue longer than the maximum expiry time, it's dropped. This helps prevent slow subscribers from using too much memory.
+The `maxMessageExpirySeconds` setting controls how long a message can stay in the queue before it expires. If a message stays in the queue longer than the maximum expiry time, it is marked as expired. However, expired messages are only discarded when they reach the beginning of the queue. This passive expiry mechanism helps manage memory usage by ensuring that old messages are eventually removed.
+
+### Session expiry
+
+The `maxSessionExpirySeconds` setting works with the subscriber queue limit to ensure that messages aren't kept in the queue indefinitely. If a session expires, all messages in the queue for that session are dropped. This helps prevent slow subscribers from using too much memory by eventually clearing the entire queue.
+
+Both message expiry and session expiry are important for managing slow subscribers and ensuring efficient memory usage.
 
 ## Next steps
 

articles/iot-operations/manage-mqtt-broker/howto-configure-authentication.md

@@ -8,7 +8,7 @@ ms.subservice: azure-mqtt-broker
 ms.topic: how-to
 ms.custom:
   - ignite-2023
-ms.date: 11/02/2024
+ms.date: 11/11/2024
 
 #CustomerIntent: As an operator, I want to configure authentication so that I have secure MQTT broker communications.
 ---
@@ -330,7 +330,7 @@ To learn more about each of the authentication options, see the next sections fo
 
 For more information about enabling secure settings by configuring an Azure Key Vault and enabling workload identities, see [Enable secure settings in Azure IoT Operations deployment](../deploy-iot-ops/howto-enable-secure-settings.md).
 
-### X.509
+## X.509
 
 A trusted root CA certificate is required to validate the client certificate. Client certificates must be rooted in this CA for MQTT broker to authenticate them. Both EC and RSA keys are supported, but all certificates in the chain must use the same key algorithm. If you're importing your own CA certificates, ensure that the client certificate uses the same key algorithm as the CAs. To import a root certificate that can be used to validate client certificates, import the certificate PEM as *ConfigMap* under the key `client_ca.pem`. For example:
 
@@ -363,7 +363,7 @@ BinaryData
 
 Once the trusted client root CA certificate and the certificate-to-attribute mapping are imported, enable X.509 client authentication by adding it as one of the authentication methods as part of a *BrokerAuthentication* resource linked to a TLS-enabled listener. 
 
-#### Certificate attributes for authorization
+### Certificate attributes for authorization
 
 X.509 attributes can be specified in the *BrokerAuthentication* resource, and they're used to authorize clients based on their certificate properties. The attributes are defined in the `authorizationAttributes` field.
 
@@ -476,13 +476,13 @@ The matching for attributes always starts from the leaf client certificate and t
 
 Authorization rules can be applied to clients using X.509 certificates with these attributes. To learn more, see [Authorize clients that use X.509 authentication](./howto-configure-authorization.md).
 
-#### Connect mosquitto client to MQTT broker with X.509 client certificate
+### Connect mosquitto client to MQTT broker with X.509 client certificate
 
 A client like mosquitto needs three files to be able to connect to MQTT broker with TLS and X.509 client authentication. For example:
 
 ```bash
 mosquitto_pub -q 1 -t hello -d -V mqttv5 -m world -i thermostat \
--h "<IOT_MQ_EXTERNAL_IP>" \
+-h "<BROKER_HOST>" \
 --cert thermostat_cert.pem \
 --key thermostat_key.pem \
 --cafile chain.pem
@@ -496,7 +496,7 @@ In the example:
   - When mosquitto client connects to MQTT broker over TLS, it validates the server certificate. It searches for root certificates in the database to create a trusted chain to the server certificate. Because of this, the server root certificate needs to be copied into this file.
   - When the MQTT broker requests a client certificate from mosquitto client, it also requires a valid certificate chain to send to the server. The `--cert` parameter tells mosquitto which certificate to send, but it's not enough. MQTT broker can't verify this certificate alone because it also needs the intermediate certificate. Mosquitto uses the database file to build the necessary certificate chain. To support this, the `cafile` must contain both the intermediate and root certificates.
 
-#### Understand MQTT broker X.509 client authentication flow
+### Understand MQTT broker X.509 client authentication flow
 
 ![Diagram of the X.509 client authentication flow.](./media/howto-configure-authentication/x509-client-auth-flow.svg)
 
@@ -514,7 +514,7 @@ The following are the steps for client authentication flow:
 1. Authentication service returns decision to frontend broker.
 1. The frontend broker knows the client attributes and if it's allowed to connect. If so, then the MQTT connection is completed and the client can continue to send and receive MQTT packets determined by its authorization rules.
 
-### Kubernetes Service Account Tokens
+## Kubernetes Service Account Tokens
 
 Kubernetes Service Account Tokens (SATs) are JSON Web Tokens associated with Kubernetes Service Accounts. Clients present SATs to the MQTT broker to authenticate themselves.
 
@@ -528,19 +528,19 @@ Launched in Kubernetes 1.13, and becoming the default format in 1.21, bound toke
 
 The broker verifies tokens using the [Kubernetes Token Review API](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/). Enable Kubernetes `TokenRequestProjection` feature to specify `audiences` (default since 1.21). If this feature isn't enabled, SATs can't be used.
 
-#### Create a service account
+### Create a service account
 
 To create SATs, first create a service account. The following command creates a service account called `mqtt-client`.
 
 ```bash
 kubectl create serviceaccount mqtt-client -n azure-iot-operations
 ```
 
-#### Add attributes for authorization
+### Add attributes for authorization
 
 Clients authentication via SAT can optionally have their SATs annotated with attributes to be used with custom authorization policies. To learn more, see [Authorize clients that use Kubernetes Service Account Tokens](./howto-configure-authentication.md).
 
-#### Enable Service Account Token (SAT) authentication
+### Enable Service Account Token (SAT) authentication
 
 Modify the `authenticationMethods` setting in a *BrokerAuthentication* resource to specify `ServiceAccountToken` as a valid authentication method. The `audiences` specifies the list of valid audiences for tokens. Choose unique values that identify the MQTT broker service. You must specify at least one audience, and all SATs must match one of the specified audiences.
 
@@ -620,67 +620,33 @@ Apply your changes with `kubectl apply`. It might take a few minutes for the cha
 
 ---
 
-#### Test SAT authentication
+### Test SAT authentication
 
-SAT authentication must be used from a client in the same cluster as MQTT broker. Only enhanced authentication fields are permitted. Set authentication method to `K8S-SAT` and authentication data to the token.
+SAT authentication must be used from a client in the same cluster as MQTT broker. Only MQTTv5 enhanced authentication fields are permitted. Set authentication method to `K8S-SAT` and authentication data to the token.
 
-The following command specifies a pod that has the mosquitto client and mounts the SAT created in the previous steps into the pod. 
-
-```yaml
-apiVersion: v1
-kind: Pod
-metadata:
-  name: mqtt-client
-  namespace: azure-iot-operations
-spec:
-  serviceAccountName: mqtt-client
-  containers:
-  - image: efrecon/mqtt-client
-    name: mqtt-client
-    command: ["sleep", "infinity"]
-    volumeMounts:
-    - name: mqtt-client-token
-      mountPath: /var/run/secrets/tokens
-  volumes:
-  - name: mqtt-client-token
-    projected:
-      sources:
-      - serviceAccountToken:
-          path: mqtt-client-token
-          audience: my-audience
-          expirationSeconds: 86400
-```
-
-Here, the `serviceAccountName` field in the pod configuration must match the service account associated with the token being used. Also, The `serviceAccountToken.audience` field in the pod configuration must be one of the `audiences` configured in the *BrokerAuthentication* resource.
-
-Once the pod is created, start a shell in the pod:
+For example using mosquitto (some fields omitted for brevity):
 
 ```bash
-kubectl exec --stdin --tty mqtt-client -n azure-iot-operations -- sh
+mosquitto_pub ... -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data <TOKEN>
 ```
 
-Inside the pod's shell, run the following command to publish a message to the broker:
+Here, `<TOKEN>` is the service account token. To get the token, run:
 
 ```bash
-mosquitto_pub --host aio-broker --port 18883 --message "hello" --topic "world" --debug --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat)
+kubectl create token <SERVICE_ACCOUNT> -n azure-iot-operations --audience <AUDIENCE>
 ```
 
-The output should look similar to the following:
+Here, `<SERVICE_ACCOUNT>` is the name of the service account you create, and `<AUDIENCE>` is one of the audiences configured in the *BrokerAuthentication* resource.
 
-```Output
-Client (null) sending CONNECT
-Client (null) received CONNACK (0)
-Client (null) sending PUBLISH (d0, q0, r0, m1, 'world', ... (5 bytes))
-Client (null) sending DISCONNECT
-```
+For an example on how to configure a Kubernetes pod to use SAT authentication, see [Connect to the default listener inside the cluster](./howto-test-connection.md#connect-to-the-default-listener-inside-the-cluster).
 
-The mosquitto client uses the service account token mounted at `/var/run/secrets/tokens/broker-sat` to authenticate with the broker. The token is valid for 24 hours. The client also uses the default root CA cert mounted at `/var/run/certs/ca.crt` to verify the broker's TLS certificate chain.
+In the pod configuration, the `serviceAccountName` field in must match the service account associated with the token being used, and the `serviceAccountToken.audience` field must be one of the `audiences` configured in the *BrokerAuthentication* resource.
 
-#### Refresh service account tokens
+### Refresh service account tokens
 
 Service account tokens are valid for a limited time and configured with `expirationSeconds`. However, Kubernetes [automatically refreshes the token before it expires](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/). The token is refreshed in the background, and the client doesn't need to do anything other than to fetch it again.
 
-For example, if the client is a pod that uses the token mounted as a volume, like in the [test SAT authentication](#test-sat-authentication) example, then the latest token is available at the same path `/var/run/secrets/tokens/mqtt-client-token`. When making a new connection, the client can fetch the latest token and use it to authenticate. The client should also have a mechanism to handle MQTT unauthorized errors by fetching the latest token and retrying the connection.
+For example, if the client is a pod that uses the token mounted as a volume, like in the [test SAT authentication](#test-sat-authentication) example, then the latest token is available at the same path `/var/run/secrets/tokens/broker-sat`. When making a new connection, the client can fetch the latest token and use it to authenticate. The client should also have a mechanism to handle MQTT unauthorized errors by fetching the latest token and retrying the connection.
 
 ## Custom authentication
 

articles/iot-operations/manage-mqtt-broker/howto-encrypt-internal-traffic.md

@@ -20,7 +20,7 @@ Ensuring the security of internal communications within your infrastructure is i
 > [!IMPORTANT]
 > This setting requires modifying the Broker resource and can only be configured at initial deployment time using the Azure CLI or Azure Portal. A new deployment is required if Broker configuration changes are needed. To learn more, see [Customize default Broker](./overview-broker.md#customize-default-broker).
 
-The **encrypt internal traffic** feature is used to encrypt the internal traffic between the MQTT broker frontend and backend pods. To use this feature, cert-manager, which is installed by default when using the Azure IoT Operations, is required.
+The **encrypt internal traffic** feature is used to encrypt the internal traffic between the MQTT broker frontend and backend pods. It's enabled by default when you deploy Azure IoT Operations.
 
 The benefits include:
 
@@ -36,9 +36,10 @@ The benefits include:
 
 - **Secure data in the message buffer on disk**: All data in the [message buffer on disk](./howto-disk-backed-message-buffer.md) is encrypted.
 
-By default, encrypting internal traffic enabled. To mitigate security threats, we recommend that you keep the encryption enabled. However, disabling the encryption can improve the performance of the MQTT broker, which can be useful in IoT deployments with high throughput requirements. 
+To disable encryption, modify the `advanced.encryptInternalTraffic` setting in the Broker resource. This can only be done using the `--broker-config-file` flag during the deployment of Azure IoT Operations with the `az iot ops create` command.
 
-To disable the encryption, change the `advanced.encryptInternalTraffic` setting in the Broker resource. Currently, disabling encryption is only supported using the `--broker-config-file` flag when you deploy the Azure IoT Operations using the `az iot ops create` command. 
+> [!CAUTION]
+> Disabling encryption can enhance the performance of the MQTT broker, which may be beneficial in high-throughput IoT deployments. However, to safeguard against security threats like man-in-the-middle attacks, we strongly recommended to keep this setting enabled. Only disable encryption in controlled, non-production environments for testing purposes.
 
 ```json
 {

articles/iot-operations/manage-mqtt-broker/howto-test-connection.md

@@ -97,7 +97,7 @@ kubectl exec --stdin --tty mqtt-client --namespace azure-iot-operations -- sh
 
 Inside the pod's shell, run the following command to publish a message to the broker:
 
-```console
+```bash
 mosquitto_pub --host aio-broker --port 18883 --message "hello" --topic "world" --debug --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat)
 ```
 
@@ -122,7 +122,7 @@ The mosquitto client uses the service account token mounted at `/var/run/secrets
 
 To subscribe to the topic, run the following command:
 
-```console
+```bash
 mosquitto_sub --host aio-broker --port 18883 --topic "world" --debug --cafile /var/run/certs/ca.crt -D CONNECT authentication-method 'K8S-SAT' -D CONNECT authentication-data $(cat /var/run/secrets/tokens/broker-sat)
 ```
 

articles/iot-operations/troubleshoot/known-issues.md

@@ -94,5 +94,7 @@ kubectl delete pod aio-opc-opc.tcp-1-f95d76c54-w9v9c -n azure-iot-operations
 
 - X.509 authentication for custom Kafka endpoints isn't supported yet.
 
+- Deserializing and validating messages using a schema is not supported yet. Specifying `schemaRef` in the source configuration only allows the operations experience UX to display the list of data points, but the data points are not validated against the schema.
+
 <!-- TODO: double check -->
 - Creating a X.509 secret in the operations experience portal results in a secret with incorrectly encoded data. To work around this issue, create the [multi-line secrets through Azure Key Vault](/azure/key-vault/secrets/multiline-secrets), then select it from the list of secrets in the operations experience portal.