Merge branch 'release-aio-ga' into ga-1112sync

Author: kgremban

articles/iot-operations/create-edge-apps/concept-about-state-store-protocol.md

@@ -8,16 +8,17 @@ ms.topic: concept-article
 ms.date: 10/30/2024
 
 # CustomerIntent: As a developer, I want understand what the MQTT broker state store protocol is, so
-# that I can implement a client app to interact with the MQ state store.
+# that I can implement a client app to interact with the state store.
 ms.service: azure-iot-operations
 ---
 
-# MQTT broker state store protocol
+# State store protocol
 
-The MQ state store is a distributed storage system within the Azure IoT Operations cluster. The state store offers the same high availability guarantees as MQTT messages in MQTT broker. According to the MQTT5/RPC protocol guidelines, clients should use MQTT5 to interact with the MQ state store. This article provides protocol guidance for developers who need to implement their own MQTT broker state store clients. 
+The state store is a distributed storage system within the Azure IoT Operations cluster. The state store offers the same high availability guarantees as MQTT messages in MQTT broker. According to the MQTT5/RPC protocol guidelines, clients should use MQTT5 to interact with the state store. This article provides protocol guidance for developers who need to implement their own state store clients. 
 
-## State store protocol overview
-The MQ state store supports the following commands:
+## Overview
+
+The state store supports the following commands:
 
 - `SET` \<keyName\> \<keyValue\> \<setOptions\>
 - `GET` \<keyName\>
@@ -71,7 +72,7 @@ sequenceDiagram
 
 The commands `SET`, `GET`, and `DEL` behave as expected.
 
-The values that the `SET` command sets, and the `GET` command retrieves, are arbitrary binary data. The size of the values is only limited by the maximum MQTT payload size, and resource limitations of MQ and the client.
+The values that the `SET` command sets, and the `GET` command retrieves, are arbitrary binary data. The size of the values is only limited by the maximum MQTT payload size, and resource limitations of MQTT broker and the client.
 
 ### `SET` options
 
@@ -120,7 +121,7 @@ The following example output shows state store RESP3 payloads:
 ```
 
 > [!NOTE]
-> Note that `SET` requires additional MQTT5 properties, as explained in the section [Versioning and hybrid logical clocks](#versioning-and-hybrid-logical-clocks).
+> `SET` requires additional MQTT5 properties, as explained in the section [Versioning and hybrid logical clocks](#versioning-and-hybrid-logical-clocks).
 
 ### Response format
 
@@ -421,7 +422,7 @@ The topic is defined in the following example. The `clientId` is an upper-case h
 clients/statestore/v1/FA9AE35F-2F64-47CD-9BFF-08E2B32A0FE8/{clientId}/command/notify/{keyName}
 ```
 
-As an example, MQ publishes a `NOTIFY` message sent to `client-id1` with the modified key name `SOMEKEY` to the topic:
+As an example, MQTT broker publishes a `NOTIFY` message sent to `client-id1` with the modified key name `SOMEKEY` to the topic:
 
 ```console
 clients/statestore/v1/FA9AE35F-2F64-47CD-9BFF-08E2B32A0FE8/636C69656E742D696431/command/notify/534F4D454B4559`

articles/iot-operations/create-edge-apps/edge-apps-overview.md

@@ -21,7 +21,7 @@ The following sections explain the settings and features that contribute to a ro
 
 ## Quality of service (QoS)
 
-Both publishers and subscribers should use [QoS-1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901236) to guarantee message delivery at least once. The broker stores and retransmits messages until it receives an acknowledgment (ACK) from the recipient, ensuring no messages are lost during transmission.
+Both publishers and subscribers should use [QoS-1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901236) to guarantee message delivery at least once. The MQTT broker stores and retransmits messages until it receives an acknowledgment (ACK) from the recipient, ensuring no messages are lost during transmission.
 
 ## Session type and Clean-Session flag
 
@@ -55,13 +55,13 @@ MQTT subscriptions with QoS-1 ensure eventual consistency across identical appli
 
 [Shared subscriptions](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901250) enable load balancing across multiple instances of a highly available application. Instead of each subscriber receiving a copy of every message, the messages are distributed evenly among the subscribers. MQTT broker currently only supports a round robin algorithm to distribute messages allowing an application to scale out. A typical use case is to deploy multiple pods using Kubernetes ReplicaSet that all subscribe to MQTT broker using the same topic filter in shared subscription.
 
-## Use MQTT broker's built-in key-value store (distributed HashMap)
+## State store
 
-MQTT broker's built-in key-value store is a simple, replicated in-memory *HashMap* for managing application processing state. Unlike *etcd*, for example, MQTT broker prioritizes high-velocity throughput, horizontal scaling, and low latency through in-memory data structures, partitioning, and chain-replication. It allows applications to use the broker's distributed nature and fault tolerance while accessing a consistent state quickly across instances. To use the built-in key-value store provided by the distributed broker:
+State store is a replicated in-memory *HashMap* for managing application processing state. Unlike *etcd*, for example, state store prioritizes high-velocity throughput, horizontal scaling, and low latency through in-memory data structures, partitioning, and chain-replication. It allows applications to use the state stores distributed nature and fault tolerance while accessing a consistent state quickly across instances. To use the built-in key-value store provided by the distributed broker:
 
 * Implement ephemeral storage and retrieval operations using the broker's key-value store API, ensuring proper error handling and data consistency. Ephemeral state is a short-lived data storage used in stateful processing for fast access to intermediate results or metadata during real-time computations. In the context of HA application, an ephemeral state helps recover application states between crashes. It can be written to disk but remains temporary, as opposed to cold storage that's designed for long-term storage of infrequently accessed data.
 
-* Use the key-value store for sharing state, caching, configuration, or other essential data among multiple instances of the application, allowing them to keep a consistent view of the data.
+* Use the state store for sharing state, caching, configuration, or other essential data among multiple instances of the application, allowing them to keep a consistent view of the data.
 
 ## Use MQTT broker's built-in Dapr integration
 
@@ -75,25 +75,15 @@ For simpler use cases an application might utilize [Dapr](https://dapr.io) (Dist
 
 ## Checklist to develop a highly available application
 
-  - Choose an appropriate MQTT client library for your programming language. The client should support MQTT v5. Use a C or Rust based library if your application is sensitive to latency.
-  - Configure the client library to connect to MQTT broker with *clean-session* flag set to false and the desired QoS level (QoS-1).
-  - Decide a suitable value for session expiry, message expiry, and keep-alive intervals.
-  - Implement the message processing logic for the subscriber application, including sending an acknowledgment when the message has been successfully delivered or processed.
-  - For multithreaded applications, configure the *max-receive* parameter to enable parallel message processing.
-  - Utilize retained messages for keeping temporary application state.
-  - Utilize MQTT broker's built-in key-value store to manage ephemeral application state.
-  - Evaluate Dapr to develop your application if your use case is simple and doesn't require detailed control over the MQTT connection or message handling.
-  - Implement shared subscriptions to distribute messages evenly among multiple instances of the application, allowing for efficient scaling.
-
-## Example
-
-The following example implements contextualization and normalization of data with a highly available northbound connector
-
-A northbound application consists of input and output stages, and an optional processing stage. The input stage subscribes to a distributed MQTT broker to receive data, while the output stage ingests messages into a cloud data-lake. The processing stage executes contextualization and normalization logic on the received data.
-
-![Diagram of a highly available app architecture.](./media/edge-apps-overview/highly-available-app.png)
-
-To ensure high availability, the input stage connects to MQTT broker and sets the *clean-session* flag to false for persistent sessions, using QoS-1 for reliable message delivery, acknowledging messages post-processing by the output stage. Additionally, the application might use the built-in *HashMap* key-value store for temporary state management and the round robin algorithm to load-balance multiple instances using shared subscriptions.
+* Choose an appropriate MQTT client library for your programming language. The client should support MQTT v5. Use a C or Rust based library if your application is sensitive to latency.
+* Configure the client library to connect to MQTT broker with *clean-session* flag set to `false` and the desired QoS level (QoS-1).
+* Decide a suitable value for session expiry, message expiry, and keep-alive intervals.
+* Implement the message processing logic for the subscriber application, including sending an acknowledgment when the message has been successfully delivered or processed.
+* For multithreaded applications, configure the *max-receive* parameter to enable parallel message processing.
+* Utilize retained messages for keeping temporary application state.
+* Utilize the distributed state store to manage ephemeral application state.
+* Evaluate Dapr to develop your application if your use case is simple and doesn't require detailed control over the MQTT connection or message handling.
+* Implement shared subscriptions to distribute messages evenly among multiple instances of the application, allowing for efficient scaling.
 
 ## Related content
 

articles/iot-operations/create-edge-apps/howto-deploy-dapr.md

@@ -31,7 +31,7 @@ To install the Dapr runtime, use the following Helm command:
 ```bash
 helm repo add dapr https://dapr.github.io/helm-charts/
 helm repo update
-helm upgrade --install dapr dapr/dapr --version=1.13 --namespace dapr-system --create-namespace --wait
+helm upgrade --install dapr dapr/dapr --version=1.14 --namespace dapr-system --create-namespace --wait
 ```
 
 ## Register MQTT broker pluggable components
@@ -112,7 +112,7 @@ To create the yaml file, use the following component definitions:
         value: /var/run/secrets/tokens/mqtt-client-token    
     ```
 
-1. Apply the component yaml to your cluster by running the following command:
+1. Apply the Component to your cluster by running the following command:
 
     ```bash
     kubectl apply -f components.yaml
@@ -159,7 +159,7 @@ To configure authorization policies to MQTT broker, first you create a [BrokerAu
                   - "clients/{principal.clientId}/services/statestore/#"
     ```
 
-1. Apply the BrokerAuthorizaion definition to the cluster:
+1. Apply the BrokerAuthorization definition to the cluster:
 
     ```bash
     kubectl apply -f aio-dapr-authz.yaml

articles/iot-operations/create-edge-apps/media/edge-apps-overview/highly-available-app.png

@@ -0,0 +1,54 @@
+---
+title: Persisting data in the MQTT broker state store
+description: Understand how to develop application that persist data between sessions using the state store.
+author: PatAltimore
+ms.subservice: azure-mqtt-broker
+ms.author: patricka
+ms.topic: concept-article
+ms.custom:
+  - ignite-2023
+ms.date: 11/12/2024
+
+#CustomerIntent: As an developer, I want understand how to develop application that persist data between sessions using the state store.
+ms.service: azure-iot-operations
+---
+
+# Persisting data in the state store
+
+The state store is a distributed storage system, deployed as part of Azure IoT Operations. Using the state store, applications can get, set, and delete key-value pairs, without needing to install more services, such as Redis. The state store also provides versioning of the data, and also the primitives for building distributed locks, ideal for highly available applications.
+
+Like Redis, the state store uses in memory storage. Stopping or restarting the Kubernetes cluster causes the state store contents to be lost.
+
+The state store is implemented via MQTTv5. Its service is integrated directly into MQTT broker and is automatically started when the broker starts. The state store provides the same high availability as the MQTT broker.
+
+## Why use the state store?
+
+The state store allows an edge application to persist data on the edge. Typical uses of the state store include:
+
+* Creating stateless applications
+* Sharing state between applications
+* Developing highly available applications
+* Storing data to be used by dataflows
+
+## State store authorization
+
+The state store extends MQTT broker's authorization mechanism, allowing individual clients to have optional read and write access to specific keys. Read more on how to [configure MQTT broker authorization](../manage-mqtt-broker/howto-configure-authorization.md) for the state store.
+
+## Interacting with the state store
+
+A CLI tool is available which enables interaction with the state store from a shell running on any machine. 
+
+1. Generate an X.509 certificate chain for authenticating with MQTT broker
+1. Create a `BrokerAuthentication` using x.509 certificates
+1. Create a `BrokerListener` of type LoadBalancer which enables off-cluster access
+1. Open ports on your cluster to enable access to the MQTT broker.
+
+For instructions on setting up your cluster and using the tool, refer to the [state store CLI GitHub](https://github.com/Azure-Samples/explore-iot-operations/tree/main/tools/state-store-cli) page.
+
+> [!NOTE]
+> SDKs to interact with the state store are currently in active development, and will be available in the near future to enable edge applications to interact with the state store.
+
+## Related content
+
+* [Learn about the MQTT broker state store protocol](concept-about-state-store-protocol.md)
+* [Configure MQTT broker authorization](../manage-mqtt-broker/howto-configure-authorization.md)

articles/iot-operations/create-edge-apps/overview-state-store.md

@@ -31,7 +31,7 @@ Create an Arc-enabled K3s cluster that meets the system requirements.
 * [Configure the cluster](./howto-prepare-cluster.md) according to documentation.
 * If you expect intermittent connectivity for your cluster, ensure that you've allocated enough disk space to the cluster cache data and messages while the [cluster is offline](../overview-iot-operations.md#offline-support).
 * If possible, have a second cluster as a staging area for testing new changes before deploying to the primary production cluster.
-* [Turn off auto-upgrade for Azure Arc](/azure/azure-arc/kubernetes/agent-upgrade#toggle-automatic-upgrade-on-or-off-when-connecting-a-cluster-to-azure-arc) to have complete control over when new updates are applied to your cluster.
+* [Turn off auto-upgrade for Azure Arc](/azure/azure-arc/kubernetes/agent-upgrade#toggle-automatic-upgrade-on-or-off-when-connecting-a-cluster-to-azure-arc) to have complete control over when new updates are applied to your cluster. Instead, [manually upgrade agents](/azure/azure-arc/kubernetes/agent-upgrade#manually-upgrade-agents) as needed.
 * *For multi-node clusters*: [Configure clusters with Edge Volumes](./howto-prepare-cluster.md#configure-multi-node-clusters-for-azure-container-storage) to prepare for enabling fault tolerance during deployment.
 
 ### Security
@@ -64,7 +64,7 @@ In the Azure portal deployment wizard, the broker resource is set up in the **Co
 
   | Setting | Single node | Multi node |
   | ------- | ----------- | ---------- |
-  | **frontendReplicas** | 2 | 5 |
+  | **frontendReplicas** | 1 | 5 |
   | **frontendWorkers** | 4 | 8 |
   | **backendRedundancyFactor** | 2 | 2 |
   | **backendWorkers** | 1 | 4 |

articles/iot-operations/deploy-iot-ops/concept-production-guidelines.md

@@ -48,7 +48,7 @@ A cluster host:
 
 * Have an Azure Arc-enabled Kubernetes cluster with the custom location and workload identity features enabled. If you don't have one, follow the steps in [Prepare your Azure Arc-enabled Kubernetes cluster](./howto-prepare-cluster.md).
 
-  If you deployed Azure IoT Operations to your cluster previously, uninstall those resources before continuing. For more information, see [Update Azure IoT Operations](./howto-manage-update-uninstall.md#upgrade).
+  If you deployed Azure IoT Operations to your cluster previously, uninstall those resources before continuing. For more information, see [Update Azure IoT Operations](./howto-manage-update-uninstall.md#uninstall).
 
 * (Optional) Prepare your cluster for observability before deploying Azure IoT Operations: [Configure observability](../configure-observability-monitoring/howto-configure-observability.md).
 

articles/iot-operations/deploy-iot-ops/howto-deploy-iot-operations.md

@@ -186,39 +186,3 @@ az iot ops delete --name <INSTANCE_NAME> --resource-group <RESOURCE_GROUP> --inc
 ```
 
 ---
-
-## Upgrade
-
-Azure IoT Operations supports upgrading instances to new versions as they're released.
-
-You can't upgrade from a preview installation to a GA version.
-
-> [!NOTE]
-> There's a known issue with upgrading Azure IoT Operations if the MQTT broker only has one backend replica. Only upgrade Azure IoT Operations if the Broker has more than one backend replica.
-
-### [Azure portal](#tab/portal)
-
-1. In the [Azure portal](https://portal.azure.com), go to the resource group that contains your Azure IoT Operations instance, or search for and select **Azure IoT Operations**.
-
-1. Select the name of your Azure IoT Operations instance.
-
-1. On the **Overview** page of your instance, select **Upgrade**.
-
-1. The **Upgrade Azure IoT Operations** wizard prompts you to make sure you have the latest version for the Azure IoT Operations CLI extension. Copy and run the provided `az extension add` command.
-
-1. Update to the latest version of Azure IoT Operations instance. Copy and run the provided `az iot ops upgrade` command.
-
-1. Once the upgrade command completes successfully, you can exit the wizard and refresh your instance page.
-
-### [Azure CLI](#tab/cli)
-
-Use the `az iot ops upgrade` command to upgrade an Azure IoT Operations deployment. This command:
-
-* Upgrades Azure Arc extensions on your cluster.
-* Upgrades the Azure IoT Operations instance.
-
-```azurecli
-az iot ops upgrade --resource-group <RESOURCE_GROUP> --name <INSTANCE_NAME>
-```
-
----

articles/iot-operations/deploy-iot-ops/howto-manage-update-uninstall.md

@@ -161,9 +161,11 @@ Connect your cluster to Azure Arc so that it can be managed remotely.
 1. Use the [az connectedk8s connect](/cli/azure/connectedk8s#az-connectedk8s-connect) command to Arc-enable your Kubernetes cluster and manage it as part of your Azure resource group.
 
    ```azurecli
-   az connectedk8s connect --name <CLUSTER_NAME> -l <REGION> --resource-group <RESOURCE_GROUP> --subscription <SUBSCRIPTION_ID> --enable-oidc-issuer --enable-workload-identity
+   az connectedk8s connect --name <CLUSTER_NAME> -l <REGION> --resource-group <RESOURCE_GROUP> --subscription <SUBSCRIPTION_ID> --enable-oidc-issuer --enable-workload-identity --disable-auto-upgrade
    ```
 
+   To prevent unplanned updates to Azure Arc and the system Arc extensions that Azure IoT Operations uses as dependencies, this command disables auto-upgrade. Instead, [manually upgrade agents](/azure/azure-arc/kubernetes/agent-upgrade#manually-upgrade-agents) as needed.
+
 1. Get the cluster's issuer URL.
 
    ```azurecli
@@ -216,7 +218,9 @@ For instructions on running the script, see [Configure an AKS Edge Essentials cl
 
 ### [AKS on Azure Local](#tab/azure-local)
 
-For instructions on creating and Arc-enabling a cluster on Azure Local, see [Create Kubernetes clusters using Azure CLI](/azure/aks/hybrid/aks-create-clusters-cli).
+For instructions on creating and Arc-enabling an AKS cluster on Azure Local, see [Create Kubernetes clusters using Azure CLI](/azure/aks/hybrid/aks-create-clusters-cli).
+
+Then, once you have an Azure Arc-enabled Kubernetes cluster, you can [deploy Azure IoT Operations](howto-deploy-iot-operations.md).
 
 ---