Merge pull request #268178 from sethmanheim/esa-pp

Add Edge Storage Accelerator docs under Arc

Author: AnnaMHuff

articles/azure-arc/breadcrumb/toc.yml

@@ -0,0 +1,170 @@
+---
+title: Attach your application using the Azure IoT Operations data processor or Kubernetes native application (preview)
+description: Learn how to attach your app using the Azure IoT Operations data processor or Kubernetes native application in Edge Storage Accelerator.
+author: sethmanheim
+ms.author: sethm
+ms.topic: how-to
+ms.date: 04/08/2024
+zone_pivot_groups: attach-app
+---
+
+# Attach your application (preview)
+
+This article assumes you created a Persistent Volume (PV) and a Persistent Volume Claim (PVC). For information about creating a PV, see [Create a persistent volume](create-pv.md). For information about creating a PVC, see [Create a Persistent Volume Claim](create-pvc.md).
+
+::: zone pivot="attach-iot-op"
+## Configure the Azure IoT Operations data processor
+
+When you use Azure IoT Operations (AIO), the Data Processor is spawned without any mounts for Edge Storage Accelerator. You can perform the following tasks:
+
+- Add a mount for the Edge Storage Accelerator PVC you created previously.
+- Reconfigure all pipelines' output stage to output to the Edge Storage Accelerator mount you just created.  
+
+## Add Edge Storage Accelerator to your aio-dp-runner-worker-0 pods
+
+These pods are part of a **statefulSet**. You can't edit the statefulSet in place to add mount points. Instead, follow this procedure:
+
+1. Dump the statefulSet to yaml:
+
+    ```bash
+    kubectl get statefulset -o yaml -n azure-iot-operations aio-dp-runner-worker > stateful_worker.yaml
+    ```
+
+1. Edit the statefulSet to include the new mounts for ESA in volumeMounts and volumes:
+
+    ```yaml
+    volumeMounts: 
+    - mountPath: /etc/bluefin/config 
+      name: config-volume 
+      readOnly: true 
+    - mountPath: /var/lib/bluefin/registry 
+      name: nfs-volume 
+    - mountPath: /var/lib/bluefin/local 
+      name: runner-local
+      ### Add the next 2 lines ###
+    - mountPath: /mnt/esa 
+      name: esa4 
+    
+    volumes: 
+    - configMap: 
+        defaultMode: 420 
+        name: file-config 
+      name: config-volume 
+    - name: nfs-volume 
+    persistentVolumeClaim: 
+      claimName: nfs-provisioner
+      ### Add the next 3 lines ### 
+    - name: esa4 
+      persistentVolumeClaim: 
+        claimName: esa4
+    ```
+
+1. Delete the existing statefulSet:
+
+    ```bash
+    kubectl delete statefulset -n azure-iot-operations aio-dp-runner-worker
+    ```
+
+    This deletes all `aio-dp-runner-worker-n` pods. This is an outage-level event.  
+
+1. Create a new statefulSet of aio-dp-runner-worker(s) with the ESA mounts:
+
+    ```bash
+    kubectl apply -f stateful_worker.yaml -n azure-iot-operations
+    ```
+
+    When the `aio-dp-runner-worker-n` pods start, they include mounts to ESA. The PVC should convey this in the state.
+
+1. Once you reconfigure your Data Processor workers to have access to the ESA volumes, you must manually update the pipeline configuration to use a local path that corresponds to the mounted location of your ESA volume on the worker PODs.
+
+   In order to modify the pipeline, use `kubectl edit pipeline <name of your pipeline>`. In that pipeline, replace your output stage with the following YAML:
+
+   ```yaml
+   output:
+     batch:
+       path: .payload
+       time: 60s
+     description: An example file output stage
+     displayName: Sample File output
+     filePath: '{{{instanceId}}}/{{{pipelineId}}}/{{{partitionId}}}/{{{YYYY}}}/{{{MM}}}/{{{DD}}}/{{{HH}}}/{{{mm}}}/{{{fileNumber}}}'
+     format:
+       type: jsonStream
+     rootDirectory: /mnt/esa
+     type: output/file@v1
+   ```
+
+::: zone-end
+
+::: zone pivot="attach-kubernetes"
+## Configure a Kubernetes native application
+
+1. To configure a generic single pod (Kubernetes native application) against the Persistent Volume Claim (PVC), create a file named `configPod.yaml` with the following contents:
+
+   ```yaml
+   kind: Deployment
+   apiVersion: apps/v1
+   metadata:
+     name: example-static
+     labels:
+       app: example-static
+     ### Uncomment the next line and add your namespace only if you are not using the default namespace (if you are using azure-iot-operations) as specified from Line 6 of your pvc.yaml. If you are not using the default namespace, all future kubectl commands require "-n YOUR_NAMESPACE" to be added to the end of your command.
+     # namespace: YOUR_NAMESPACE
+   spec:
+     replicas: 1
+     selector:
+       matchLabels:
+         app: example-static
+     template:
+       metadata:
+         labels:
+           app: example-static
+       spec:
+         containers:
+           - image: mcr.microsoft.com/cbl-mariner/base/core:2.0
+             name: mariner
+             command:
+               - sleep
+               - infinity
+             volumeMounts:
+               ### This name must match the 'volumes.name' attribute in the next section. ###
+               - name: blob
+                 ### This mountPath is where the PVC is attached to the pod's filesystem. ###
+                 mountPath: "/mnt/blob"
+         volumes:
+           ### User-defined 'name' that's used to link the volumeMounts. This name must match 'volumeMounts.name' as specified in the previous section. ###
+           - name: blob
+             persistentVolumeClaim:
+               ### This claimName must refer to the PVC resource 'name' as defined in the PVC config. This name must match what your PVC resource was actually named. ###
+               claimName: YOUR_CLAIM_NAME_FROM_YOUR_PVC
+   ```
+
+   > [!NOTE]
+   > If you are using your own namespace, all future `kubectl` commands require `-n YOUR_NAMESPACE` to be appended to the command. For example, you must use `kubectl get pods -n YOUR_NAMESPACE` instead of the standard `kubectl get pods`.
+
+1. To apply this .yaml file, run the following command:
+
+    ```bash
+    kubectl apply -f "configPod.yaml"
+    ```
+
+1. Use `kubectl get pods` to find the name of your pod. Copy this name, as you need it for the next step.
+
+1. Run the following command and replace `POD_NAME_HERE` with your copied value from the previous step:
+
+   ```bash
+   kubectl exec -it POD_NAME_HERE -- bash
+   ```
+
+1. Change directories into the `/mnt/blob` mount path as specified from your `configPod.yaml`.
+
+1. As an example, to write a file, run `touch file.txt`.
+
+1. In the Azure portal, navigate to your storage account and find the container. This is the same container you specified in your `pv.yaml` file. When you select your container, you see `file.txt` populated within the container.
+
+::: zone-end
+
+## Next steps
+
+After you complete these steps, begin monitoring your deployment using Azure Monitor and Kubernetes Monitoring or third-party monitoring with Prometheus and Grafana:
+
+[Third-party monitoring](third-party-monitoring.md)

articles/azure-arc/edge-storage-accelerator/attach-app.md

@@ -0,0 +1,45 @@
+---
+title: Azure Monitor and Kubernetes monitoring (preview)
+description: Learn how to monitor your deployment using Azure Monitor and Kubernetes monitoring in Edge Storage Accelerator.
+author: sethmanheim
+ms.author: sethm
+ms.topic: how-to
+ms.date: 04/08/2024
+
+---
+
+# Azure Monitor and Kubernetes monitoring (preview)
+
+This article describes how to monitor your deployment using Azure Monitor and Kubernetes monitoring.
+
+## Azure Monitor
+
+[Azure Monitor](/azure/azure-monitor/essentials/monitor-azure-resource) is a full-stack monitoring service that you can use to monitor Azure resources for their availability, performance, and operation.
+
+## Azure Monitor metrics
+
+[Azure Monitor metrics](/azure/azure-monitor/essentials/data-platform-metrics) is a feature of Azure Monitor that collects data from monitored resources into a time-series database.
+
+These metrics can originate from a number of different sources, including native platform metrics, native custom metrics via [Azure Monitor agent Application Insights](/azure/azure-monitor/insights/insights-overview), and [Azure Managed Prometheus](/azure/azure-monitor/essentials/prometheus-metrics-overview).
+
+Prometheus metrics can be stored in an [Azure Monitor workspace](/azure/azure-monitor/essentials/azure-monitor-workspace-overview) for subsequent visualization via [Azure Managed Grafana](/azure/managed-grafana/overview).
+
+### Metrics configuration
+
+To configure the scraping of Prometheus metrics data into Azure Monitor, see the [Azure Monitor managed service for Prometheus scrape configuration](/azure/azure-monitor/containers/prometheus-metrics-scrape-configuration#enable-pod-annotation-based-scraping) article, which builds upon [this configmap](https://aka.ms/azureprometheus-addon-settings-configmap). Edge Storage Accelerator specifies the `prometheus.io/scrape:true` and `prometheus.io/port` values, and relies on the default of `prometheus.io/path: '/metrics'`. You must specify the Edge Storage Accelerator installation namespace under `pod-annotation-based-scraping` to properly scope your metrics' ingestion.
+
+Once the Prometheus configuration has been completed, follow the [Azure Managed Grafana instructions](/azure/managed-grafana/overview) to create an [Azure Managed Grafana instance](/azure/managed-grafana/quickstart-managed-grafana-portal).
+
+## Azure Monitor logs
+
+[Azure Monitor logs](/azure/azure-monitor/logs/data-platform-logs) is a feature of Azure Monitor that collects and organizes log and performance data from monitored resources, and can be used to [analyze this data in many ways](/azure/azure-monitor/logs/data-platform-logs#what-can-you-do-with-azure-monitor-logs).
+
+### Logs configuration
+
+If you want to access log data via Azure Monitor, you must enable [Azure Monitor Container Insights](/azure/azure-monitor/containers/container-insights-overview) on your Arc-enabled Kubernetes cluster, and then analyze the collected data with [a collection of views](/azure/azure-monitor/containers/container-insights-analyze) and [workbooks](/azure/azure-monitor/containers/container-insights-reports).
+
+Additionally, you can use [Azure Monitor Log Analytics](/azure/azure-monitor/logs/log-analytics-tutorial) to query collected log data.
+
+## Next steps
+
+[Edge Storage Accelerator overview](overview.md)

articles/azure-arc/edge-storage-accelerator/azure-monitor-kubernetes.md

@@ -0,0 +1,101 @@
+---
+title: Create a persistent volume (preview)
+description: Learn about creating persistent volumes in Edge Storage Accelerator.
+author: sethmanheim
+ms.author: sethm
+ms.topic: how-to
+ms.date: 04/08/2024
+
+---
+
+# Create a persistent volume (preview)
+
+This article describes how to create a persistent volume using storage key authentication.
+
+## Prerequisites
+
+This section describes the prerequisites for creating a persistent volume (PV).
+
+1. Create a storage account [following the instructions here](/azure/storage/common/storage-account-create?tabs=azure-portal).
+
+   When you create your storage account, create it under the same resource group and region/location as your Kubernetes cluster.
+
+1. Create a container in the storage account that you created in the previous step, [following the instructions here](/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container).
+
+## Storage key authentication configuration
+
+1. Create a file named **add-key.sh** with the following contents. No edits or changes are necessary:
+
+   ```bash
+   #!/usr/bin/env bash
+    
+   while getopts g:n:s: flag
+   do
+       case "${flag}" in
+           g) RESOURCE_GROUP=${OPTARG};;
+           s) STORAGE_ACCOUNT=${OPTARG};;
+           n) NAMESPACE=${OPTARG};;
+       esac
+   done
+    
+   SECRET=$(az storage account keys list -g $RESOURCE_GROUP -n $STORAGE_ACCOUNT --query [0].value --output tsv)
+    
+   kubectl create secret generic -n "${NAMESPACE}" "${STORAGE_ACCOUNT}"-secret --from-literal=azurestorageaccountkey="${SECRET}" --from-literal=azurestorageaccountname="${STORAGE_ACCOUNT}"
+   ```
+
+1. After you create the file, change the write permissions on the file and execute the shell script using the following commands. Running these commands creates a secret named `{YOUR_STORAGE_ACCOUNT}-secret`. This secret name is used for the `secretName` value when configuring your PV:
+
+   ```bash
+   chmod +x add-key.sh
+   ./add-key.sh -g "$YOUR_RESOURCE_GROUP_NAME" -s "$YOUR_STORAGE_ACCOUNT_NAME" -n "$YOUR_KUBERNETES_NAMESPACE"
+   ```
+
+## Create Persistent Volume (PV)
+
+You must create a Persistent Volume (PV) for the Edge Storage Accelerator to create a local instance and bind to a remote BLOB storage account.  
+
+Note the `metadata: name:` (`esa4` in this example), as you must specify it in the `spec: volumeName` of the PVC that binds to it. Use your storage account and container that you created as part of the [prerequisites](#prerequisites).
+
+1. Create a file named **pv.yaml**:
+
+   ```yaml
+   apiVersion: v1
+   kind: PersistentVolume
+   metadata:
+       ### Create a name here ###
+       name: CREATE_A_NAME_HERE
+       ### Use a namespace that matches your intended consuming pod, or "default" ###
+       namespace: INTENDED_CONSUMING_POD_OR_DEFAULT_HERE
+   spec:
+       capacity:
+           ### This storage capacity value is not enforced at this layer. ###
+           storage: 10Gi
+       accessModes:
+           - ReadWriteMany
+       persistentVolumeReclaimPolicy: Retain
+       storageClassName: esa
+       csi:
+           driver: edgecache.csi.azure.com
+           readOnly: false
+           ### Make sure this volumeid is unique in the cluster. You must specify it in the spec:volumeName of the PVC. ###
+           volumeHandle: YOUR_NAME_FROM_METADATA_NAME_IN_LINE_4_HERE
+           volumeAttributes:
+               protocol: edgecache
+               edgecache-storage-auth: AccountKey
+               ### Fill in the next two/three values with your information. ###
+               secretName: YOUR_SECRET_NAME_HERE ### From the previous step, this name is "{YOUR_STORAGE_ACCOUNT}-secret" ###
+               ### If you use a non-default namespace, uncomment the following line and add your namespace. ###
+               ### secretNamespace: YOUR_NAMESPACE_HERE
+               containerName: YOUR_CONTAINER_NAME_HERE
+   ```
+
+1. To apply this .yaml file, run:
+
+   ```bash
+   kubectl apply -f "pv.yaml"
+   ```
+
+## Next steps
+
+- [Create a persistent volume claim](create-pvc.md)
+- [Edge Storage Accelerator overview](overview.md)

articles/azure-arc/edge-storage-accelerator/create-pv.md

@@ -0,0 +1,59 @@
+---
+title: Create a Persistent Volume Claim (PVC) (preview)
+description: Learn how to create a Persistent Volume Claim (PVC) in Edge Storage Accelerator.
+author: sethmanheim
+ms.author: sethm
+ms.topic: how-to
+ms.date: 04/08/2024
+
+---
+
+# Create a Persistent Volume Claim (PVC) (preview)
+
+The PVC is a persistent volume claim against the persistent volume that you can use to mount a Kubernetes pod.
+
+This size does not affect the ceiling of blob storage used in the cloud to support this local cache. Note the name of this PVC, as you need it when you create your application pod.  
+
+## Create PVC
+
+1. Create a file named **pvc.yaml** with the following contents:
+
+   ```yaml
+   apiVersion: v1 
+   kind: PersistentVolumeClaim 
+   metadata:
+       ### Create a name for your PVC ###
+       name: CREATE_A_NAME_HERE
+       ### Use a namespace that matched your intended consuming pod, or "default" ###
+       namespace: INTENDED_CONSUMING_POD_OR_DEFAULT_HERE
+   spec: 
+       accessModes: 
+           - ReadWriteMany 
+       resources: 
+           requests: 
+               storage: 5Gi 
+       storageClassName: esa
+       volumeMode: Filesystem
+       ### This name references your PV name in your PV config ###
+       volumeName: INSERT_YOUR_PV_NAME
+   status: 
+       accessModes: 
+           - ReadWriteMany 
+       capacity: 
+           storage: 5Gi
+   ```
+
+   > [!NOTE]
+   > If you intend to use your PVC with the Azure IoT Operations Data Processor, use `azure-iot-operations` as the `namespace` on line 7.
+
+1. To apply this .yaml file, run:
+
+    ```bash
+    kubectl apply -f "pvc.yaml"
+    ```
+
+## Next steps
+
+After you create a Persistent Volume Claim (PVC), attach your app (Azure IoT Operations Data Processor or Kubernetes Native Application):
+
+[Attach your app](attach-app.md)