Merge pull request #284139 from spelluru/asaehub0808

Note about Event Hubs Geo-replication with Stream Analytics jobs

Author: ttorble

articles/event-hubs/process-data-azure-stream-analytics.md

@@ -94,5 +94,7 @@ Your Azure Stream Analytics job defaults to three streaming units (SUs). To adju
 
 :::image type="content" source="./media/process-data-azure-stream-analytics/scale.png" alt-text="Screenshots showing the Scale page for a Stream Analytics job." lightbox="./media/process-data-azure-stream-analytics/scale.png":::
 
+[!INCLUDE [geo-replication-stream-analytics-job](../stream-analytics/includes/geo-replication-stream-analytics-job.md)]
+
 ## Related content
 To learn more about Stream Analytics queries, see [Stream Analytics Query Language](/stream-analytics-query/built-in-functions-azure-stream-analytics)

articles/stream-analytics/capture-event-hub-data-delta-lake.md

@@ -65,6 +65,8 @@ Verify that the parquet files with Delta lake format are generated in the Azure
 
 :::image type="content" source="./media/capture-event-hub-data-delta-lake/verify-captured-data.png" alt-text="Screenshot showing the generated Parquet files in the Azure Data Lake Storage (ADLS) container." lightbox="./media/capture-event-hub-data-delta-lake/verify-captured-data.png" :::
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
 Now you know how to use the Stream Analytics no code editor to create a job that captures Event Hubs data to Azure Data Lake Storage Gen2 in Delta lake format. Next, you can learn more about Azure Stream Analytics and how to monitor the job that you created.

articles/stream-analytics/capture-event-hub-data-parquet.md

@@ -43,7 +43,7 @@ Use the following steps to configure a Stream Analytics job to capture data in A
         :::image type="content" source="./media/capture-event-hub-data-parquet/edit-fields.png" alt-text="Screenshot showing sample data under Data Preview." lightbox="./media/capture-event-hub-data-parquet/edit-fields.png" :::
 1. Select the **Azure Data Lake Storage Gen2** tile to edit the configuration. 
 1. On the **Azure Data Lake Storage Gen2** configuration page, follow these steps:     
-    1. Select the subscription, storage account name and container from the drop-down menu. 
+    1. Select the subscription, storage account name, and container from the drop-down menu. 
     1. Once the subscription is selected, the authentication method and storage account key should be automatically filled in.  
     1. Select **Parquet** for **Serialization** format. 
 
@@ -72,7 +72,7 @@ Use the following steps to configure a Stream Analytics job to capture data in A
 1. On the Event Hubs instance page for your event hub, select **Generate data**, select **Stocks data** for dataset, and then select **Send** to send some sample data to the event hub. 
 1. Verify that the Parquet files are generated in the Azure Data Lake Storage container. 
 
-    :::image type="content" source="./media/capture-event-hub-data-parquet/verify-captured-data.png" alt-text="Screenshot showing the generated Parquet files in the ADLS container." lightbox="./media/capture-event-hub-data-parquet/verify-captured-data.png" :::
+    :::image type="content" source="./media/capture-event-hub-data-parquet/verify-captured-data.png" alt-text="Screenshot showing the generated Parquet files in the Azure Data Lake Storage container." lightbox="./media/capture-event-hub-data-parquet/verify-captured-data.png" :::
 1. Select **Process data** on the left menu. Switch to the **Stream Analytics jobs** tab. Select **Open metrics** to monitor it. 
 
     :::image type="content" source="./media/capture-event-hub-data-parquet/open-metrics-link.png" alt-text="Screenshot showing Open Metrics link selected." lightbox="./media/capture-event-hub-data-parquet/open-metrics-link.png" :::
@@ -81,9 +81,13 @@ Use the following steps to configure a Stream Analytics job to capture data in A
 
     :::image type="content" source="./media/capture-event-hub-data-parquet/job-metrics.png" alt-text="Screenshot showing metrics of the Stream Analytics job." lightbox="./media/capture-event-hub-data-parquet/job-metrics.png" :::    
 
+
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
 Now you know how to use the Stream Analytics no code editor to create a job that captures Event Hubs data to Azure Data Lake Storage Gen2 in Parquet format. Next, you can learn more about Azure Stream Analytics and how to monitor the job that you created.
 
+
 * [Introduction to Azure Stream Analytics](stream-analytics-introduction.md)
 * [Monitor Stream Analytics job with Azure portal](stream-analytics-monitoring.md)

articles/stream-analytics/filter-ingest-data-lake-storage-gen2.md

@@ -71,6 +71,7 @@ This article describes how you can use the no code editor to easily create a Str
     {"RecordType":"MO","SystemIdentity":"d0","FileNum":"559","SwitchNum":"US","CallingNum":"456757102","CallingIMSI":"466920401237309","CalledNum":"345617823","CalledIMSI":"466923000886460","DateS":"20220524","TimeType":1,"CallPeriod":696,"ServiceType":"V","Transfer":1,"OutgoingTrunk":"419","MSRN":"886932429155","callrecTime":"2022-05-25T02:07:22Z","EventProcessedUtcTime":"2022-05-25T02:07:50.5478116Z","PartitionId":0,"EventEnqueuedUtcTime":"2022-05-25T02:07:21.9190000Z", "TimeS":null,"CallingCellID":null,"CalledCellID":null,"IncomingTrunk":null,"CalledNum2":null,"FCIFlag":null}
     ```
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
 
 ## Next steps
 

articles/stream-analytics/filter-ingest-synapse-sql.md

@@ -56,6 +56,8 @@ Use the following steps to develop a Stream Analytics job to filter and ingest r
 
     :::image type="content" source="./media/filter-ingest-synapse-sql/no-code-list-jobs.png" alt-text="Screenshot of the Stream Analytics jobs tab where you view the running jobs status." lightbox="./media/filter-ingest-synapse-sql/no-code-list-jobs.png" :::
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
 Learn more about Azure Stream Analytics and how to monitor the job you've created.

articles/stream-analytics/includes/geo-replication-stream-analytics-job.md

@@ -0,0 +1,19 @@
+---
+title: Stream Analytics job with Event Hubs Geo-replication
+description: This article has important notes about the behavior of Azure Stream Analytics jobs that process events from Azure Event Hubs. 
+author: spelluru
+ms.topic: include
+ms.date: 08/08/2024
+ms.author: spelluru
+---
+
+## Considerations when using the Event Hubs Geo-replication feature
+Azure Event Hubs recently launched the [Geo-Replication](../../event-hubs/geo-replication.md) feature in public preview. This feature is different from the [Geo Disaster Recovery](../../event-hubs/event-hubs-geo-dr.md) feature of Azure Event Hubs.
+
+When the failover type is **Forced** and replication consistency is **Asynchronous**, Stream Analytics job doesn't guarantee exactly once output to an Azure Event Hubs output. 
+
+Azure Stream Analytics, as **producer** with an event hub an output, might observe watermark delay on the job during failover duration and during throttling by Event Hubs in case replication lag  between primary and secondary reaches the maximum configured lag.
+
+Azure Stream Analytics, as **consumer** with Event Hubs as Input, might observe watermark delay on the job during failover duration and might skip data or find duplicate data after failover is complete. 
+
+Due to these caveats, we recommend that you restart the Stream Analytics job with appropriate start time right after Event Hubs failover is complete. Also, since Event Hubs Geo-replication feature is in public preview, we don't recommend using this pattern for production Stream Analytics jobs at this point. The current Stream Analytics behavior will improve before the Event Hubs Geo-replication feature is generally available and can be used in Stream Analytics production jobs.

articles/stream-analytics/no-code-build-power-bi-dashboard.md

@@ -79,6 +79,7 @@ Now, you have the Azure Stream Analytics job running and the data is continuousl
 5. Then, you can adjust its size and get the continuously updated dashboard as shown in the following example.
     :::image type="content" source="./media/no-code-build-power-bi-dashboard/pbi-dashboard-report.png" alt-text="Screenshot of the pbi dashboard report." lightbox="./media/no-code-build-power-bi-dashboard/pbi-dashboard-report.png" :::
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
 
 ## Next steps
 

articles/stream-analytics/no-code-enrich-event-hub-data.md

@@ -69,6 +69,8 @@ This article describes how you can use the no code editor to easily create a Str
 
     :::image type="content" source="./media/no-code-enrich-event-hub-data/no-code-list-jobs.png" alt-text="Screenshot of the Stream Analytics jobs tab where you view the running jobs status." lightbox="./media/no-code-enrich-event-hub-data/no-code-list-jobs.png" :::
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
 Learn more about Azure Stream Analytics and how to monitor the job you've created.

articles/stream-analytics/no-code-filter-ingest-data-explorer.md

@@ -22,7 +22,7 @@ This article describes how you can use the no code editor to easily create a Str
 1. In the [Azure portal](https://portal.azure.com), locate and select the Azure Event Hubs instance.
 1. Select **Features** > **Process Data** and then select **Start** on the **Filter and store data to Azure Data Explorer** card.
 
-    :::image type="content" source="./media/no-code-filter-ingest-data-explorer/event-hub-process-data-templates.png" alt-text="Screenshot showing the Filter and ingest to ADLS Gen2 card where you select Start." lightbox="./media/no-code-filter-ingest-data-explorer/event-hub-process-data-templates.png" :::
+    :::image type="content" source="./media/no-code-filter-ingest-data-explorer/event-hub-process-data-templates.png" alt-text="Screenshot showing the Filter and ingest to Azure Data Lake Storage Gen2 card where you select Start." lightbox="./media/no-code-filter-ingest-data-explorer/event-hub-process-data-templates.png" :::
 
 1. Enter a name for the Stream Analytics job, then select **Create**.  
 
@@ -71,6 +71,8 @@ This article describes how you can use the no code editor to easily create a Str
 
     :::image type="content" source="./media/no-code-filter-ingest-data-explorer/no-code-list-jobs.png" alt-text="Screenshot of the Stream Analytics jobs tab where you view the running jobs status." lightbox="./media/no-code-filter-ingest-data-explorer/no-code-list-jobs.png" :::
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
 Learn more about Azure Stream Analytics and how to monitor the job you've created.

articles/stream-analytics/no-code-materialize-cosmos-db.md

@@ -41,9 +41,11 @@ To start the job, you must specify:
     - **Output data error handling** allows you to specify the behavior you want when a job’s output to your destination fails due to data errors. By default, your job retries until the write operation succeeds. You can also choose to drop  output events.
 9.	After you select **Start**, the job starts running within two minutes. View the job under the **Process Data** section in the Stream Analytics jobs tab. You can explore job metrics and stop and restart it as needed.
 
+[!INCLUDE [geo-replication-stream-analytics-job](./includes/geo-replication-stream-analytics-job.md)]
+
 ## Next steps
 
-Now you know how to use the Stream Analytics no code editor to develop a job that reads from Event Hubs and calculates aggregates such as counts, averages and writes it to your Azure Cosmos DB resource.
+Now you know how to use the Stream Analytics no code editor to develop a job that reads from Event Hubs and calculates aggregates such as counts, averages, and writes it to your Azure Cosmos DB resource.
 
 * [Introduction to Azure Stream Analytics](stream-analytics-introduction.md)
 * [Monitor Stream Analytics job with Azure portal](./stream-analytics-monitoring.md)