You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/stream-analytics/sql-database-upsert.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,12 +3,12 @@ title: Update or merge records in Azure SQL Database with Azure Functions
3
3
description: This article describes how to use Azure Functions to update or merge records from Azure Stream Analytics to Azure SQL Database
4
4
ms.service: stream-analytics
5
5
ms.topic: how-to
6
-
ms.date: 12/03/2021
6
+
ms.date: 02/27/2024
7
7
---
8
8
9
9
# Update or merge records in Azure SQL Database with Azure Functions
10
10
11
-
Currently, [Azure Stream Analytics](./index.yml) (ASA) only supports inserting (appending) rows to SQL outputs ([Azure SQL Databases](./sql-database-output.md), and [Azure Synapse Analytics](./azure-synapse-analytics-output.md)). This article discusses workarounds to enable UPDATE, UPSERT, or MERGE on SQL databases, with Azure Functions as the intermediary layer.
11
+
Currently, [Azure Stream Analytics](./index.yml) (ASA) supports only inserting (appending) rows to SQL outputs ([Azure SQL Databases](./sql-database-output.md), and [Azure Synapse Analytics](./azure-synapse-analytics-output.md)). This article discusses workarounds to enable UPDATE, UPSERT, or MERGE on SQL databases, with Azure Functions as the intermediary layer.
12
12
13
13
Alternative options to Azure Functions are presented at the end.
14
14
@@ -22,14 +22,14 @@ Writing data in a table can generally be done in the following manner:
|Accumulate|MERGE (UPSERT) with compound assignment [operator](/sql/t-sql/queries/update-transact-sql#arguments) (`+=`, `-=`...)|Unique key and accumulator|
24
24
25
-
To illustrate the differences, we can look at what happens when ingesting the following two records:
25
+
To illustrate the differences, look at what happens when ingesting the following two records:
26
26
27
27
|Arrival_Time|Device_Id|Measure_Value|
28
28
|-|-|-|
29
29
|10:00|A|1|
30
30
|10:05|A|20|
31
31
32
-
In **append** mode, we insert the two records. The equivalent T-SQL statement is:
32
+
In the **append** mode, we insert two records. The equivalent T-SQL statement is:
33
33
34
34
```SQL
35
35
INSERT INTO [target] VALUES (...);
@@ -42,7 +42,7 @@ Resulting in:
42
42
|10:00|A|1|
43
43
|10:05|A|20|
44
44
45
-
In **replace** mode, we get only the last value by key. Here we will use **Device_Id as the key.** The equivalent T-SQL statement is:
45
+
In **replace** mode, we get only the last value by key. Here we use **Device_Id as the key.** The equivalent T-SQL statement is:
46
46
47
47
```SQL
48
48
MERGE INTO [target] t
@@ -65,7 +65,7 @@ Resulting in:
65
65
|-|-|-|
66
66
|10:05|A|20|
67
67
68
-
Finally, in **accumulate** mode we sum `Value` with a compound assignment operator (`+=`). Here also we will use Device_Id as the key:
68
+
Finally, in **accumulate** mode we sum `Value` with a compound assignment operator (`+=`). Here also we use Device_Id as the key:
69
69
70
70
```SQL
71
71
MERGE INTO [target] t
@@ -90,15 +90,15 @@ Resulting in:
90
90
91
91
For **performance** considerations, the ASA SQL database output adapters currently only support append mode natively. These adapters use bulk insert to maximize throughput and limit back pressure.
92
92
93
-
This article shows how to use Azure Functions to implement Replace and Accumulate modes for ASA. By using a function as an intermediary layer, the potential write performance won't affect the streaming job. In this regard, using Azure Functions will work best with Azure SQL. With Synapse SQL, switching from bulk to row-by-row statements may create greater performance issues.
93
+
This article shows how to use Azure Functions to implement Replace and Accumulate modes for ASA. When you use a function as an intermediary layer, the potential write performance won't affect the streaming job. In this regard, using Azure Functions works best with Azure SQL. With Synapse SQL, switching from bulk to row-by-row statements might create greater performance issues.
94
94
95
95
## Azure Functions Output
96
96
97
-
In our job, we'll replace the ASA SQL output by the [ASA Azure Functions output](./azure-functions-output.md). The UPDATE, UPSERT, or MERGE capabilities will be implemented in the function.
97
+
In our job, we replace the ASA SQL output by the [ASA Azure Functions output](./azure-functions-output.md). The UPDATE, UPSERT, or MERGE capabilities are implemented in the function.
98
98
99
99
There are currently two options to access a SQL Database in a function. First is the [Azure SQL output binding](../azure-functions/functions-bindings-azure-sql.md). It's currently limited to C#, and only offers replace mode. Second is to compose a SQL query to be submitted via the appropriate [SQL driver](/sql/connect/sql-connection-libraries) ([Microsoft.Data.SqlClient](https://github.com/dotnet/SqlClient) for .NET).
100
100
101
-
For both samples below, we'll assume the following table schema. The binding option requires **a primary key** to be set on the target table. It's not necessary, but recommended, when using a SQL driver.
101
+
For both the following samples, we assume the following table schema. The binding option requires **a primary key** to be set on the target table. It's not necessary, but recommended, when using a SQL driver.
102
102
103
103
```SQL
104
104
CREATE TABLE [dbo].[device_updated](
@@ -130,7 +130,7 @@ This sample was built on:
130
130
131
131
To better understand the binding approach, it's recommended to follow [this tutorial](https://github.com/Azure/azure-functions-sql-extension#quick-start).
132
132
133
-
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information will be used:
133
+
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information is used:
134
134
135
135
- Language: `C#`
136
136
- Runtime: `.NET 6` (under function/runtime v4)
@@ -233,7 +233,7 @@ Update the `Device` class and mapping section to match your own schema:
233
233
publicDateTimeTimestamp { get; set; }
234
234
```
235
235
236
-
You can now test the wiring between the local function and the database bydebugging (F5inVSCode). The SQL database needs to be reachable from your machine. [SSMS](/sql/ssms/sql-server-management-studio-ssms) can be used to check connectivity. Then a tool like [Postman](https://www.postman.com/) can be used to issue POST requests to the local endpoint. A request with an empty body should return http 204. A request with an actual payload should be persisted in the destination table (in replace / update mode). Here's a sample payload corresponding to the schema used in this sample:
236
+
You can now test the wiring between the local function and the database bydebugging (F5inVisualStudio Code). The SQL database needs to be reachable from your machine. [SSMS](/sql/ssms/sql-server-management-studio-ssms) can be used to check connectivity. Then a tool like [Postman](https://www.postman.com/) can be used to issue POST requests to the local endpoint. A request with an empty body should return http 204. A request with an actual payload should be persisted in the destination table (in replace / update mode). Here's a sample payload corresponding to the schema used in this sample:
First, createadefaultHttpTriggerfunctionappbyfollowing this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information will be used:
259
+
First, createadefaultHttpTriggerfunctionappbyfollowing this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information is used:
260
260
261
261
- Language: `C#`
262
262
- Runtime: `.NET 6` (underfunction/runtimev4)
@@ -371,11 +371,11 @@ The function can then be defined as an output in the ASA job, and used to replac
371
371
372
372
## Alternatives
373
373
374
-
Outside of Azure Functions, there are multiple ways to achieve the expected result. We'll mention the most likely solutions below.
374
+
Outside of Azure Functions, there are multiple ways to achieve the expected result. This section provides some of them.
375
375
376
376
### Post-processing in the target SQL Database
377
377
378
-
A background task will operate once the data is inserted in the database via the standard ASA SQL outputs.
378
+
A background task operates once the data is inserted in the database via the standard ASA SQL outputs.
379
379
380
380
For Azure SQL, `INSTEAD OF`[DML triggers](/sql/relational-databases/triggers/dml-triggers?view=azuresqldb-current&preserve-view=true) can be used to intercept the INSERT commands issued by ASA:
381
381
@@ -402,13 +402,13 @@ END;
402
402
403
403
For Synapse SQL, ASA can insert into a [staging table](../synapse-analytics/sql/data-loading-best-practices.md#load-to-a-staging-table). A recurring task can then transform the data as needed into an intermediary table. Finally the [data is moved](../synapse-analytics/sql-data-warehouse/sql-data-warehouse-tables-partition.md#partition-switching) to the production table.
404
404
405
-
### Pre-processing in Azure Cosmos DB
405
+
### Preprocessing in Azure Cosmos DB
406
406
407
407
Azure Cosmos DB [supports UPSERT natively](./stream-analytics-documentdb-output.md#upserts-from-stream-analytics). Here only append/replace is possible. Accumulations must be managed client-side in Azure Cosmos DB.
408
408
409
409
If the requirements match, an option is to replace the target SQL database by an Azure Cosmos DB instance. Doing so requires an important change in the overall solution architecture.
410
410
411
-
For Synapse SQL, Azure Cosmos DB can be used as an intermediary layer via [Azure Synapse Link for Azure Cosmos DB](../cosmos-db/synapse-link.md). Synapse Link can be used to create an [analytical store](../cosmos-db/analytical-store-introduction.md). This data store can then be queried directly in Synapse SQL.
411
+
For Synapse SQL, Azure Cosmos DB can be used as an intermediary layer via [Azure Synapse Link for Azure Cosmos DB](../cosmos-db/synapse-link.md). Azure Synapse Link can be used to create an [analytical store](../cosmos-db/analytical-store-introduction.md). This data store can then be queried directly in Synapse SQL.
412
412
413
413
### Comparison of the alternatives
414
414
@@ -422,7 +422,7 @@ Each approach offers different value proposition and capabilities:
0 commit comments