Skip to content

Commit bad9cea

Browse files
committed
Freshness review
1 parent 47eaf36 commit bad9cea

File tree

1 file changed

+17
-17
lines changed

1 file changed

+17
-17
lines changed

articles/stream-analytics/sql-database-upsert.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@ title: Update or merge records in Azure SQL Database with Azure Functions
33
description: This article describes how to use Azure Functions to update or merge records from Azure Stream Analytics to Azure SQL Database
44
ms.service: stream-analytics
55
ms.topic: how-to
6-
ms.date: 12/03/2021
6+
ms.date: 02/27/2024
77
---
88

99
# Update or merge records in Azure SQL Database with Azure Functions
1010

11-
Currently, [Azure Stream Analytics](./index.yml) (ASA) only supports inserting (appending) rows to SQL outputs ([Azure SQL Databases](./sql-database-output.md), and [Azure Synapse Analytics](./azure-synapse-analytics-output.md)). This article discusses workarounds to enable UPDATE, UPSERT, or MERGE on SQL databases, with Azure Functions as the intermediary layer.
11+
Currently, [Azure Stream Analytics](./index.yml) (ASA) supports only inserting (appending) rows to SQL outputs ([Azure SQL Databases](./sql-database-output.md), and [Azure Synapse Analytics](./azure-synapse-analytics-output.md)). This article discusses workarounds to enable UPDATE, UPSERT, or MERGE on SQL databases, with Azure Functions as the intermediary layer.
1212

1313
Alternative options to Azure Functions are presented at the end.
1414

@@ -22,14 +22,14 @@ Writing data in a table can generally be done in the following manner:
2222
|Replace|[MERGE](/sql/t-sql/statements/merge-transact-sql) (UPSERT)|Unique key|
2323
|Accumulate|MERGE (UPSERT) with compound assignment [operator](/sql/t-sql/queries/update-transact-sql#arguments) (`+=`, `-=`...)|Unique key and accumulator|
2424

25-
To illustrate the differences, we can look at what happens when ingesting the following two records:
25+
To illustrate the differences, look at what happens when ingesting the following two records:
2626

2727
|Arrival_Time|Device_Id|Measure_Value|
2828
|-|-|-|
2929
|10:00|A|1|
3030
|10:05|A|20|
3131

32-
In **append** mode, we insert the two records. The equivalent T-SQL statement is:
32+
In the **append** mode, we insert two records. The equivalent T-SQL statement is:
3333

3434
```SQL
3535
INSERT INTO [target] VALUES (...);
@@ -42,7 +42,7 @@ Resulting in:
4242
|10:00|A|1|
4343
|10:05|A|20|
4444

45-
In **replace** mode, we get only the last value by key. Here we will use **Device_Id as the key.** The equivalent T-SQL statement is:
45+
In **replace** mode, we get only the last value by key. Here we use **Device_Id as the key.** The equivalent T-SQL statement is:
4646

4747
```SQL
4848
MERGE INTO [target] t
@@ -65,7 +65,7 @@ Resulting in:
6565
|-|-|-|
6666
|10:05|A|20|
6767

68-
Finally, in **accumulate** mode we sum `Value` with a compound assignment operator (`+=`). Here also we will use Device_Id as the key:
68+
Finally, in **accumulate** mode we sum `Value` with a compound assignment operator (`+=`). Here also we use Device_Id as the key:
6969

7070
```SQL
7171
MERGE INTO [target] t
@@ -90,15 +90,15 @@ Resulting in:
9090

9191
For **performance** considerations, the ASA SQL database output adapters currently only support append mode natively. These adapters use bulk insert to maximize throughput and limit back pressure.
9292

93-
This article shows how to use Azure Functions to implement Replace and Accumulate modes for ASA. By using a function as an intermediary layer, the potential write performance won't affect the streaming job. In this regard, using Azure Functions will work best with Azure SQL. With Synapse SQL, switching from bulk to row-by-row statements may create greater performance issues.
93+
This article shows how to use Azure Functions to implement Replace and Accumulate modes for ASA. When you use a function as an intermediary layer, the potential write performance won't affect the streaming job. In this regard, using Azure Functions works best with Azure SQL. With Synapse SQL, switching from bulk to row-by-row statements might create greater performance issues.
9494

9595
## Azure Functions Output
9696

97-
In our job, we'll replace the ASA SQL output by the [ASA Azure Functions output](./azure-functions-output.md). The UPDATE, UPSERT, or MERGE capabilities will be implemented in the function.
97+
In our job, we replace the ASA SQL output by the [ASA Azure Functions output](./azure-functions-output.md). The UPDATE, UPSERT, or MERGE capabilities are implemented in the function.
9898

9999
There are currently two options to access a SQL Database in a function. First is the [Azure SQL output binding](../azure-functions/functions-bindings-azure-sql.md). It's currently limited to C#, and only offers replace mode. Second is to compose a SQL query to be submitted via the appropriate [SQL driver](/sql/connect/sql-connection-libraries) ([Microsoft.Data.SqlClient](https://github.com/dotnet/SqlClient) for .NET).
100100

101-
For both samples below, we'll assume the following table schema. The binding option requires **a primary key** to be set on the target table. It's not necessary, but recommended, when using a SQL driver.
101+
For both the following samples, we assume the following table schema. The binding option requires **a primary key** to be set on the target table. It's not necessary, but recommended, when using a SQL driver.
102102

103103
```SQL
104104
CREATE TABLE [dbo].[device_updated](
@@ -130,7 +130,7 @@ This sample was built on:
130130

131131
To better understand the binding approach, it's recommended to follow [this tutorial](https://github.com/Azure/azure-functions-sql-extension#quick-start).
132132

133-
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information will be used:
133+
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information is used:
134134

135135
- Language: `C#`
136136
- Runtime: `.NET 6` (under function/runtime v4)
@@ -233,7 +233,7 @@ Update the `Device` class and mapping section to match your own schema:
233233
public DateTime Timestamp { get; set; }
234234
```
235235

236-
You can now test the wiring between the local function and the database by debugging (F5 in VS Code). The SQL database needs to be reachable from your machine. [SSMS](/sql/ssms/sql-server-management-studio-ssms) can be used to check connectivity. Then a tool like [Postman](https://www.postman.com/) can be used to issue POST requests to the local endpoint. A request with an empty body should return http 204. A request with an actual payload should be persisted in the destination table (in replace / update mode). Here's a sample payload corresponding to the schema used in this sample:
236+
You can now test the wiring between the local function and the database by debugging (F5 in Visual Studio Code). The SQL database needs to be reachable from your machine. [SSMS](/sql/ssms/sql-server-management-studio-ssms) can be used to check connectivity. Then a tool like [Postman](https://www.postman.com/) can be used to issue POST requests to the local endpoint. A request with an empty body should return http 204. A request with an actual payload should be persisted in the destination table (in replace / update mode). Here's a sample payload corresponding to the schema used in this sample:
237237
238238
```JSON
239239
[{"DeviceId":3,"Value":13.4,"Timestamp":"2021-11-30T03:22:12.991Z"},{"DeviceId":4,"Value":41.4,"Timestamp":"2021-11-30T03:22:12.991Z"}]
@@ -256,7 +256,7 @@ This sample was built on:
256256
- [.NET 6.0](/dotnet/core/whats-new/dotnet-6)
257257
- Microsoft.Data.SqlClient [4.0.0](https://www.nuget.org/packages/Microsoft.Data.SqlClient/)
258258
259-
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information will be used:
259+
First, create a default HttpTrigger function app by following this [tutorial](../azure-functions/create-first-function-vs-code-csharp.md?tabs=in-process). The following information is used:
260260

261261
- Language: `C#`
262262
- Runtime: `.NET 6` (under function/runtime v4)
@@ -371,11 +371,11 @@ The function can then be defined as an output in the ASA job, and used to replac
371371

372372
## Alternatives
373373

374-
Outside of Azure Functions, there are multiple ways to achieve the expected result. We'll mention the most likely solutions below.
374+
Outside of Azure Functions, there are multiple ways to achieve the expected result. This section provides some of them.
375375

376376
### Post-processing in the target SQL Database
377377

378-
A background task will operate once the data is inserted in the database via the standard ASA SQL outputs.
378+
A background task operates once the data is inserted in the database via the standard ASA SQL outputs.
379379

380380
For Azure SQL, `INSTEAD OF` [DML triggers](/sql/relational-databases/triggers/dml-triggers?view=azuresqldb-current&preserve-view=true) can be used to intercept the INSERT commands issued by ASA:
381381

@@ -402,13 +402,13 @@ END;
402402

403403
For Synapse SQL, ASA can insert into a [staging table](../synapse-analytics/sql/data-loading-best-practices.md#load-to-a-staging-table). A recurring task can then transform the data as needed into an intermediary table. Finally the [data is moved](../synapse-analytics/sql-data-warehouse/sql-data-warehouse-tables-partition.md#partition-switching) to the production table.
404404

405-
### Pre-processing in Azure Cosmos DB
405+
### Preprocessing in Azure Cosmos DB
406406

407407
Azure Cosmos DB [supports UPSERT natively](./stream-analytics-documentdb-output.md#upserts-from-stream-analytics). Here only append/replace is possible. Accumulations must be managed client-side in Azure Cosmos DB.
408408

409409
If the requirements match, an option is to replace the target SQL database by an Azure Cosmos DB instance. Doing so requires an important change in the overall solution architecture.
410410

411-
For Synapse SQL, Azure Cosmos DB can be used as an intermediary layer via [Azure Synapse Link for Azure Cosmos DB](../cosmos-db/synapse-link.md). Synapse Link can be used to create an [analytical store](../cosmos-db/analytical-store-introduction.md). This data store can then be queried directly in Synapse SQL.
411+
For Synapse SQL, Azure Cosmos DB can be used as an intermediary layer via [Azure Synapse Link for Azure Cosmos DB](../cosmos-db/synapse-link.md). Azure Synapse Link can be used to create an [analytical store](../cosmos-db/analytical-store-introduction.md). This data store can then be queried directly in Synapse SQL.
412412

413413
### Comparison of the alternatives
414414

@@ -422,7 +422,7 @@ Each approach offers different value proposition and capabilities:
422422
|Pre-Processing|||||
423423
||Azure Functions|Replace, Accumulate|+|- (row-by-row performance)|
424424
||Azure Cosmos DB replacement|Replace|N/A|N/A|
425-
||Azure Cosmos DB Synapse Link|Replace|N/A|+|
425+
||Azure Cosmos DB Azure Synapse Link|Replace|N/A|+|
426426

427427
## Get support
428428

0 commit comments

Comments
 (0)