Skip to content

Commit c768163

Browse files
committed
Merge branch 'main' of https://github.com/MicrosoftDocs/azure-docs-pr into appsfi2
2 parents 957103d + 00fbd23 commit c768163

32 files changed

+349
-301
lines changed

articles/data-factory/connector-sharepoint-online-list.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -299,7 +299,7 @@ You can copy file from SharePoint Online by using **Web activity** to authentica
299299
- Copy activity source:
300300
- **Request method**: GET
301301
- **Additional header**: use the following expression`@{concat('Authorization: Bearer ', activity('<Web-activity-name>').output.access_token)}`, which uses the Bearer token generated by the upstream Web activity as authorization header. Replace the Web activity name.
302-
- Configure the copy activity sink as usual.
302+
- Configure the copy activity sink for any supported sink destination.
303303

304304
> [!NOTE]
305305
> Even if a Microsoft Entra application has `FullControl` permissions on SharePoint Online, you can't copy files from document libraries with IRM enabled.

articles/data-factory/connector-snowflake.md

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ author: jianleishen
77
ms.subservice: data-movement
88
ms.topic: conceptual
99
ms.custom: synapse
10-
ms.date: 07/25/2024
10+
ms.date: 09/23/2024
11+
ai-usage: ai-assisted
1112
---
1213

1314
# Copy and transform data in Snowflake using Azure Data Factory or Azure Synapse Analytics
@@ -403,6 +404,27 @@ To use this feature, create an [Azure Blob storage linked service](connector-azu
403404
]
404405
```
405406

407+
When performing a staged copy from Snowflake, it is crucial to set the Sink Copy Behavior to **Merge Files**. This setting ensures that all partitioned files are correctly handled and merged, preventing the issue where only the last partitioned file is copied.
408+
409+
**Example Configuration**
410+
411+
```json
412+
{
413+
"type": "Copy",
414+
"source": {
415+
"type": "SnowflakeSource",
416+
"query": "SELECT * FROM my_table"
417+
},
418+
"sink": {
419+
"type": "AzureBlobStorage",
420+
"copyBehavior": "MergeFiles"
421+
}
422+
}
423+
```
424+
425+
>[!NOTE]
426+
>Failing to set the Sink Copy Behavior to **Merge Files** may result in only the last partitioned file being copied.
427+
406428
### Snowflake as sink
407429

408430
Snowflake connector utilizes Snowflake’s [COPY into [table]](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html) command to achieve the best performance. It supports writing data to Snowflake on Azure.

articles/data-factory/data-flow-window.md

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ ms.author: makromer
77
ms.subservice: data-flows
88
ms.topic: conceptual
99
ms.custom: synapse
10-
ms.date: 05/15/2024
10+
ms.date: 09/23/2024
1111
---
1212

1313
# Window transformation in mapping data flow
@@ -16,36 +16,34 @@ ms.date: 05/15/2024
1616

1717
[!INCLUDE[data-flow-preamble](includes/data-flow-preamble.md)]
1818

19-
The Window transformation is where you will define window-based aggregations of columns in your data streams. In the Expression Builder, you can define different types of aggregations that are based on data or time windows (SQL OVER clause) such as LEAD, LAG, NTILE, CUMEDIST, RANK, etc.). A new field will be generated in your output that includes these aggregations. You can also include optional group-by fields.
19+
The Window transformation is where you define window-based aggregations of columns in your data streams. In the Expression Builder, you can define different types of aggregations that are based on data or time windows (SQL OVER clause) such as LEAD, LAG, NTILE, CUMEDIST, and RANK. A new field is generated in your output that includes these aggregations. You can also include optional group-by fields.
2020

2121
:::image type="content" source="media/data-flow/windows1.png" alt-text="Screenshot shows Windowing selected from the menu.":::
2222

2323
> [!VIDEO https://www.microsoft.com/videoplayer/embed/RE4IAVu]
2424
2525
## Over
26-
Set the partitioning of column data for your window transformation. The SQL equivalent is the ```Partition By``` in the Over clause in SQL. If you wish to create a calculation or create an expression to use for the partitioning, you can do that by hovering over the column name and select "computed column".
26+
Set the partitioning of column data for your window transformation. The SQL equivalent is the ```Partition By``` in the Over clause in SQL. If you wish to create a calculation or create an expression to use for the partitioning, you can do that by hovering over the column name and selecting **Computed column**.
2727

2828
:::image type="content" source="media/data-flow/windows-4.png" alt-text="Screenshot shows Windowing Settings with the Over tab selected.":::
2929

3030
## Sort
31-
Another part of the Over clause is setting the ```Order By```. This will set the data sort ordering. You can also create an expression for a calculate value in this column field for sorting.
31+
Another part of the Over clause is setting the ```Order By```. This clause sets the data sort ordering. You can also create an expression for a calculate value in this column field for sorting.
3232

3333
:::image type="content" source="media/data-flow/windows-5.png" alt-text="Screenshot shows Windowing Settings with the Sort tab selected.":::
3434

3535
## Range By
36-
Next, set the window frame as Unbounded or Bounded. To set an unbounded window frame, set the slider to Unbounded on both ends. If you choose a setting between Unbounded and Current Row, then you must set the Offset start and end values. Both values will be positive integers. You can use either relative numbers or values from your data.
36+
Next, set the window frame as Unbounded or Bounded. To set an unbounded window frame, set the slider to Unbounded on both ends. If you choose a setting between Unbounded and Current Row, then you must set the Offset start and end values. Both values are positive integers. You can use either relative numbers or values from your data.
3737

38-
The window slider has two values to set: the values before the current row and the values after the current row. The Start and End offset matches the two selectors on the slider.
38+
The window slider has two values to set: the values before the current row and the values after the current row. The offset between start and end matches the two selectors on the slider.
3939

4040
:::image type="content" source="media/data-flow/windows6.png" alt-text="Screenshot shows Windowing Settings with the Range by tab selected.":::
4141

4242
## Window columns
4343
Lastly, use the Expression Builder to define the aggregations you wish to use with the data windows such as RANK, COUNT, MIN, MAX, DENSE RANK, LEAD, LAG, etc.
4444

45-
:::image type="content" source="media/data-flow/windows7.png" alt-text="Screenshot shows the result of the windowing action.":::
46-
4745
The full list of aggregation and analytical functions available for you to use in the Data Flow Expression Language via the Expression Builder are listed in [Data transformation expressions in mapping data flow](data-transformation-functions.md).
4846

4947
## Related content
5048

51-
If you are looking for a simple group-by aggregation, use the [Aggregate transformation](data-flow-aggregate.md)
49+
If you're looking for a simple group-by aggregation, use the [Aggregate transformation](data-flow-aggregate.md)
Binary file not shown.

articles/data-factory/quickstart-create-data-factory.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,14 +38,14 @@ A quick creation experience provided in the Azure Data Factory Studio to enable
3838

3939
:::image type="content" source="media/quickstart-create-data-factory/create-with-azure-data-factory-studio.png" alt-text="Shows a screenshot of the Azure Data Factory Studio page to create a new data factory.":::
4040

41-
1. After creation, you can directly enter the homepage of the Azure Data Factory Studio.
41+
1. After creation, you can directly enter the homepage of the [Azure Data Factory Studio](https://adf.azure.com).
4242

4343
:::image type="content" source="media/quickstart-create-data-factory/azure-data-factory-studio-home-page.png" alt-text="Shows a screenshot of the Azure Data Factory Studio home page.":::
4444

4545
### Advanced creation in the Azure portal
4646

4747
1. Launch **Microsoft Edge** or **Google Chrome** web browser. Currently, Data Factory UI is supported only in Microsoft Edge and Google Chrome web browsers.
48-
1. Go to the [Azure portal data factories page](https://portal.azure.com).
48+
1. Go to the [Azure portal data factories page](https://portal.azure.com/#browse/Microsoft.DataFactory%2FdataFactories).
4949
1. After landing on the data factories page of the Azure portal, click **Create**.
5050

5151
:::image type="content" source="media/quickstart-create-data-factory/data-factory-create-from-portal.png" alt-text="Shows a screenshot of the Azure portal data factories Create button.":::
Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
title: "Create a private network connector: Azure Modeling and Simulation Workbench"
3+
description: Learn how to deploy a connector on a private virtual network.
4+
author: yousefi-msft
5+
ms.author: yousefi
6+
ms.service: modeling-simulation-workbench
7+
ms.topic: how-to
8+
ms.date: 09/21/2024
9+
10+
#CustomerIntent: As a Workench Owner for Azure Modeling and Simulation Workbench, I want to deploy a connector onto a private virtual network.
11+
---
12+
13+
# Set up a private networking connector
14+
15+
In Azure Modeling and Simulation Workbench, you can deploy a [connector](./concept-connector.md) to a virtual network, rather than to public facing IP addresses. Deploying to a private address virtual network allows you to enable access to your workbench through a virtual private network (VPN) gateway or from other Azure resources without exposing it to the internet.
16+
17+
## Prerequisites
18+
19+
[!INCLUDE [prerequisite-account-sub](includes/prerequisite-account-sub.md)]
20+
21+
[!INCLUDE [prerequisite-mswb-chamber](includes/prerequisite-chamber.md)]
22+
23+
## Create or designate a virtual network
24+
25+
Modeling and Simulation Workbench requires a virtual network with a subnet name 'default.' If you don't have a virtual network already created, [create one before continuing](/azure/virtual-network/quick-create-portal).
26+
27+
## Assign roles
28+
29+
Before you create a [connector](./concept-connector.md) for private IP networking via VPN or ExpressRoute, the Workbench needs a role assignment to allow it to deploy resources into your resource group. Modeling and Simulation Workbench requires the **Network Contributor** role for the resource group in which you're hosting your virtual network.
30+
31+
| Setting | Value |
32+
|:---------------------|:--------------------------------------------|
33+
| **Role** | **Network Contributor** |
34+
| **Assign access to** | **Resource group** |
35+
| **Members** | **Azure Modeling and Simulation Workbench** |
36+
37+
[!INCLUDE [azure-hpc-workbench-alert](includes/azure-hpc-workbench-alert.md)]
38+
39+
## Create the private network connector
40+
41+
Each chamber can have only one connector. If you have a public IP connector or other type already associated with the target chamber, you must first [delete the connector](#cleaning-up-resources). In the chamber where you want to create a private network connector:
42+
43+
1. Select the **Connector** option in the **Settings** at the left.
44+
:::image type="content" source="media/howtoguide-private-network/chamber-select-connector.png" alt-text="Screenshot of chamber overview with Connector option outlined in red rectangle.":::
45+
1. In the **Connector** list screen, select **Create** from the action bar along the top.
46+
:::image type="content" source="media/howtoguide-private-network/connector-create.png" alt-text="Screenshot of Connector overview page with Create button highlighted in red.":::
47+
1. On the **Create chamber connector** page, on **Chamber Connector** tab, enter a **Name** for the connector.
48+
1. Choose whether the copy/paste permission should be enabled for the chamber. You can learn about security boundary implications copy and paste in the [Enable copy/paste in Azure Modeling and Simulation Workbench](how-to-guide-enable-copy-paste.md) article.
49+
1. Under **Network Access**, select **VPN** in **Connect on-premises network**.
50+
1. In **Virtual Network**, select the virtual network you designated or created in [Create or designate a virtual network](#create-or-designate-a-virtual-network) earlier.
51+
1. Select the *default* **Subnet**.
52+
:::image type="content" source="media/howtoguide-private-network/create-private-network.png" alt-text="Screenshot of chamber connector with VPN and Review+Create button highlighted in red.":::
53+
1. Select **Review + create**.
54+
1. If validation passes, select **Create**. Private networking connectors take approximately 30 minutes to deploy.
55+
56+
## Deployed resources
57+
58+
When the Modeling and Simulation Workbench creates a private connector, it deploys the following resources in the same resource group and location as the workbench.
59+
60+
### Network interfaces and private endpoints
61+
62+
Six [network interfaces](/azure/virtual-network/virtual-network-network-interface) (NIC) and corresponding [private endpoints](/azure/private-link/private-endpoint-overview) are created. The NICs are all joined to the private virtual network and subnet specified during setup and given an address on the subnet. The private endpoint connects the NIC to Modeling and Simulation resources hosted in the Microsoft managed environment. The resulting connection becomes part of an [Azure Private Link](/azure/private-link/private-link-overview) service.
63+
64+
* Two connections are created for connection nodes. As users and virtual machines (VM) are added to a chamber, more connection nodes are created.
65+
* One connection for data in pipeline.
66+
* One connection for data out pipeline.
67+
* One connection for load balancer.
68+
* One connection for user authentication services.
69+
70+
### DNS zones
71+
72+
Modeling and Simulation Workbench creates three private domain name service (DNS) zones for a private network deployment. Each zone corresponds to one of the workbench services for file uploading, file downloading, and desktop connections. No DNS server is created. Administrators must join the zones to their own services.
73+
74+
| Zone | Resolves for |
75+
|:----------------------------------|:--------------------------------------|
76+
| mswb.azure.com | Connector desktop dashboard and nodes |
77+
| privateLink.blob.core.windows.net | Data in pipeline endpoint |
78+
| privateLink.file.core.windows.net | Data out pipeline endpoint |
79+
80+
## Starting, stopping, or restarting a connector
81+
82+
Connectors are controllable resources that can be stopped, started, restarted as needed. Instructions on how to are included in [Start, stop, and restart chambers, connectors, and VMs](how-to-guide-start-stop-restart.md). Stopping or restarting the connector interrupts desktop services for all users of the chamber. Stopping the connector is required to [idle a chamber](how-to-guide-chamber-idle.md) to reduce consumption costs.
83+
84+
## Cleaning up resources
85+
86+
If you wish to delete the workbench or change the connector type, you must first delete the connector. Child resources must be deleted first.
87+
88+
1. Delete all private endpoints and network interfaces.
89+
1. Delete virtual network links within each DNS zone.
90+
1. Delete each DNS zone.
91+
92+
Once those resources are deleted, the connector can be deleted. The virtual network doesn't need to be deleted since it has no dependencies.
93+
94+
## Related content
95+
96+
* [Manage chamber idle mode](how-to-guide-chamber-idle.md)
97+
* [Export data from Azure Modeling and Simulation Workbench](how-to-guide-download-data.md)
98+
* [Import data into Azure Modeling and Simulation Workbench](how-to-guide-upload-data.md)
52.4 KB
Loading
35.9 KB
Loading
45 KB
Loading

articles/modeling-simulation-workbench/toc.yml

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ items:
3737
href: ./how-to-guide-manage-users.md
3838
- name: Set up networking
3939
href: ./how-to-guide-set-up-networking.md
40+
- name: Create a private network connector for VPN
41+
href: ./how-to-guide-private-network.md
4042
- name: Manage license service
4143
href: ./how-to-guide-licenses.md
4244
- name: Import data
@@ -53,18 +55,20 @@ items:
5355
href: ./how-to-guide-start-stop-restart.md
5456
- name: Enable copy/paste
5557
href: ./how-to-guide-enable-copy-paste.md
56-
- name: Place a chamber into Idle mode
58+
- name: Place a chamber into idle mode
5759
href: ./how-to-guide-chamber-idle.md
58-
- name: Add redirect URIs
59-
href: ./how-to-guide-add-redirect-uris.md
6060
- name: Manage VMs
6161
href: ./how-to-guide-chamber-vm.md
62-
- name: Register a resource provider
63-
href: ./how-to-guide-register-resource-provider.md
6462
- name: Create a chamber
6563
href: ./how-to-guide-chamber.md
66-
- name: Refresh remote connection keys
67-
href: ./refresh-remote-connection-keys.md
64+
- name: Management and configuration
65+
items:
66+
- name: Refresh remote connection keys
67+
href: ./refresh-remote-connection-keys.md
68+
- name: Register a resource provider
69+
href: ./how-to-guide-register-resource-provider.md
70+
- name: Add redirect URIs
71+
href: ./how-to-guide-add-redirect-uris.md
6872
- name: Resources
6973
items:
7074
- name: Troubleshoot

0 commit comments

Comments
 (0)