Skip to content

Commit 3641ae0

Browse files
v-albemiv-albemi
authored
Merge pull request #112600 from chugugrace/mi
MI master doc
2 parents 1605dda + 5c8c292 commit 3641ae0

File tree

5 files changed

+179
-0
lines changed

5 files changed

+179
-0
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -611,6 +611,8 @@
611611
href: configure-bcdr-azure-ssis-integration-runtime.md
612612
- name: Clean up SSISDB logs with Elastic Database Jobs
613613
href: how-to-clean-up-ssisdb-logs-with-elastic-jobs.md
614+
- name: Use Azure SQL Managed Instance with Azure-SSIS IR
615+
href: how-to-use-sql-managed-instance-with-ir.md
614616
- name: Migrate SSIS jobs with SSMS
615617
href: how-to-migrate-ssis-job-ssms.md
616618
- name: Create triggers

‎articles/data-factory/how-to-use-sql-managed-instance-with-ir.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
---
2+
title: Use Azure SQL Database Managed Instance with Azure-SQL Server Integration Services (SSIS) in Azure Data Factory
3+
description: Learn how to use Azure SQL Database Managed Instance with SQL Server Integration Services (SSIS) in Azure Data Factory.
4+
services: data-factory
5+
documentationcenter: ''
6+
author: chugugrace
7+
ms.author: chugu
8+
ms.reviewer:
9+
manager:
10+
ms.service: data-factory
11+
ms.workload: data-services
12+
13+
ms.topic: conceptual
14+
ms.date: 4/15/2020
15+
---
16+
# Use Azure SQL Database Managed Instance with SQL Server Integration Services (SSIS) in Azure Data Factory
17+
18+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-xxx-md.md)]
19+
20+
You can now move your SQL Server Integration Services (SSIS) projects, packages, and workloads to the Azure cloud. Deploy, run, and manage SSIS projects and packages on Azure SQL Database or SQL Database Managed Instance with familiar tools such as SQL Server Management Studio (SSMS). This article highlights the following specific areas when using Azure SQL Database Managed Instance with Azure-SSIS integration runtime (IR):
21+
22+
- [Provision an Azure-SSIS IR with SSIS catalog (SSISDB) hosted by Azure SQL Database managed instance](#provision-azure-ssis-ir-with-ssisdb-hosted-by-azure-sql-database-managed-instance)
23+
- [Execute SSIS packages by Azure SQL Managed Instance Agent job](how-to-invoke-ssis-package-managed-instance-agent.md)
24+
- [Clean up SSISDB logs by Azure SQL Managed Instance Agent job](#clean-up-ssisdb-logs)
25+
- [Azure-SSIS IR failover with Azure SQL Database Managed Instance](configure-bcdr-azure-ssis-integration-runtime.md#azure-ssis-ir-failover-with-azure-sql-database-managed-instance)
26+
- [Migrate on-premises SSIS workloads to SSIS in ADF with Azure SQL Database managed instance as database workload destination](scenario-ssis-migration-overview.md#azure-sql-database-managed-instance-as-database-workload-destination)
27+
28+
## Provision Azure-SSIS IR with SSISDB hosted by Azure SQL Database managed instance
29+
30+
### Prerequisites
31+
32+
1. [Enable Azure Active Directory (Azure AD) on Azure SQL Database managed instance](enable-aad-authentication-azure-ssis-ir.md#configure-azure-ad-authentication-for-azure-sql-database-managed-instance), when choosing Azure Active Directory authentication.
33+
34+
1. Choose how to connect SQL managed instance, over private endpoint or over public endpoint:
35+
36+
- Over private endpoint (preferred)
37+
38+
1. Choose the virtual network for Azure-SSIS IR to join:
39+
- Inside the same virtual network as SQL managed instance, with **different subnet**.
40+
- Inside a different virtual network than the SQL managed instance, via virtual network peering (which is limited to the same region due to Global VNet peering constraints) or a connection from virtual network to virtual network.
41+
42+
For more info on SQL managed instance connectivity, see [Connect your application to Azure SQL Database managed instance](https://review.docs.microsoft.com/azure/sql-database/sql-database-managed-instance-connect-app).
43+
44+
1. [Configure virtual network](#configure-virtual-network).
45+
46+
- Over public endpoint
47+
48+
Azure SQL Database managed instances can provide connectivity over [public endpoints](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-public-endpoint-configure). Inbound and outbound requirements need to meet to allow traffic between SQL managed instance and Azure-SSIS IR:
49+
50+
- when Azure-SSIS IR not inside a virtual network (preferred)
51+
52+
**Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
53+
54+
| Transport protocol | Source | Source port range | Destination | Destination port range |
55+
|---|---|---|---|---|
56+
|TCP|Azure Cloud service tag|*|VirtualNetwork|3342|
57+
58+
For more information, see [Allow public endpoint traffic on the network security group](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-public-endpoint-configure#allow-public-endpoint-traffic-on-the-network-security-group).
59+
60+
- when Azure-SSIS IR inside a virtual network
61+
62+
There is a special scenario when SQL managed instance is in a region that Azure-SSIS IR does not support, Azure-SSIS IR is inside a virtual network without VNet peering due to Global VNet peering limitation. In this scenario, **Azure-SSIS IR inside a virtual network** connects SQL managed instance **over public endpoint**. Use below Network Security Group(NSG) rules to allow traffic between SQL managed instance and Azure-SSIS IR:
63+
64+
1. **Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
65+
66+
| Transport protocol | Source | Source port range | Destination |Destination port range |
67+
|---|---|---|---|---|
68+
|TCP|Static IP address of Azure-SSIS IR <br> For details, see [Bring Your Own Public IP for Azure-SSIS IR](join-azure-ssis-integration-runtime-virtual-network.md#publicIP).|*|VirtualNetwork|3342|
69+
70+
1. **Outbound requirement of Azure-SSIS IR**, to allow outbound traffic to SQL managed instance.
71+
72+
| Transport protocol | Source | Source port range | Destination |Destination port range |
73+
|---|---|---|---|---|
74+
|TCP|VirtualNetwork|*|[SQL managed instance public endpoint IP address](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-find-management-endpoint-ip-address)|3342|
75+
76+
### Configure virtual network
77+
78+
1. **User permission**. The user who creates the Azure-SSIS IR must have the [role assignment](https://docs.microsoft.com/azure/role-based-access-control/role-assignments-list-portal#list-role-assignments-for-a-user-at-a-scope) at least on Azure Data Factory resource with one of the options below:
79+
80+
- Use the built-in Network Contributor role. This role comes with the _Microsoft.Network/\*_ permission, which has a much larger scope than necessary.
81+
- Create a custom role that includes only the necessary _Microsoft.Network/virtualNetworks/\*/join/action_ permission. If you also want to bring your own public IP addresses for Azure-SSIS IR while joining it to an Azure Resource Manager virtual network, also include _Microsoft.Network/publicIPAddresses/*/join/action_ permission in the role.
82+
83+
1. **Virtual network**.
84+
85+
1. Make sure that the virtual network's resource group can create and delete certain Azure network resources.
86+
87+
The Azure-SSIS IR needs to create certain network resources under the same resource group as the virtual network. These resources include:
88+
- An Azure load balancer, with the name *\<Guid>-azurebatch-cloudserviceloadbalancer*
89+
- A network security group, with the name *\<Guid>-azurebatch-cloudservicenetworksecuritygroup
90+
- An Azure public IP address, with the name -azurebatch-cloudservicepublicip
91+
92+
Those resources will be created when your Azure-SSIS IR starts. They'll be deleted when your Azure-SSIS IR stops. To avoid blocking your Azure-SSIS IR from stopping, don't reuse these network resources in your other resources.
93+
94+
1. Make sure that you have no [resource lock](https://docs.microsoft.com/azure/azure-resource-manager/management/lock-resources) on the resource group/subscription to which the virtual network belongs. If you configure a read-only/delete lock, starting and stopping your Azure-SSIS IR will fail, or it will stop responding.
95+
96+
1. Make sure that you don't have an Azure policy that prevents the following resources from being created under the resource group/subscription to which the virtual network belongs:
97+
- Microsoft.Network/LoadBalancers
98+
- Microsoft.Network/NetworkSecurityGroups
99+
100+
1. Allow traffic on Network Security Group (NSG) rule, to allow traffic between SQL managed instance and Azure-SSIS IR, and traffic needed by Azure-SSIS IR.
101+
1. **Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
102+
103+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
104+
|---|---|---|---|---|---|
105+
|TCP|VirtualNetwork|*|VirtualNetwork|1433, 11000-11999|If your SQL Database server connection policy is set to **Proxy** instead of **Redirect**, only port 1433 is needed.|
106+
107+
1. **Outbound requirement of Azure-SSIS IR**, to allow outbound traffic to SQL managed instance, and other traffic needed by Azure-SSIS IR.
108+
109+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
110+
|---|---|---|---|---|---|
111+
| TCP | VirtualNetwork | * | VirtualNetwork | 1433, 11000-11999 |Allow outbound traffic to SQL managed instance. If connection policy is set to **Proxy** instead of **Redirect**, only port 1433 is needed. |
112+
| TCP | VirtualNetwork | * | AzureCloud | 443 | The nodes of your Azure-SSIS IR in the virtual network use this port to access Azure services, such as Azure Storage and Azure Event Hubs. |
113+
| TCP | VirtualNetwork | * | Internet | 80 | (Optional) The nodes of your Azure-SSIS IR in the virtual network use this port to download a certificate revocation list from the internet. If you block this traffic, you might experience performance downgrade when start IR and lose capability to check certificate revocation list for certificate usage. If you want to further narrow down destination to certain FQDNs, refer to [Use Azure ExpressRoute or User Defined Route(UDR)](https://docs.microsoft.com/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network#route).|
114+
| TCP | VirtualNetwork | * | Storage | 445 | (Optional) This rule is only required when you want to execute SSIS package stored in Azure Files. |
115+
|||||||
116+
117+
1. **Inbound requirement of Azure-SSIS IR**, to allow traffic needed by Azure-SSIS IR.
118+
119+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
120+
|---|---|---|---|---|---|
121+
| TCP | BatchNodeManagement | * | VirtualNetwork | 29876, 29877 (if you join the IR to a Resource Manager virtual network) <br/><br/>10100, 20100, 30100 (if you join the IR to a classic virtual network)| The Data Factory service uses these ports to communicate with the nodes of your Azure-SSIS IR in the virtual network. <br/><br/> Whether or not you create a subnet-level NSG, Data Factory always configures an NSG at the level of the network interface cards (NICs) attached to the virtual machines that host the Azure-SSIS IR. Only inbound traffic from Data Factory IP addresses on the specified ports is allowed by that NIC-level NSG. Even if you open these ports to internet traffic at the subnet level, traffic from IP addresses that aren't Data Factory IP addresses is blocked at the NIC level. |
122+
| TCP | CorpNetSaw | * | VirtualNetwork | 3389 | (Optional) This rule is only required when Microsoft supporter asks customer to open for advanced troubleshooting, and can be closed right after troubleshooting. **CorpNetSaw** service tag permits only secure access workstations on the Microsoft corporate network to use remote desktop. And this service tag can't be selected from portal and is only available via Azure PowerShell or Azure CLI. <br/><br/> At NIC level NSG, port 3389 is open by default and we allow you to control port 3389 at subnet level NSG, meanwhile Azure-SSIS IR has disallowed port 3389 outbound by default at windows firewall rule on each IR node for protection. |
123+
|||||||
124+
125+
1. See [virtual network configuration](join-azure-ssis-integration-runtime-virtual-network.md#virtual-network-configuration) for more info:
126+
- If you bring your own public IP addresses for the Azure-SSIS IR
127+
- If you use your own Domain Name System (DNS) server
128+
- If you use Azure ExpressRoute or a user-defined route (UDR)
129+
- If you use customized Azure-SSIS IR
130+
131+
### Provision Azure-SSIS Integration Runtime
132+
133+
1. Select SQL managed instance private endpoint or public endpoint.
134+
135+
When [provisioning Azure-SSIS IR](create-azure-ssis-integration-runtime.md#provision-an-azure-ssis-integration-runtime) in Azure portal/ADF app, on SQL Settings page, use SQL managed instance **private endpoint** or **public endpoint** when creating SSIS catalog (SSISDB).
136+
137+
Public endpoint host name comes in the format <mi_name>.public.<dns_zone>.database.windows.net and that the port used for the connection is 3342.
138+
139+
![catalog-public-endpoint](./media/how-to-use-sql-managed-instance-with-ir/catalog-public-endpoint.png)
140+
141+
1. Select Azure AD authentication when applies.
142+
143+
![catalog-public-endpoint](./media/how-to-use-sql-managed-instance-with-ir/catalog-aad.png)
144+
145+
For more info about how to enable Azure AD authentication, see [Enable Azure AD on Azure SQL Database managed instance](enable-aad-authentication-azure-ssis-ir.md#configure-azure-ad-authentication-for-azure-sql-database-managed-instance).
146+
147+
1. Join Azure-SSIS IR to the virtual network when applies.
148+
149+
On advanced setting page, select the Virtual Network and subnet to join.
150+
151+
When inside the same virtual network as SQL managed instance, choose a **different subnet** than SQL managed instance.
152+
153+
For more information about how to join Azure-SSIS IR into a virtual network, see [Join an Azure-SSIS integration runtime to a virtual network](join-azure-ssis-integration-runtime-virtual-network.md).
154+
155+
![join-virtual-network](./media/how-to-use-sql-managed-instance-with-ir/join-virtual-network.png)
156+
157+
For more info about how to create an Azure-SSIS IR, see [Create an Azure-SSIS integration runtime in Azure Data Factory](create-azure-ssis-integration-runtime.md#provision-an-azure-ssis-integration-runtime).
158+
159+
## Clean up SSISDB logs
160+
161+
SSISDB logs retention policy are defined by below properties in [catalog.catalog_properties](https://docs.microsoft.com/sql/integration-services/system-views/catalog-catalog-properties-ssisdb-database?view=sql-server-ver15):
162+
163+
- OPERATION_CLEANUP_ENABLED
164+
165+
When the value is TRUE, operation details and operation messages older than RETENTION_WINDOW (days) are deleted from the catalog. When the value is FALSE, all operation details and operation messages are stored in the catalog. Note: a SQL Server job performs the operation cleanup.
166+
167+
- RETENTION_WINDOW
168+
169+
The number of days that operation details and operation messages are stored in the catalog. When the value is -1, the retention window is infinite. Note: If no cleanup is desired, set OPERATION_CLEANUP_ENABLED to FALSE.
170+
171+
To remove SSISDB logs that are outside the retention window set by the administrator, you can trigger the stored procedure `[internal].[cleanup_server_retention_window_exclusive]`. Optionally, you can schedule SQL managed instance agent job execution to trigger the stored procedure.
172+
173+
## Next steps
174+
175+
- [Execute SSIS packages by Azure SQL Managed Instance Agent job](how-to-invoke-ssis-package-managed-instance-agent.md)
176+
- [Set up Business continuity and disaster recovery (BCDR)](configure-bcdr-azure-ssis-integration-runtime.md)
177+
- [Migrate on-premises SSIS workloads to SSIS in ADF](scenario-ssis-migration-overview.md)

‎articles/data-factory/media/how-to-use-sql-managed-instance-with-ir/catalog-aad.png

42.5 KB
Loading

‎articles/data-factory/media/how-to-use-sql-managed-instance-with-ir/catalog-public-endpoint.png

49.8 KB
Loading

‎articles/data-factory/media/how-to-use-sql-managed-instance-with-ir/join-virtual-network.png

53.3 KB
Loading

0 commit comments

Comments
 (0)