Skip to content

Commit add41a3

Browse files
chugugracechugugrace
committed
Initial updates to use sql managed instance with ir
1 parent 618f8b7 commit add41a3

File tree

9 files changed

+340
-0
lines changed

9 files changed

+340
-0
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -589,6 +589,8 @@
589589
href: how-to-invoke-ssis-package-stored-procedure-activity.md
590590
- name: Schedule Azure-SSIS integration runtime
591591
href: how-to-schedule-azure-ssis-integration-runtime.md
592+
- name: Host SSISDB by Azure SQL managed instance
593+
href: how-to-host-ssisdb-sql-managed-instance.md
592594
- name: Join Azure-SSIS IR to a virtual network
593595
href: join-azure-ssis-integration-runtime-virtual-network.md
594596
- name: Configure Self-Hosted IR as a proxy for Azure-SSIS IR

‎articles/data-factory/how-to-host-ssisdb-sql-managed-instance.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
---
2+
title: Host SQL Service Integration Services (SSIS) catalog (SSISDB) by an Azure SQL Database managed instance
3+
description: Learn the connectivity options to Azure SQL Database managed instance when provisioning an Azure-SSIS IR in Azure Data Factory with SSIS catalog (SSISDB) hosted by an Azure SQL Database managed instance.
4+
services: data-factory
5+
documentationcenter: ''
6+
author: chugugrace
7+
ms.author: chugu
8+
ms.reviewer:
9+
manager:
10+
ms.service: data-factory
11+
ms.workload: data-services
12+
13+
ms.topic: conceptual
14+
ms.date: 4/15/2020
15+
---
16+
# Host SQL Server Integration Services (SSIS) catalog (SSISDB) by an Azure SQL Database managed instance
17+
18+
When [provisioning an Azure-SQL Server Integration Services (SSIS) integration runtime (IR) in Azure Data Factory](create-azure-ssis-integration-runtime.md), SSIS Catalog (SSISDB) can be hosted on Azure SQL Database or SQL Database managed instance. This article highlights following specific areas when choosing Azure SQL Database managed instance to host SSISDB:
19+
20+
- Provision an Azure-SSIS IR with SSISDB hosted by Azure SQL Database managed instance.
21+
- [Execute SSIS packages by Azure SQL Managed Instance Agent job](how-to-invoke-ssis-package-managed-instance-agent.md).
22+
- [Clean up SSISDB logs by Azure SQL Managed Instance Agent job](#clean-up-ssisdb-logs).
23+
- [Set up Business continuity and disaster recovery (BCDR)](configure-bcdr-azure-ssis-integration-runtime.md).
24+
- [Migrate on-premises SSIS workloads to SSIS in ADF](scenario-ssis-migration-overview.md).
25+
26+
## Prerequisites
27+
28+
1. [Enable Azure Active Directory on Azure SQL Database managed instance](enable-aad-authentication-azure-ssis-ir.md#configure-azure-ad-authentication-for-azure-sql-database-managed-instance), when choosing Azure Active Directory (AAD) authentication.
29+
30+
1. Choose how to connect SQL managed instance, over private endpoint or over public endpoint:
31+
32+
- Over private endpoint (preferred)
33+
34+
1. Choose the virtual network for Azure-SSIS IR to join:
35+
- Inside the same virtual network as SQL managed instance, with **different subnet**.
36+
- Inside a different virtual network than the SQL managed instance, via virtual network peering (which is limited to the same region due to Global VNet peering constraints) or a connection from virtual network to virtual network.
37+
38+
For more info on SQL managed instance connectivity, see [Connect your application to Azure SQL Database managed instance](https://review.docs.microsoft.com/azure/sql-database/sql-database-managed-instance-connect-app).
39+
40+
1. [Configure virtual network](#configure-virtual-network).
41+
42+
- Over public endpoint
43+
44+
Azure SQL Database managed instances can provide connectivity over [public endpoints](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-public-endpoint-configure). Inbound and outbound requirements need to meet to allow traffic between SQL managed instance and Azure-SSIS IR:
45+
46+
- when Azure-SSIS IR not inside a virtual network (preferred)
47+
48+
**Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
49+
50+
| Transport protocol | Source | Source port range | Destination | Destination port range |
51+
|---|---|---|---|---|
52+
|TCP|Azure Cloud service tag|*|VirtualNetwork|3342|
53+
54+
For more information, see [Allow public endpoint traffic on the network security group](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-public-endpoint-configure#allow-public-endpoint-traffic-on-the-network-security-group).
55+
56+
- when Azure-SSIS IR inside a virtual network
57+
58+
There is a special scenario when SQL managed instance is in a region that Azure-SSIS IR does not support, Azure-SSIS IR is inside a virtual network without VNet peering due to Global VNet peering limitation. In this scenario, **Azure-SSIS IR inside a virtual network** connects SQL managed instance **over public endpoint**. Use below NSG rules to allow traffic between SQL managed instance and Azure-SSIS IR:
59+
60+
1. **Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
61+
62+
| Transport protocol | Source | Source port range | Destination |Destination port range |
63+
|---|---|---|---|---|
64+
|TCP|Static IP address of Azure-SSIS IR <br> For details, see [Bring Your Own Public IP for Azure-SSIS IR](join-azure-ssis-integration-runtime-virtual-network.md#publicIP).|*|VirtualNetwork|3342|
65+
66+
1. **Outbound requirement of Azure-SSIS IR**, to allow outbound traffic to SQL managed instance.
67+
68+
| Transport protocol | Source | Source port range | Destination |Destination port range |
69+
|---|---|---|---|---|
70+
|TCP|VirtualNetwork|*|[SQL managed instance public endpoint IP address](https://docs.microsoft.com/azure/sql-database/sql-database-managed-instance-find-management-endpoint-ip-address)|3342|
71+
72+
### Configure virtual network
73+
74+
1. **User permission**. The user who creates the Azure-SSIS IR must have the [role assignment](https://docs.microsoft.com/azure/role-based-access-control/role-assignments-list-portal#list-role-assignments-for-a-user-at-a-scope) at least on Azure Data Factory resource with one of the options below:
75+
76+
- Use the built-in Network Contributor role. This role comes with the _Microsoft.Network/\*_ permission, which has a much larger scope than necessary.
77+
- Create a custom role that includes only the necessary _Microsoft.Network/virtualNetworks/\*/join/action_ permission. If you also want to bring your own public IP addresses for Azure-SSIS IR while joining it to an Azure Resource Manager virtual network, also include _Microsoft.Network/publicIPAddresses/*/join/action_ permission in the role.
78+
79+
1. **Virtual network**.
80+
81+
1. Make sure that the virtual network's resource group can create and delete certain Azure network resources.
82+
83+
The Azure-SSIS IR needs to create certain network resources under the same resource group as the virtual network. These resources include:
84+
- An Azure load balancer, with the name *\<Guid>-azurebatch-cloudserviceloadbalancer*
85+
- A network security group, with the name *\<Guid>-azurebatch-cloudservicenetworksecuritygroup
86+
- An Azure public IP address, with the name -azurebatch-cloudservicepublicip
87+
88+
Those resources will be created when your Azure-SSIS IR starts. They'll be deleted when your Azure-SSIS IR stops. To avoid blocking your Azure-SSIS IR from stopping, don't reuse these network resources in your other resources.
89+
90+
1. Make sure that you have no [resource lock](https://docs.microsoft.com/azure/azure-resource-manager/management/lock-resources) on the resource group/subscription to which the virtual network belongs. If you configure a read-only/delete lock, starting and stopping your Azure-SSIS IR will fail, or it will stop responding.
91+
92+
1. Make sure that you don't have an Azure policy that prevents the following resources from being created under the resource group/subscription to which the virtual network belongs:
93+
- Microsoft.Network/LoadBalancers
94+
- Microsoft.Network/NetworkSecurityGroups
95+
96+
1. Allow traffic on Network Security Group (NSG) rule, to allow traffic between SQL managed instance and Azure-SSIS IR, and traffic needed by Azure-SSIS IR.
97+
1. **Inbound requirement of SQL managed instance**, to allow inbound traffic from Azure-SSIS IR.
98+
99+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
100+
|---|---|---|---|---|---|
101+
|TCP|VirtualNetwork|*|VirtualNetwork|1433, 11000-11999|If your SQL Database server connection policy is set to **Proxy** instead of **Redirect**, only port 1433 is needed.|
102+
103+
1. **Outbound requirement of Azure-SSIS IR**, to allow outbound traffic to SQL managed instance, and other traffic needed by Azure-SSIS IR.
104+
105+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
106+
|---|---|---|---|---|---|
107+
| TCP | VirtualNetwork | * | VirtualNetwork | 1433, 11000-11999 |Allow outbound traffic to SQL managed instance. If connection policy is set to **Proxy** instead of **Redirect**, only port 1433 is needed. |
108+
| TCP | VirtualNetwork | * | AzureCloud | 443 | The nodes of your Azure-SSIS IR in the virtual network use this port to access Azure services, such as Azure Storage and Azure Event Hubs. |
109+
| TCP | VirtualNetwork | * | Internet | 80 | (Optional) The nodes of your Azure-SSIS IR in the virtual network use this port to download a certificate revocation list from the internet. If you block this traffic, you might experience performance downgrade when start IR and lose capability to check certificate revocation list for certificate usage. If you want to further narrow down destination to certain FQDNs, refer to **Use Azure ExpressRoute or UDR** section|
110+
| TCP | VirtualNetwork | * | Storage | 445 | (Optional) This rule is only required when you want to execute SSIS package stored in Azure Files. |
111+
|||||||
112+
113+
1. **Inbound requirement of Azure-SSIS IR**, to allow traffic needed by Azure-SSIS IR.
114+
115+
| Transport protocol | Source | Source port range | Destination | Destination port range | Comments |
116+
|---|---|---|---|---|---|
117+
| TCP | BatchNodeManagement | * | VirtualNetwork | 29876, 29877 (if you join the IR to a Resource Manager virtual network) <br/><br/>10100, 20100, 30100 (if you join the IR to a classic virtual network)| The Data Factory service uses these ports to communicate with the nodes of your Azure-SSIS IR in the virtual network. <br/><br/> Whether or not you create a subnet-level NSG, Data Factory always configures an NSG at the level of the network interface cards (NICs) attached to the virtual machines that host the Azure-SSIS IR. Only inbound traffic from Data Factory IP addresses on the specified ports is allowed by that NIC-level NSG. Even if you open these ports to internet traffic at the subnet level, traffic from IP addresses that aren't Data Factory IP addresses is blocked at the NIC level. |
118+
| TCP | CorpNetSaw | * | VirtualNetwork | 3389 | (Optional) This rule is only required when Microsoft supporter asks customer to open for advanced troubleshooting, and can be closed right after troubleshooting. **CorpNetSaw** service tag permits only secure access workstations on the Microsoft corporate network to use remote desktop. And this service tag can't be selected from portal and is only available via Azure PowerShell or Azure CLI. <br/><br/> At NIC level NSG, port 3389 is open by default and we allow you to control port 3389 at subnet level NSG, meanwhile Azure-SSIS IR has disallowed port 3389 outbound by default at windows firewall rule on each IR node for protection. |
119+
|||||||
120+
121+
1. See [virtual network configuration](join-azure-ssis-integration-runtime-virtual-network.md#virtual-network-configuration) for more info:
122+
- If you bring your own public IP addresses for the Azure-SSIS IR.
123+
- If you use your own Domain Name System (DNS) server.
124+
- If you use Azure ExpressRoute or a user-defined route (UDR).
125+
- If you use customized Azure-SSIS IR.
126+
127+
## Provision Azure-SSIS Integration Runtime
128+
129+
1. Select SQL managed instance private endpoint or public endpoint.
130+
131+
When [provisioning Azure-SSIS IR](create-azure-ssis-integration-runtime.md#provision-an-azure-ssis-integration-runtime) in Azure portal/ADF app, on SQL Settings page, use SQL managed instance **private endpoint** or **public endpoint** when creating SSIS catalog (SSISDB).
132+
133+
Public endpoint host name comes in the format <mi_name>.public.<dns_zone>.database.windows.net and that the port used for the connection is 3342.
134+
135+
![catalog-public-endpoint](./media/how-to-host-ssisdb-sql-managed-instance/catalog-public-endpoint.png)
136+
137+
1. Select AAD authentication when applies.
138+
139+
![catalog-public-endpoint](./media/how-to-host-ssisdb-sql-managed-instance/catalog-aad.png)
140+
141+
For more info about how to enable AAD authentication, see [Enable Azure AD on Azure SQL Database managed instance](enable-aad-authentication-azure-ssis-ir.md#configure-azure-ad-authentication-for-azure-sql-database-managed-instance).
142+
143+
1. Join Azure-SSIS IR to the virtual network when applies.
144+
145+
On advanced setting page, select the Virtual Network and subnet to join.
146+
147+
When inside the same virtual network as SQL managed instance, choose a **different subnet** than SQL managed instance.
148+
149+
For more information about how to join Azure-SSIS IR into a virtual network, see [Join an Azure-SSIS integration runtime to a virtual network](join-azure-ssis-integration-runtime-virtual-network.md).
150+
151+
![join-virtual-network](./media/how-to-host-ssisdb-sql-managed-instance/join-virtual-network.png)
152+
153+
For more info about how to create an Azure-SSIS IR, see [Create an Azure-SSIS integration runtime in Azure Data Factory](create-azure-ssis-integration-runtime.md#provision-an-azure-ssis-integration-runtime).
154+
155+
## Clean up SSISDB logs
156+
157+
To remove SSISDB logs that are outside the retention window set by the administrator, you can trigger the stored procedure `[internal].[cleanup_server_retention_window_exclusive]`. Optionally, you can schedule SQL managed instance agent job execution to trigger the stored procedure.
158+
159+
## Next steps
160+
161+
- [Execute SSIS packages by Azure SQL Managed Instance Agent job](how-to-invoke-ssis-package-managed-instance-agent.md)
162+
- [Monitor the Azure-SSIS integration runtime](monitor-integration-runtime.md#azure-ssis-integration-runtime)
163+
- [Set up Business continuity and disaster recovery (BCDR)](configure-bcdr-azure-ssis-integration-runtime.md)
164+
- [Migrate on-premises SSIS workloads to SSIS in ADF](scenario-ssis-migration-overview.md)
165+
- [Configure the Azure-SSIS Integration Runtime for high performance](configure-azure-ssis-integration-runtime-performance.md)

0 commit comments

Comments
 (0)