Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions website/.tool-versions
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
nodejs 20.17.0

68 changes: 67 additions & 1 deletion website/docs/docs/cloud/secure/about-private-connectivity.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,24 @@ import PrivateConnectivityMatrix from '/snippets/_private-connectivity-matrix.md

<SetUpPages features={'/snippets/_available-tiers-private-connection.md'}/>

Private connections enables secure communication from any <Constant name="cloud" /> environment to your data platform hosted on a cloud provider, such as [AWS](https://aws.amazon.com/privatelink/) or [Azure](https://azure.microsoft.com/en-us/products/private-link), using that providers private connection technology. Private connections allow <Constant name="cloud" /> customers to meet security and compliance controls as it allows connectivity between <Constant name="cloud" /> and your data platform without traversing the public internet. This feature is supported in most regions across North America, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability.
Private connections enables secure communication from any <Constant name="cloud" /> environment to your data platform hosted on a cloud provider, such as [AWS](https://aws.amazon.com/privatelink/) or [Azure](https://azure.microsoft.com/en-us/products/private-link), using that provider's private connection technology. Private connections allow <Constant name="cloud" /> customers to meet security and compliance controls as it allows connectivity between <Constant name="cloud" /> and your data platform without traversing the public internet. This feature is supported in most regions across North America, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability.

<CloudProviders type='a data platform' />

<PrivateConnectivityMatrix/>

---

## Setting up private connectivity

### Cross-region private connections

dbt Labs has globally connected private networks specifically used to host private endpoints, which are connected to <Constant name="cloud" /> instance environments. This connectivity allows for <Constant name="cloud" /> environments to connect to any supported region from any <Constant name="cloud" /> instance within the same cloud provider network. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards, in addition to the authentication and authorization mechanisms provided by each of the connected platforms.

:::note GCP regional considerations
Some GCP services, such as BigQuery, may have regional restrictions for Private Service Connect endpoints. Refer to [Google's Private Service Connect documentation](https://cloud.google.com/vpc/docs/private-service-connect) for service-specific regional availability.
:::

### Configuring private connections

<Constant name="cloud" /> supports the following data platforms for use with the private connections feature. Instructions for enabling private connections for the various data platform providers are unique. The following guides will walk you through the necessary steps, including working with [dbt Support](/community/resources/getting-help#dbt-cloud-support) to complete the connection in the dbt private network and setting up the endpoint in <Constant name="cloud" />.
Expand All @@ -38,9 +46,67 @@ dbt Labs has globally connected private networks specifically used to host priva
- [Databricks](/docs/cloud/secure/databricks-private-link)
- [Database for Postgres Flexible Server](/docs/cloud/secure/az-postgres-private-link)
- [Synapse](/docs/cloud/secure/az-synapse-private-link)
- [Self-hosted services](/docs/cloud/secure/az-self-hosted-private-link)

#### GCP
- [Snowflake](/docs/cloud/secure/snowflake-psc)
- [BigQuery](/docs/cloud/secure/bigquery-psc)
- [Self-hosted services](/docs/cloud/secure/gcp-self-hosted-psc)

<PrivateLinkHostnameWarning features={'/snippets/_private-connection-hostname-restriction.md'}/>

---

## Terminology

### Parties

<table>
<thead>
<tr>
<th>Term</th>
<th>Definition</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Consumer</strong></td>
<td>The party that creates a private endpoint to connect to a service. The consumer initiates the connection.</td>
</tr>
<tr>
<td><strong>Service producer</strong></td>
<td>The party that provisions and manages the service that the consumer connects to. The service producer publishes a resource ID that the consumer uses to finalize and establish the connection.</td>
</tr>
</tbody>
</table>

### Provisioning models

These models describe who acts as the **service producer** (the party that provisions the service that dbt Cloud connects to or that you connect to).

<table>
<thead>
<tr>
<th>Term</th>
<th>Definition</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Native</strong></td>
<td>The cloud platform (AWS, Azure, GCP) is the service producer for its own services (Redshift, Synapse, BigQuery). You obtain the resource ID from the cloud platform and share it with dbt; dbt is the consumer and creates the private endpoint.</td>
</tr>
<tr>
<td><strong>Vendor</strong></td>
<td>A third-party vendor (Snowflake, Databricks, Teradata) is the service producer. You obtain the resource ID from the vendor and share it with dbt; dbt is the consumer and creates the private endpoint.</td>
</tr>
<tr>
<td><strong>Customer-provisioned</strong></td>
<td>You are the service producer. You generate your own resource ID (endpoint service name, alias, or service attachment URI) and share it with dbt; dbt is the consumer and creates the private endpoint.</td>
</tr>
<tr>
<td><strong>dbt-provisioned</strong></td>
<td>dbt is the service producer. You are the consumer and create the private endpoint in your environment to connect to dbt Cloud. This applies only to connections TO dbt Cloud.</td>
</tr>
</tbody>
</table>
149 changes: 149 additions & 0 deletions website/docs/docs/cloud/secure/az-self-hosted-private-link.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
---
title: "Configuring Azure Private Link to your self-hosted service"
id: az-self-hosted-private-link
description: "Setting up an Azure Private Link connection between dbt and your self-hosted service."
sidebar_label: "Azure Private Link for Self-Hosted Service"
---

# Configuring Azure Private Link for a self-hosted service <Lifecycle status="managed_plus" />

import SetUpPages from '/snippets/_available-tiers-private-connection.md';

<SetUpPages features={'/snippets/_available-tiers-private-connection.md'}/>

Azure Private Link enables secure, private connectivity between <Constant name="cloud" /> and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With Private Link, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the Azure [Private Link documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview).

## What this guide covers
The focus of this guide is not on any particular service or backend architecture, but on the [Private Link Service](#terminology) that interconnects <Constant name="cloud" /> with your self-hosted service. This process should be standard across most use cases.

<!-- TODO: Add architecture diagram showing scope of guide -->
<Lightbox src="/img/docs/dbt-cloud/az-self-hosted-privatelink/scope-of-guide.png" width="90%" title="The scope of this guide" />

## Audience
This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's Microsoft Azure environment.

## Terminology
This guide uses several important terms related to Azure Private Link. Understanding these definitions will help ensure successful implementation. For a more detailed explanation of these concepts, refer to the [Azure Private Link Service documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview).

- **Consumer:** In this context, the Consumer is <Constant name="cloud" />, which creates a private endpoint to connect to your Private Link Service.
- **Service provider:** Your organization, which owns and operates the service behind the Standard Load Balancer and creates the Private Link Service.
- **Private Link Service:** The Azure resource that exposes your service to consumers, allowing them to create private endpoints to access it. This is tied to a Standard Load Balancer frontend IP configuration.
- **Alias:** A globally unique name generated by Azure for your Private Link Service. You share this alias with dbt Support to establish the connection to your service as a consumer.
- **Standard Load Balancer:** The required load balancer type that sits in front of your service. Your application must run behind a Standard Load Balancer to use Private Link Service.
- **NAT subnet:** A dedicated subnet in your VNet used for Source Network Address Translation (SNAT) IP addresses for the Private Link Service. Consumer traffic appears to originate from this pool of private IP addresses.

## Prerequisites
Before you begin, make sure to review the following requirements:

1. **Supported Load Balancer Types**

dbt has officially validated Private Link functionality with the following load balancer type:
- Standard Load Balancer (Internal)

> While other configurations may be compatible with Azure Private Link Services, this guide assumes your service is configured behind a Standard Internal Load Balancer.
> For more details, see the [Azure Load Balancer documentation](https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview).

2. **Service Health**

- Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding.

3. **dbt Azure Subscription ID**

- Contact [dbt Support](/community/resources/getting-help#dbt-cloud-support) to obtain the dbt Azure subscription ID. You will need this in order to allow dbt Cloud to connect to your Private Link Service.


## Instructions
1. Log in to the [Azure Portal](https://portal.azure.com).
2. Navigate to the Azure Subscription and Resource Group where your self-hosted service is located.

### Create a NAT subnet for the Private Link Service
3. Under the **Resources** section, search for **Virtual network** and go into the VNet where your self-hosted service is running.
4. Expand the **Settings** in the left side panel, and go into **Subnets**. Click the **+ Subnet** button to create a new subnet.
5. In the subnet creation panel:

a. **Subnet purpose:** Leave as **Default**

b. **Name:** Provide a descriptive name, such as **private-link-nat-subnet**

c. **IPv4 address range:** Choose the appropriate CIDR block from your VNet that you want to create a NAT subnet from. In this example, the CIDR is 10.30.0.0/16, as seen in the screenshot below.

d. **Starting address:** Your desired starting address of the new subnet

e. **Size:** The smallest available size is recommended (for example, /28).

f. Check the **Enable private subnet (no default outbound access)** checkbox.

g. **NAT gateway:** Leave as **None**

h. Leave **Network security group** and **Route table** fields as **None** unless your environment requires specific values here.

i. Leave all remaining fields as their default values.

j. Click **Add** to create the subnet.

<Lightbox src="/img/docs/dbt-cloud/az-self-hosted-privatelink/vnet-search.png" width="90%" title="Screenshot of step 3: Search for VNet of self-hosted service" />
<Lightbox src="/img/docs/dbt-cloud/az-self-hosted-privatelink/nat-subnet-creation.png" width="90%" title="Screenshot of steps 4-5: NAT Subnet creation for Private Link Service" />

### Create a Private Link Service
6. After the subnet creation has completed, in the search field at the top-middle of the portal, search for **Private link services**, and click on its page.
7. Click the **+ Create** button.
8. In the Create private link service page:

**Under Basics**

a. Select your **Subscription** and **Resource group**

b. **Name:** Give a descriptive name, such as **pls-to-my-vcs**

c. **Region:** Select the region where your self-hosted service is located

**Under Outbound settings**

d. **Load balancer:** In the dropdown, choose the Standard Internal Load Balancer that is in front of your self-hosted service

e. **Load balancer frontend IP address:** Choose the frontend IP configuration for your load balancer

f. **Source NAT subnet:** Select the NAT subnet you created in step 5 above

g. **Source NAT Virtual network:** This will auto-populate based on your subnet selection

h. **Enable TCP proxy V2:** Leave this disabled

**Under Access security**

i. Select **Restricted by subscription**

j. Click **Add subscriptions** and add dbt's Azure subscription ID that you acquired from support

k. Set **Request Auto-approve** selection to **Yes** for dbt's subscription

l. Click **Next: Review + create**, then **Create**

<!-- TODO: Add screenshot of Private Link Service creation -->
<Lightbox src="/img/docs/dbt-cloud/az-self-hosted-privatelink/privatelink-service-creation.png" width="90%" title="Screenshot of step 8: Creation of Azure Private Link Service" />

9. After the Private Link Service has been created, click on it to open its details page.
10. Copy the **Alias** value (this is the identifier you'll share with dbt Support).

<!-- TODO: Add screenshot of Private Link Service details page showing Alias -->
<Lightbox src="/img/docs/dbt-cloud/az-self-hosted-privatelink/alias-info.png" width="90%" title="Screenshot of step 10: Copy the Private Link Service Alias" />

### Providing dbt Support with connection details

11. Add the required information to the template below, and submit your request to [dbt Support](/community/resources/getting-help#dbt-cloud-support):

```
Subject: New Azure Self-hosted Private Link Request
- Type: Self-hosted Private Link
- Private Link Service Alias:
- Custom DNS (if HTTPS/TLS)
- DNS record:
- Service Region: (for example, East US, West Europe)
- dbt Azure multi-tenant environment (EMEA):
```

import PrivateLinkSLA from '/snippets/_private-connection-SLA.md';

<PrivateLinkSLA />


Loading
Loading