diff --git a/website/.tool-versions b/website/.tool-versions new file mode 100644 index 00000000000..987b725dfd4 --- /dev/null +++ b/website/.tool-versions @@ -0,0 +1,2 @@ +nodejs 20.17.0 + diff --git a/website/docs/docs/cloud/secure/about-private-connectivity.md b/website/docs/docs/cloud/secure/about-private-connectivity.md index 01185f9e816..86831aac82b 100644 --- a/website/docs/docs/cloud/secure/about-private-connectivity.md +++ b/website/docs/docs/cloud/secure/about-private-connectivity.md @@ -12,16 +12,24 @@ import PrivateConnectivityMatrix from '/snippets/_private-connectivity-matrix.md -Private connections enables secure communication from any environment to your data platform hosted on a cloud provider, such as [AWS](https://aws.amazon.com/privatelink/) or [Azure](https://azure.microsoft.com/en-us/products/private-link), using that provider’s private connection technology. Private connections allow customers to meet security and compliance controls as it allows connectivity between and your data platform without traversing the public internet. This feature is supported in most regions across North America, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability. +Private connections enables secure communication from any environment to your data platform hosted on a cloud provider, such as [AWS](https://aws.amazon.com/privatelink/) or [Azure](https://azure.microsoft.com/en-us/products/private-link), using that provider's private connection technology. Private connections allow customers to meet security and compliance controls as it allows connectivity between and your data platform without traversing the public internet. This feature is supported in most regions across North America, Europe, and Asia, but [contact us](https://www.getdbt.com/contact/) if you have questions about availability. +--- + +## Setting up private connectivity + ### Cross-region private connections dbt Labs has globally connected private networks specifically used to host private endpoints, which are connected to instance environments. This connectivity allows for environments to connect to any supported region from any instance within the same cloud provider network. To ensure security, access to these endpoints is protected by security groups, network policies, and application connection safeguards, in addition to the authentication and authorization mechanisms provided by each of the connected platforms. +:::note GCP regional considerations +Some GCP services, such as BigQuery, may have regional restrictions for Private Service Connect endpoints. Refer to [Google's Private Service Connect documentation](https://cloud.google.com/vpc/docs/private-service-connect) for service-specific regional availability. +::: + ### Configuring private connections supports the following data platforms for use with the private connections feature. Instructions for enabling private connections for the various data platform providers are unique. The following guides will walk you through the necessary steps, including working with [dbt Support](/community/resources/getting-help#dbt-cloud-support) to complete the connection in the dbt private network and setting up the endpoint in . @@ -38,9 +46,67 @@ dbt Labs has globally connected private networks specifically used to host priva - [Databricks](/docs/cloud/secure/databricks-private-link) - [Database for Postgres Flexible Server](/docs/cloud/secure/az-postgres-private-link) - [Synapse](/docs/cloud/secure/az-synapse-private-link) +- [Self-hosted services](/docs/cloud/secure/az-self-hosted-private-link) #### GCP - [Snowflake](/docs/cloud/secure/snowflake-psc) - [BigQuery](/docs/cloud/secure/bigquery-psc) +- [Self-hosted services](/docs/cloud/secure/gcp-self-hosted-psc) + +--- + +## Terminology + +### Parties + + + + + + + + + + + + + + + + + + +
TermDefinition
ConsumerThe party that creates a private endpoint to connect to a service. The consumer initiates the connection.
Service producerThe party that provisions and manages the service that the consumer connects to. The service producer publishes a resource ID that the consumer uses to finalize and establish the connection.
+ +### Provisioning models + +These models describe who acts as the **service producer** (the party that provisions the service that dbt Cloud connects to or that you connect to). + + + + + + + + + + + + + + + + + + + + + + + + + + +
TermDefinition
NativeThe cloud platform (AWS, Azure, GCP) is the service producer for its own services (Redshift, Synapse, BigQuery). You obtain the resource ID from the cloud platform and share it with dbt; dbt is the consumer and creates the private endpoint.
VendorA third-party vendor (Snowflake, Databricks, Teradata) is the service producer. You obtain the resource ID from the vendor and share it with dbt; dbt is the consumer and creates the private endpoint.
Customer-provisionedYou are the service producer. You generate your own resource ID (endpoint service name, alias, or service attachment URI) and share it with dbt; dbt is the consumer and creates the private endpoint.
dbt-provisioneddbt is the service producer. You are the consumer and create the private endpoint in your environment to connect to dbt Cloud. This applies only to connections TO dbt Cloud.
diff --git a/website/docs/docs/cloud/secure/az-self-hosted-private-link.md b/website/docs/docs/cloud/secure/az-self-hosted-private-link.md new file mode 100644 index 00000000000..6cf4f8c780d --- /dev/null +++ b/website/docs/docs/cloud/secure/az-self-hosted-private-link.md @@ -0,0 +1,149 @@ +--- +title: "Configuring Azure Private Link to your self-hosted service" +id: az-self-hosted-private-link +description: "Setting up an Azure Private Link connection between dbt and your self-hosted service." +sidebar_label: "Azure Private Link for Self-Hosted Service" +--- + +# Configuring Azure Private Link for a self-hosted service + +import SetUpPages from '/snippets/_available-tiers-private-connection.md'; + + + +Azure Private Link enables secure, private connectivity between and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With Private Link, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the Azure [Private Link documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview). + +## What this guide covers +The focus of this guide is not on any particular service or backend architecture, but on the [Private Link Service](#terminology) that interconnects with your self-hosted service. This process should be standard across most use cases. + + + + +## Audience +This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's Microsoft Azure environment. + +## Terminology +This guide uses several important terms related to Azure Private Link. Understanding these definitions will help ensure successful implementation. For a more detailed explanation of these concepts, refer to the [Azure Private Link Service documentation](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview). + +- **Consumer:** In this context, the Consumer is , which creates a private endpoint to connect to your Private Link Service. +- **Service provider:** Your organization, which owns and operates the service behind the Standard Load Balancer and creates the Private Link Service. +- **Private Link Service:** The Azure resource that exposes your service to consumers, allowing them to create private endpoints to access it. This is tied to a Standard Load Balancer frontend IP configuration. +- **Alias:** A globally unique name generated by Azure for your Private Link Service. You share this alias with dbt Support to establish the connection to your service as a consumer. +- **Standard Load Balancer:** The required load balancer type that sits in front of your service. Your application must run behind a Standard Load Balancer to use Private Link Service. +- **NAT subnet:** A dedicated subnet in your VNet used for Source Network Address Translation (SNAT) IP addresses for the Private Link Service. Consumer traffic appears to originate from this pool of private IP addresses. + +## Prerequisites +Before you begin, make sure to review the following requirements: + +1. **Supported Load Balancer Types** + + dbt has officially validated Private Link functionality with the following load balancer type: + - Standard Load Balancer (Internal) + + > While other configurations may be compatible with Azure Private Link Services, this guide assumes your service is configured behind a Standard Internal Load Balancer. + > For more details, see the [Azure Load Balancer documentation](https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview). + +2. **Service Health** + + - Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding. + +3. **dbt Azure Subscription ID** + + - Contact [dbt Support](/community/resources/getting-help#dbt-cloud-support) to obtain the dbt Azure subscription ID. You will need this in order to allow dbt Cloud to connect to your Private Link Service. + + +## Instructions +1. Log in to the [Azure Portal](https://portal.azure.com). +2. Navigate to the Azure Subscription and Resource Group where your self-hosted service is located. + +### Create a NAT subnet for the Private Link Service +3. Under the **Resources** section, search for **Virtual network** and go into the VNet where your self-hosted service is running. +4. Expand the **Settings** in the left side panel, and go into **Subnets**. Click the **+ Subnet** button to create a new subnet. +5. In the subnet creation panel: + + a. **Subnet purpose:** Leave as **Default** + + b. **Name:** Provide a descriptive name, such as **private-link-nat-subnet** + + c. **IPv4 address range:** Choose the appropriate CIDR block from your VNet that you want to create a NAT subnet from. In this example, the CIDR is 10.30.0.0/16, as seen in the screenshot below. + + d. **Starting address:** Your desired starting address of the new subnet + + e. **Size:** The smallest available size is recommended (for example, /28). + + f. Check the **Enable private subnet (no default outbound access)** checkbox. + + g. **NAT gateway:** Leave as **None** + + h. Leave **Network security group** and **Route table** fields as **None** unless your environment requires specific values here. + + i. Leave all remaining fields as their default values. + + j. Click **Add** to create the subnet. + + + + +### Create a Private Link Service +6. After the subnet creation has completed, in the search field at the top-middle of the portal, search for **Private link services**, and click on its page. +7. Click the **+ Create** button. +8. In the Create private link service page: + + **Under Basics** + + a. Select your **Subscription** and **Resource group** + + b. **Name:** Give a descriptive name, such as **pls-to-my-vcs** + + c. **Region:** Select the region where your self-hosted service is located + + **Under Outbound settings** + + d. **Load balancer:** In the dropdown, choose the Standard Internal Load Balancer that is in front of your self-hosted service + + e. **Load balancer frontend IP address:** Choose the frontend IP configuration for your load balancer + + f. **Source NAT subnet:** Select the NAT subnet you created in step 5 above + + g. **Source NAT Virtual network:** This will auto-populate based on your subnet selection + + h. **Enable TCP proxy V2:** Leave this disabled + + **Under Access security** + + i. Select **Restricted by subscription** + + j. Click **Add subscriptions** and add dbt's Azure subscription ID that you acquired from support + + k. Set **Request Auto-approve** selection to **Yes** for dbt's subscription + + l. Click **Next: Review + create**, then **Create** + + + + +9. After the Private Link Service has been created, click on it to open its details page. +10. Copy the **Alias** value (this is the identifier you'll share with dbt Support). + + + + +### Providing dbt Support with connection details + +11. Add the required information to the template below, and submit your request to [dbt Support](/community/resources/getting-help#dbt-cloud-support): + +``` +Subject: New Azure Self-hosted Private Link Request +- Type: Self-hosted Private Link +- Private Link Service Alias: +- Custom DNS (if HTTPS/TLS) + - DNS record: +- Service Region: (for example, East US, West Europe) +- dbt Azure multi-tenant environment (EMEA): +``` + +import PrivateLinkSLA from '/snippets/_private-connection-SLA.md'; + + + + diff --git a/website/docs/docs/cloud/secure/gcp-self-hosted-psc.md b/website/docs/docs/cloud/secure/gcp-self-hosted-psc.md new file mode 100644 index 00000000000..2a688f20a0e --- /dev/null +++ b/website/docs/docs/cloud/secure/gcp-self-hosted-psc.md @@ -0,0 +1,126 @@ +--- +title: "Configuring GCP Private Service Connect to your self-hosted service" +id: gcp-self-hosted-psc +description: "Setting up a GCP Private Service Connect connection between dbt and your self-hosted service." +sidebar_label: "GCP Private Service Connect for Self-Hosted Service" +--- + +# Configuring GCP Private Service Connect for a self-hosted service + + + +import SetUpPages from '/snippets/_available-tiers-private-connection.md'; + + + +GCP Private Service Connect (PSC) enables secure, private connectivity between and your self-hosted services. These services may include version control systems (VCS), data warehouses, or any other applications you manage. With PSC, you do not need to expose your service to the public internet. All communication occurs over a private network, significantly enhancing security. For more details, refer to the GCP [Private Service Connect documentation](https://cloud.google.com/private-service-connect). + +## What this guide covers +The focus of this guide is not on any particular service or [Backend](#terminology) architecture, but on the [Service Attachment](#terminology) that interconnects with your self-hosted service. This attachment process should be standard across most use cases. + + + +## Audience +This guide is intended for cloud network administrators or engineers responsible for configuring and maintaining secure network communications within your organization's Google Cloud Platform (GCP) environment. + +## Terminology +This guide uses several important terms related to Private Service Connect. Understanding these definitions will help ensure successful implementation. For a more detailed explanation of these concepts, refer to the [GCP Private Service Connect documentation](https://docs.cloud.google.com/vpc/docs/private-service-connect#managed-services). + +- **Consumer:** In this context, the Consumer is , which establishes the PSC connection as the client. +- **Published Service:** The service you are exposing via PSC to dbt Cloud, such as your version control system (VCS), data warehouse, or another application. +- **Service Attachment:** Refers to the resource that is shared with consumer(s) of your Published Service, so that they can establish endpoints to it. +- **Backend:** Can also be referred to as Network Endpoint Groups (NEGs). This is the particular architecture that your service is running on. For example, this may be VMs, GKE Instance Groups, or even on-prem IPs. + +## Prerequisites +Before you begin, make sure to review the following requirements: + +1. **Supported Load Balancer Types** + + dbt has officially validated Private Service Connect (PSC) functionality with the following load balancer types: + - Regional Internal Proxy Load Balancer + - Cross-Regional Internal Proxy Load Balancer + + > While other load balancer types can be compatible with PSC Service Attachments, this guide assumes your service is configured behind one of the officially supported Proxy Load Balancers. + > For more details, see the [Proxy Load Balancers documentation](https://docs.cloud.google.com/load-balancing/docs/tcp/internal-proxy). + +2. **Service Health** + + - Confirm that your service or application is operational and healthy behind the designated load balancer before proceeding. + +3. **dbt GCP Project ID** + + - Contact [dbt Support](/community/resources/getting-help#dbt-cloud-support) to obtain the dbt GCP project ID. You will need this in order to share your service attachment with dbt Cloud. + + +## Instructions +1. Log in to the Google Cloud Platform [console](https://console.cloud.google.com) +2. Navigate to the GCP Organization and Project that your self-hosted service is in. + +### Create a dedicated Service Attachment subnet +3. In the search field at the top-middle of the console, search for **VPC networks** and navigate to its product page. +4. On the product page, click the VPC network link where your self-hosted service is located. +5. Select the **Subnets** tab on the next page, and click the **Add subnet** button. +6. In the subnet creation panel: + + a. **Name:** Provide a descriptive name, such as **service-attachment-subnet** + + b. **Description:** This subnet is dedicated to service attachment(s) + + c. **Region:** Pick the region of your self-hosted service + + d. **Purpose:** Choose **Private Service Connect** + + e. Click **Add** to create the subnet + + + +### Create a Service Attachment +7. After the subnet creation for the service attachment has completed, in the search field at the top-middle of the console, search for **Private Service Connect**, and click on its product page. +8. On the product page, select the **Published services** tab, and click the **Publish service** button. +9. In the Publish service page: + + **Under Target details** + + a. Choose **Load Balancer** + + b. The load balancer types that has validated are the **Regional Internal Proxy Load Balancer** and the **Cross-Regional Internal Proxy Load Balancer**. However, the others may work as well, although not officially supported. + + c. In the **Load balancer** dropdown, choose the load balancer that is in front of your self-hosted service. + + d. Choose the relevant **Forwarding rule** from the dropdown for your load balancer. + + **Under Service details** + + e. Give a descriptive **Service Name**, such as **service-to-my-vcs** + + f. In the **Subnets** dropdown, choose the subnet that you created in step 6 above. + + **Under Connection Preference** + + g. Leave the selection on **Accept connections from selected projects** + + h. Click the **Add accepted project** button and add dbt's GCP project ID that you acquired from support. Note: This project ID may differ for each configuration. + + - Set connection limit to 1 + + i. Click **Add service** + + + +10. After the Published Service attachment has been created, click on it to open its details page. +11. Copy the **Service attachment** URI (_not_ the Service attachment ID). + + + +### Providing dbt Support with connection details + +12. Add the required information to the template below, and submit your request to [dbt Support](/community/resources/getting-help#dbt-cloud-support): + +``` +Subject: New GCP Self-hosted Private Service Connect Request +- Type: Self-hosted PSC +- Service Attachment URI: +- Custom DNS (if HTTPS/TLS) + - DNS record: +- Service Region: (for example, us-east1, us-central1) +``` \ No newline at end of file diff --git a/website/docs/docs/cloud/secure/vcs-privatelink.md b/website/docs/docs/cloud/secure/vcs-privatelink.md index f883cac90f3..6cc60a33e91 100644 --- a/website/docs/docs/cloud/secure/vcs-privatelink.md +++ b/website/docs/docs/cloud/secure/vcs-privatelink.md @@ -23,7 +23,7 @@ You will learn, at a high level, the resources necessary to implement this solut ### Required resources for creating a connection -Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in your AWS account(s) and private network containing the self-hosted VCS instance. You are responsible for provisioning and maintaining these resources. Once provisioned, connection information and permissions are shared with dbt Labs to complete the connection, allowing for direct VPC to VPC private connectivity. +Creating an Interface VPC PrivateLink connection requires creating multiple AWS resources in your AWS account(s) and private network containing the self-hosted VCS instance. You are responsible for provisioning and maintaining these resources. Once provisioned, connection information and permissions are shared with dbt Labs to complete the connection, allowing for direct VPC to VPC private connectivity. This approach is distinct from and does not require you to implement VPC peering between your AWS account(s) and . @@ -71,11 +71,11 @@ Once the VPC Endpoint Service is provisioned and configured find the service nam :::note Custom DNS configuration - + If the connection to the VCS service requires a custom domain and/or URL for TLS, a private hosted zone can be configured by the dbt Labs Infrastructure team in the private network. For example: - Private hosted zone: examplecorp.com - DNS record: github.examplecorp.com - + ::: ### 4. Add the required information to the template below, and submit your request to [dbt Support](/community/resources/getting-help#dbt-cloud-support): @@ -104,7 +104,7 @@ Once you accept the endpoint connection request, you can use the PrivateLink end ## Configure in dbt -Once dbt confirms that the PrivateLink integration is complete, you can use it in a new or existing git configuration. +Once dbt confirms that the PrivateLink integration is complete, you can use it in a new or existing git configuration. **To configure a new git integration with PrivateLink:** @@ -113,15 +113,15 @@ Once dbt confirms that the PrivateLink integration is complete, you can use it i 3. Name your project and configure your development environment. 4. Under **Set up repository**, click **Git clone**. 5. Select **PrivateLink Endpoint** as the connection type. - Your configured integrations will appear in the dropdown menu. + Your configured integrations will appear in the dropdown menu. 6. Select the configured endpoint from the dropdown list. 7. Click **Save**. -**To configure an existing git integration with PrivateLink:** +**To configure an existing git integration with PrivateLink:** 1. Click your account name at the bottom left-hand menu and go to **Account settings** > **Integrations**. 2. Under **Gitlab**, select **PrivateLink Endpoint** as the connection type. - Your configured integrations will appear in the dropdown menu. + Your configured integrations will appear in the dropdown menu. 3. Select the configured endpoint from the dropdown list. 4. Click **Save**. diff --git a/website/docs/terms/hover-terms.md b/website/docs/terms/hover-terms.md index dae99af5595..52e5ccd2de6 100644 --- a/website/docs/terms/hover-terms.md +++ b/website/docs/terms/hover-terms.md @@ -3,7 +3,7 @@ aggregate: displayText: aggregate hoverSnippet: A UDF type that returns a single value per group, aggregating several rows. -cte: +cte: displayText: CTE hoverSnippet: A Common Table Expression (CTE) is a temporary result set that can be used in a SQL query. You can use CTEs to break up complex queries into simpler blocks of code that can connect and build on each other. @@ -31,19 +31,19 @@ data-catalog: displayText: data catalog hoverSnippet: A data catalog is an inventory of data assets from different parts of the data stack within an organization. This catalog can display metadata, lineage, and business definitions from your different data sources. -data-wrangling: +data-wrangling: displayText: data wrangling hoverSnippet: Data wrangling describes the different processes used to transform raw data into a consistent and easily usable format. The ultimate goal of data wrangling is to work in a way that allows you to dive right into analysis on a dataset or build upon that data. dataframe: - displayText: dataframe + displayText: dataframe hoverSnippet: A DataFrame is a two-dimensional data structure (rows and columns). It's the most common way of representing and interacting with large datasets in Python. ddl: - displayText: DDL + displayText: DDL hoverSnippet: Data Definition Language (DDL) is a group of SQL statements that you can execute to manage database objects, including tables, views, and more. -deploying: +deploying: displayText: Deploying hoverSnippet: Deploying dbt in production means setting up a system to run a dbt job on a schedule, rather than running dbt commands manually from the command line. @@ -52,19 +52,19 @@ dimensional-modeling: hoverSnippet: Dimensional modeling is a data modeling technique where you break data up into “facts” and “dimensions” to organize and describe entities within your data warehouse. dml: - displayText: DML - hoverSnippet: Data Manipulation Language (DML) is a class of SQL statements that are used to query, edit, add and delete row-level data from database tables or views. The main DML statements are SELECT, INSERT, DELETE, and UPDATE. + displayText: DML + hoverSnippet: Data Manipulation Language (DML) is a class of SQL statements that are used to query, edit, add and delete row-level data from database tables or views. The main DML statements are SELECT, INSERT, DELETE, and UPDATE. dry: - displayText: DRY + displayText: DRY hoverSnippet: DRY is a software development principle that stands for “Don’t Repeat Yourself.” Living by this principle means that your aim is to reduce repetitive patterns and duplicate code and logic in favor of modular and referenceable code. edw: - displayText: EDW + displayText: EDW hoverSnippet: An Enterprise Data Warehouse (EDW), like any other data warehouse, is a collection of databases that centralize a business's information from multiple sources and applications. elt: - displayText: ELT + displayText: ELT hoverSnippet: Extract, Load, Transform (ELT) is the process of first extracting data from different data sources, loading it into a target data warehouse, and finally transforming it. etl: @@ -72,7 +72,7 @@ etl: hoverSnippet: Extract, Transform, Load (ETL) is the process of first extracting data from a data source, transforming it, and then loading it into a target data warehouse. grain: - displayText: grain + displayText: grain hoverSnippet: Your data's grain is the combination of columns at which records in a table are unique. Ideally, this is captured in a single column and a unique primary key. idempotent: @@ -88,15 +88,15 @@ lsp: hoverSnippet: Language Server Protocol (LSP) enables developer features like live CTE previews, hover info, error highlighting, and more. materialization: - displayText: materialization - hoverSnippet: The exact Data Definition Language (DDL) that dbt will use when creating the model’s equivalent in a data warehouse. + displayText: materialization + hoverSnippet: The exact Data Definition Language (DDL) that dbt will use when creating the model’s equivalent in a data warehouse. -model: +model: hoverSnippet: A model is an essential building block of the DAG displayText: model monotonically-increasing: - displayText: monotonically increasing + displayText: monotonically increasing hoverSnippet: A monotonically-increasing sequence is a sequence whose values are sorted in ascending order and do not decrease. For example, the sequences 1, 6, 7, 11, 131 or 2, 5, 5, 5, 6, 10. predicate-pushdown: @@ -104,7 +104,7 @@ predicate-pushdown: hoverSnippet: A predicate pushdown is an expression used to determine what rows in a database apply to a particular query primary-key: - displayText: primary key + displayText: primary key hoverSnippet: A primary key is a non-null column in a database object that uniquely identifies each row. relational-database: @@ -113,7 +113,7 @@ relational-database: reverse-etl: displayText: reverse ETL - hoverSnippet: Reverse ETL is the process of getting your transformed data stored in your data warehouse to end business platforms, such as sales CRMs and ad platforms. + hoverSnippet: Reverse ETL is the process of getting your transformed data stored in your data warehouse to end business platforms, such as sales CRMs and ad platforms. scalar: displayText: scalar @@ -128,7 +128,7 @@ sql-expression: hoverSnippet: A SQL expression is a combination of columns, values, operators, and functions that evaluates to a single value. sql-rendering: - displayText: SQL rendering + displayText: SQL rendering hoverSnippet: The dbt Core engine takes SQL with Jinja, and renders all the macros present in the model to produce SQL that is ready to run against the database. For SQL parsing and compilation capabilities, use the Fusion engine instead to better understand your SQL structure. subquery: @@ -136,14 +136,46 @@ subquery: hoverSnippet: A subquery is a query within another query. Subqueries are often used when you need to process data in multiple steps. surrogate-key: - displayText: surrogate key + displayText: surrogate key hoverSnippet: A surrogate key is a unique identifier derived from the data itself. It often takes the form of a hashed value of multiple columns that will create a uniqueness constraint for each row. table: - displayText: table - hoverSnippet: In simplest terms, a table is the direct storage of data in rows and columns. Think excel sheet with raw values in each of the cells. + displayText: table + hoverSnippet: In simplest terms, a table is the direct storage of data in rows and columns. Think excel sheet with raw values in each of the cells. view: - displayText: view + displayText: view hoverSnippet: A view (as opposed to a table) is a defined passthrough SQL query that can be run against a database (or data warehouse). + +service-provider: + displayText: Service provider + hoverSnippet: The party that publishes a service for private access. This can be a third-party vendor (Snowflake, Databricks) or the cloud platform itself (Redshift, BigQuery). + +consumer: + displayText: Consumer + hoverSnippet: The party that creates a private endpoint to connect to a service. When dbt Cloud is the consumer, it connects to your services. + +native-provisioned: + displayText: Native + hoverSnippet: The cloud platform (AWS, Azure, GCP) is the service producer for its own services (Redshift, Synapse, BigQuery). + +vendor-provisioned: + displayText: Vendor + hoverSnippet: A third-party vendor (Snowflake, Databricks, Teradata) provisions the private connectivity infrastructure. + +customer-provisioned: + displayText: Customer-provisioned + hoverSnippet: You create and manage the private connectivity infrastructure and share access with dbt. + +dedicated-endpoint: + displayText: Dedicated endpoint + hoverSnippet: A private endpoint created specifically for your account, providing both network isolation and access controls. + +shared-endpoint: + displayText: Shared endpoint + hoverSnippet: A private endpoint maintained by dbt that multiple customers use, with traffic isolated via access controls only. + +dbt-provisioned: + displayText: dbt-provisioned + hoverSnippet: dbt provisions and manages the endpoint service that your private endpoint connects to. This applies to connections TO dbt Cloud. --- diff --git a/website/sidebars.js b/website/sidebars.js index b197b5814df..2c4709eb2c7 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -226,8 +226,10 @@ const sidebarSettings = { "docs/cloud/secure/postgres-privatelink", "docs/cloud/secure/az-postgres-private-link", "docs/cloud/secure/az-synapse-private-link", + "docs/cloud/secure/az-self-hosted-private-link", "docs/cloud/secure/bigquery-psc", "docs/cloud/secure/vcs-privatelink", + "docs/cloud/secure/gcp-self-hosted-psc", ], }, // PrivateLink ], @@ -240,7 +242,7 @@ const sidebarSettings = { items: [ "docs/about-dbt-install", "docs/core/dbt-core-environments", - { + { type: "category", label: "Install dbt Fusion engine", link: { type: "doc", id: "docs/fusion/about-fusion-install" }, @@ -653,7 +655,7 @@ const sidebarSettings = { ], }, "docs/deploy/continuous-deployment", - { + { type: "category", label: "State aware", link: { type: "doc", id: "docs/deploy/state-aware-about" }, @@ -826,13 +828,13 @@ const sidebarSettings = { collapsed: true, link: { type: "doc", id: "docs/dbt-ai/about-mcp", }, items: [ - "docs/dbt-ai/about-mcp", - "docs/dbt-ai/setup-local-mcp", - "docs/dbt-ai/setup-remote-mcp", - "docs/dbt-ai/integrate-mcp-vscode", - "docs/dbt-ai/integrate-mcp-cursor", - "docs/dbt-ai/integrate-mcp-claude", - ], + "docs/dbt-ai/about-mcp", + "docs/dbt-ai/setup-local-mcp", + "docs/dbt-ai/setup-remote-mcp", + "docs/dbt-ai/integrate-mcp-vscode", + "docs/dbt-ai/integrate-mcp-cursor", + "docs/dbt-ai/integrate-mcp-claude", + ], }, { type: 'html', @@ -949,63 +951,63 @@ const sidebarSettings = { }, ], apis: [ - "docs/dbt-cloud-apis/overview", - { - type: "category", - label: "API Access", - link: { type: "doc", id: "docs/dbt-cloud-apis/authentication" }, - items: [ - "docs/dbt-cloud-apis/authentication", - "docs/dbt-cloud-apis/user-tokens", - "docs/dbt-cloud-apis/service-tokens", - ], - }, - { - type: "category", - label: "Administrative API", - link: { type: "doc", id: "docs/dbt-cloud-apis/admin-cloud-api" }, - items: [ - "docs/dbt-cloud-apis/admin-cloud-api", - { - type: "link", - label: "API v2", - href: "/dbt-cloud/api-v2", - }, - { - type: "link", - label: "API v3", - href: "/dbt-cloud/api-v3", + "docs/dbt-cloud-apis/overview", + { + type: "category", + label: "API Access", + link: { type: "doc", id: "docs/dbt-cloud-apis/authentication" }, + items: [ + "docs/dbt-cloud-apis/authentication", + "docs/dbt-cloud-apis/user-tokens", + "docs/dbt-cloud-apis/service-tokens", + ], + }, + { + type: "category", + label: "Administrative API", + link: { type: "doc", id: "docs/dbt-cloud-apis/admin-cloud-api" }, + items: [ + "docs/dbt-cloud-apis/admin-cloud-api", + { + type: "link", + label: "API v2", + href: "/dbt-cloud/api-v2", + }, + { + type: "link", + label: "API v3", + href: "/dbt-cloud/api-v3", + }, + ], + }, + { + type: "category", + label: "Discovery API", + link: { type: "doc", id: "docs/dbt-cloud-apis/discovery-api" }, + items: [ + "docs/dbt-cloud-apis/discovery-api", + "docs/dbt-cloud-apis/discovery-use-cases-and-examples", + "docs/dbt-cloud-apis/project-state", + "docs/dbt-cloud-apis/discovery-querying", + { + type: "category", + label: "Schema", + link: { + type: "doc", + id: "docs/dbt-cloud-apis/discovery-schema-about", }, - ], - }, - { - type: "category", - label: "Discovery API", - link: { type: "doc", id: "docs/dbt-cloud-apis/discovery-api" }, - items: [ - "docs/dbt-cloud-apis/discovery-api", - "docs/dbt-cloud-apis/discovery-use-cases-and-examples", - "docs/dbt-cloud-apis/project-state", - "docs/dbt-cloud-apis/discovery-querying", - { - type: "category", - label: "Schema", - link: { - type: "doc", - id: "docs/dbt-cloud-apis/discovery-schema-about", - }, - items: [ - "docs/dbt-cloud-apis/discovery-schema-about", - { - type: "category", - label: "Environment (recommended)", - link: { - type: "doc", - id: "docs/dbt-cloud-apis/discovery-schema-environment", - }, - items: [ - "docs/dbt-cloud-apis/discovery-schema-environment", - { + items: [ + "docs/dbt-cloud-apis/discovery-schema-about", + { + type: "category", + label: "Environment (recommended)", + link: { + type: "doc", + id: "docs/dbt-cloud-apis/discovery-schema-environment", + }, + items: [ + "docs/dbt-cloud-apis/discovery-schema-environment", + { type: "category", label: "Applied", link: { @@ -1029,57 +1031,57 @@ const sidebarSettings = { "docs/dbt-cloud-apis/discovery-schema-environment-applied-lineage", ], }, - "docs/dbt-cloud-apis/discovery-schema-environment-definition", - ], - }, - // Uncomment to add Definition subpage, but need to make items non-empty - // { - // type: "category", - // label: "Definition", - // items: [ - // // insert pages here - // ], - // }, - { - type: "category", - label: "Job", - link: { - type: "doc", - id: "docs/dbt-cloud-apis/discovery-schema-job", - }, - items: [ - "docs/dbt-cloud-apis/discovery-schema-job", - "docs/dbt-cloud-apis/discovery-schema-job-model", - "docs/dbt-cloud-apis/discovery-schema-job-models", - "docs/dbt-cloud-apis/discovery-schema-job-source", - "docs/dbt-cloud-apis/discovery-schema-job-sources", - "docs/dbt-cloud-apis/discovery-schema-job-seed", - "docs/dbt-cloud-apis/discovery-schema-job-seeds", - // "docs/dbt-cloud-apis/discovery-schema-job-snapshot", - "docs/dbt-cloud-apis/discovery-schema-job-snapshots", - "docs/dbt-cloud-apis/discovery-schema-job-test", - "docs/dbt-cloud-apis/discovery-schema-job-tests", - "docs/dbt-cloud-apis/discovery-schema-job-exposure", - "docs/dbt-cloud-apis/discovery-schema-job-exposures", - // "docs/dbt-cloud-apis/discovery-schema-job-macro", - // "docs/dbt-cloud-apis/discovery-schema-job-macros", - ], + "docs/dbt-cloud-apis/discovery-schema-environment-definition", + ], + }, + // Uncomment to add Definition subpage, but need to make items non-empty + // { + // type: "category", + // label: "Definition", + // items: [ + // // insert pages here + // ], + // }, + { + type: "category", + label: "Job", + link: { + type: "doc", + id: "docs/dbt-cloud-apis/discovery-schema-job", }, - ], - }, - ], - }, - { - type: "category", - label: "Semantic Layer APIs", - link: { type: "doc", id: "docs/dbt-cloud-apis/sl-api-overview" }, - items: [ - "docs/dbt-cloud-apis/sl-api-overview", - "docs/dbt-cloud-apis/sl-graphql", - "docs/dbt-cloud-apis/sl-jdbc", - "docs/dbt-cloud-apis/sl-python", - ], - }, + items: [ + "docs/dbt-cloud-apis/discovery-schema-job", + "docs/dbt-cloud-apis/discovery-schema-job-model", + "docs/dbt-cloud-apis/discovery-schema-job-models", + "docs/dbt-cloud-apis/discovery-schema-job-source", + "docs/dbt-cloud-apis/discovery-schema-job-sources", + "docs/dbt-cloud-apis/discovery-schema-job-seed", + "docs/dbt-cloud-apis/discovery-schema-job-seeds", + // "docs/dbt-cloud-apis/discovery-schema-job-snapshot", + "docs/dbt-cloud-apis/discovery-schema-job-snapshots", + "docs/dbt-cloud-apis/discovery-schema-job-test", + "docs/dbt-cloud-apis/discovery-schema-job-tests", + "docs/dbt-cloud-apis/discovery-schema-job-exposure", + "docs/dbt-cloud-apis/discovery-schema-job-exposures", + // "docs/dbt-cloud-apis/discovery-schema-job-macro", + // "docs/dbt-cloud-apis/discovery-schema-job-macros", + ], + }, + ], + }, + ], + }, + { + type: "category", + label: "Semantic Layer APIs", + link: { type: "doc", id: "docs/dbt-cloud-apis/sl-api-overview" }, + items: [ + "docs/dbt-cloud-apis/sl-api-overview", + "docs/dbt-cloud-apis/sl-graphql", + "docs/dbt-cloud-apis/sl-jdbc", + "docs/dbt-cloud-apis/sl-python", + ], + }, ], reference: [ "reference/references-overview", @@ -1635,7 +1637,7 @@ const sidebarSettings = { "best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for", "best-practices/how-we-mesh/mesh-3-structures", "best-practices/how-we-mesh/mesh-4-implementation", - "best-practices/how-we-mesh/mesh-6-coordinate-versions", + "best-practices/how-we-mesh/mesh-6-coordinate-versions", "best-practices/how-we-mesh/mesh-5-faqs", ], }, diff --git a/website/snippets/_private-connectivity-matrix.md b/website/snippets/_private-connectivity-matrix.md index 9618c6feb6f..9382188c55a 100644 --- a/website/snippets/_private-connectivity-matrix.md +++ b/website/snippets/_private-connectivity-matrix.md @@ -1,31 +1,300 @@ +import Lifecycle from '/src/components/lifeCycle'; + ## Private connectivity feature matrix -The following feature chart outlines the availability of private connectivity features across [multi-tenant (MT) and single-tenant (ST)](/docs/cloud/about-cloud/tenancy) instances

-✅ = Available | ❌ = Not currently supported | \- = Not applicable
- -| Connectivity type | AWS MT | AWS ST | Azure MT | Azure ST | GCP MT | -|:--------------------------------------------------|:------:|:------:|:--------:|:--------:|:--------:| -| INGRESS (to ) | | | | | | -| Private Ingress | ❌ | ✅ | ❌ | ✅ | ❌ | -| Dual Ingress | ❌ | ✅ | ❌ | ❌ | ❌ | -| EGRESS - DW (from ) | | | | | | -| Snowflake | ✅ | ✅ | ✅ | ✅ | ✅ | -| - Snowflake Internal Stage | ✅ | ✅ | ✅ | ✅ | ❌ | -| Databricks | ✅ | ✅ | ✅ | ✅ | ❌ | -| Postgres (via load balancer) | ✅ | ✅ | ✅ | ✅ | ❌ | -| Azure Database for PostgreSQL Flexible Server | - | - | ✅ | ✅ | - | -| Redshift (Interface) | ✅ | ✅ | - | - | - | -| Redshift (Managed) | ✅ | ✅ | - | - | - | -| Redshift Severless (Interface) | ✅ | ✅ | - | - | - | -| Redshift Serverless (Managed) | ✅ | ✅ | - | - | - | -| Amazon Athena w/ AWS Glue | ❌ | ✅ | - | - | - | -| Azure Synapse | - | - | ✅ | ✅ | - | -| Azure Fabric (cross-tenant not supported by Azure)| - | - | ❌ | ❌ | - | -| Google BigQuery | - | - | - | - | ✅ | -| Teradata - Database Server | ✅ | ✅ | ✅ | ✅ | ❌ | -| EGRESS - VCS (from ) | | | | | | -| GitHub Enteprise Server | ✅ | ✅ | ✅ | ✅ | ❌ | -| GitLab Enterprise | ✅ | ✅ | ✅ | ✅ | ❌ | -| BitBucket | ✅ | ✅ | ✅ | ✅ | ❌ | -| AWS CodeCommit | ❌ | ✅ | - | - | - | -| Azure DevOps Repos (not supported by Azure) | - | - | ❌ | ❌ | - | +The following charts outline private connectivity options across [multi-tenant (MT) and single-tenant (ST)](/docs/cloud/about-cloud/tenancy) deployments. + +**Legend:** +- ✅ = Available +- ❌ = Not currently available +- \- = Not applicable +- \* = Shared endpoint (all others are dedicated) +- = Reported working but not yet directly tested by dbt + +:::note What "Available" means +Availability indicates whether a private endpoint can be established at the network layer. dbt evaluates common configurations, authentication methods, and integration patterns when determining support. However, due to the wide range of customizations possible in customer environments, not every configuration may be covered. If you have questions about a specific use case, [contact dbt Support](/community/resources/getting-help#dbt-cloud-support). +::: + +--- + +### Connecting to dbt Cloud + +Your services can connect to over private connectivity. This is available on Single-Tenant deployments only. All connections to dbt Cloud use the dbt-provisioned model. + + + + + + + + + + + + + + + + + + + + + +
Connectivity typeAWS STAzure ST
Private access✅✅
Dual access (public + private)✅❌
+ +--- + +### Connecting dbt Cloud to data platforms + + can establish private connections to your data platforms. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ServiceAWS MTAWS STAzure MTAzure STGCP MTProvisioning
Snowflake✅✅✅✅✅Vendor
  Snowflake Internal Stage✅✅✅✅❌Vendor
Databricks✅✅✅✅❌Vendor
Redshift✅✅---Native
Redshift Serverless✅✅---Native
Amazon Athena w/ AWS Glue*❌✅---Native
Azure Database for PostgreSQL Flexible Server--✅✅-Native
Azure Synapse--✅✅-Native
Azure Fabric--❌❌--
Google BigQuery*----✅Native
Teradata VantageCloud✅✅✅✅✅Vendor
+ +--- + +### Connecting dbt Cloud to self-hosted services + + can establish private connections to your self-hosted services. All self-hosted connections use the customer-provisioned model. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ServiceAWS MTAWS STAzure MTAzure STGCP MT
GitHub Enterprise Server✅✅✅✅❌
GitLab Self-Managed✅✅✅✅❌
Bitbucket Data Center✅✅✅✅❌
Azure DevOps Server✅ ✅ ✅✅✅
AWS CodeCommit❌✅---
Postgres✅✅✅✅✅
Spark✅✅---
Starburst / Trino✅✅✅✅✅
Teradata (self-hosted)✅✅✅✅✅
+ +For services not explicitly listed above, you can establish private connectivity using the same customer-provisioned approach. This model supports any service that can be placed behind a load balancer and exposed via your cloud platform's private connectivity technology. + +To inquire about private connectivity to additional platforms, contact your account team. + +**Prerequisites by cloud platform:** + + + + + + + + + + + + + + + + + + + + + + + + + + +
CloudLoad balancer requirementResource you create
AWSNetwork Load BalancerVPC Endpoint Service
AzureStandard Load BalancerPrivate Link Service
GCPInternal Proxy Load BalancerService Attachment
+ +Once you create the private connectivity resource, share the resource ID (endpoint service name, alias, or service attachment URI) with dbt to establish the connection. + +**Setup guides:** +- [AWS PrivateLink for self-hosted services](/docs/cloud/secure/vcs-privatelink) +- [Azure Private Link for self-hosted services](/docs/cloud/secure/az-self-hosted-private-link) +- [GCP Private Service Connect for self-hosted services](/docs/cloud/secure/gcp-self-hosted-psc) + +If you have questions about whether your configuration is supported, [contact dbt Support](/community/resources/getting-help#dbt-cloud-support). diff --git a/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/alias-info.png b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/alias-info.png new file mode 100644 index 00000000000..620f52b9cd8 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/alias-info.png differ diff --git a/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/nat-subnet-creation.png b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/nat-subnet-creation.png new file mode 100644 index 00000000000..e6f5998256b Binary files /dev/null and b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/nat-subnet-creation.png differ diff --git a/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/privatelink-service-creation.png b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/privatelink-service-creation.png new file mode 100644 index 00000000000..e58a3124c52 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/privatelink-service-creation.png differ diff --git a/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/scope-of-guide.png b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/scope-of-guide.png new file mode 100644 index 00000000000..f84a56a996b Binary files /dev/null and b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/scope-of-guide.png differ diff --git a/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/vnet-search.png b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/vnet-search.png new file mode 100644 index 00000000000..13c11e13987 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/az-self-hosted-privatelink/vnet-search.png differ diff --git a/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/scope-of-guide.png b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/scope-of-guide.png new file mode 100644 index 00000000000..2f028ea6255 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/scope-of-guide.png differ diff --git a/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-creation.png b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-creation.png new file mode 100644 index 00000000000..d712c53ea0b Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-creation.png differ diff --git a/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-details.png b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-details.png new file mode 100644 index 00000000000..6696a4f92ab Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-details.png differ diff --git a/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-subnet-creation.png b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-subnet-creation.png new file mode 100644 index 00000000000..fbe5fdc06ec Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gcp-self-hosted-psc/service-attach-subnet-creation.png differ