From 8cb9fe0701d58ee4baed47acbe292f8f09131422 Mon Sep 17 00:00:00 2001 From: pwseg Date: Fri, 16 Aug 2024 10:49:04 -0400 Subject: [PATCH 1/3] Git sync updates --- src/segment-app/extensions/dbt.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/src/segment-app/extensions/dbt.md b/src/segment-app/extensions/dbt.md index 079ea04db3..45dca99bb4 100644 --- a/src/segment-app/extensions/dbt.md +++ b/src/segment-app/extensions/dbt.md @@ -6,29 +6,28 @@ Segment's dbt extension lets you use [Reverse ETL](/docs/connections/reverse-etl With Segment's dbt extension, you can: -- Securely connect Segment to the GitHub repository that stores your dbt models. +- Securely connect Segment to a Git repository that stores your dbt models. - Use centralized dbt models to set up Reverse ETL. - Trigger Reverse ETL syncs from dbt jobs. This page explains how to set up a dbt Model and then use the model with Reverse ETL. -> info "" -> Extensions, including dbt sync, is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. - ## Before you begin Keep the following in mind as you set up the dbt extension: - The extension supports [dbt Core v1.7](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.7){:target="_blank"}. - You can use [Snowflake](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/snowflake-setup/), [Databricks](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/databricks-setup/), [Redshift](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/redshift-setup/), [Postgres](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/postgres-setup/), and [BigQuery](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup/) as Reverse ETL sources. -- dbt models aren't synchronized from the dbt cloud. The model sync connects to a Git repository that loads models into Segment for use with Reverse ETL. +- dbt models aren't synchronized from the dbt cloud. The model sync connects to a Git repository that loads models into Segment for use with Reverse ETL. +- You can connect to GitHub using a GitHub App, token, or SSH. +- For [GitLab](https://docs.gitlab.com/ee/user/ssh.html){:target="_blank"} and [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/){:target="_blank"}, use SSH to connect. ## Set up Git dbt Models and dbt Cloud To set up the dbt extension, you'll need: - an existing dbt account with a Git repository -- (for job syncs:) dbt cloud with jobs already created +- for job syncs, dbt cloud with jobs already created ### Git repository and dbt Models setup @@ -36,8 +35,8 @@ Follow these steps to connect the Git repository that stores your dbt Models: 1. In your Segment workspace, navigate to **Settings > Extensions**. 2. Click **Set up Git sync**. -3. On the **Configure service credentials** page, select a service and protocol, add your SSH private key or GitHub token, then click **Next**. -4. In the **Connect source** window, select a Reverse ETL warehouse source from the dropdown, then click **Save**. +3. On the **Configure service credentials** page, select a service and protocol, add your GitHub App, SSH private key or GitHub token, then click **Next**. +4. In the **Connect source** window, select an existing Reverse ETL warehouse source from the dropdown, then click **Save**. After you've saved your setup, you can configure your Git repository's settings to your needs by changing the repository, branch, dbt version, default schema, and project path. From 2a92eda28fe87c3ec94e8faa1ba3a9c672b7038f Mon Sep 17 00:00:00 2001 From: pwseg Date: Fri, 16 Aug 2024 11:50:20 -0400 Subject: [PATCH 2/3] Git Sync updates --- src/segment-app/extensions/git.md | 74 ++++++++++++++++++++++++++----- 1 file changed, 64 insertions(+), 10 deletions(-) diff --git a/src/segment-app/extensions/git.md b/src/segment-app/extensions/git.md index 6955b88bd0..37a6b76ba5 100644 --- a/src/segment-app/extensions/git.md +++ b/src/segment-app/extensions/git.md @@ -2,23 +2,77 @@ title: Git Sync Extension --- -Segment's Git extension lets you manage versioning by syncing changes you make in Sources and Destinations from your Segment workspace to a Git repository. +Segment's Git extension lets you manage versioning by syncing changes you make in your Segment workspace to a Git repository. -> info "" -> Extensions, including Git sync, is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. +Git Sync supports one-way synchronization from Segment to Git. This sync captures the current state of your workspace through a full sync and includes all new records and changes for supported resources. -## Set up Git sync +Segment doesn't support syncing changes from Git back to Segment. -Follow these steps to set up Git sync: +## Set up Git Sync + +Follow these steps to set up Git Sync: 1. In your Segment workspace, navigate to **Settings > Extensions**. 2. Click **Set up Git sync**. -3. On the **Configure service credentials** page, select a service and protocol, add your SSH private key or GitHub token, then click **Next**. +3. On the **Configure service credentials** page, select a service and protocol, add your GitHub App, SSH private key, or GitHub token, then click **Next**. + - To connect to GitLab or Bitbucket, use your SSH private key. + +## Working with Git Sync + +The Git sync extension syncs the following resources from Segment to your Git repository: + +- [Sources](/docs/connections/sources/) and [Destinations](/docs/connections/destinations/)) +- [Warehouses](/docs/connections/storage/warehouses/) +- [Destination Filters and Mappings](/docs/connections/destinations/destination-filters/) +- [Tracking Plans](/docs/protocols/tracking-plan/create/) +- [Functions](/docs/connections/functions/) +- [Transformations](/docs/protocols/transform/) +- [Reverse ETL](/docs/connections/reverse-etl/) +- [Users](/docs/segment-app/iam/concepts/#team-members) and [User groups](/docs/segment-app/iam/concepts/#user-groups) +- [Labels](/docs/segment-app/iam/labels/#where-can-i-create-labels) + +Reach out to [Segment support](https://app.segment.com/workspaces?contact=1){:target="blank"} to request support for additional Git Sync resources. + +After you set up the Git sync extension for the first time, Segment performs an initial sync that sends the current state of your Segment workspace to the Git repository you connected. Segment automatically tracks all following workspace updates. + +You can manually trigger syncs at any time by clicking **Full Sync** on the Git Sync page. To disable Git Sync from the Git Sync page, switch the **Enabled** toggle to off. + +## Git Sync architecture and data model + +Because a Segment workspace can represent a distinct environment (testing, staging, production), each workspace is mapped directly to a single Git repository. This direct mapping ensures a clear and organized relationship between workspace resources and a Git repository. + +Segment uses its [Terraform provider](https://registry.terraform.io/providers/segmentio/segment/1.0.3){:target="_blank"} to manage key functions like tracking changes and retrieving information about those changes in Segment. Segment stores changes in HashiCorp Configuration Language (HCL), the format used by Terraform. To learn more about HCL and how it compares to JSON or YAML, visit [HashiCorp's HCL repository on GitHub](https://github.com/hashicorp/hcl){:target="_blank"}. + +Using HCL makes it easier to document Segment's data model, especially for users managing versioning and Git Sync with Terraform. It also helps manage Segment configurations directly from Git. For more details on the Git Sync data model, read [Segment's Terraform provider documentation](https://registry.terraform.io/providers/segmentio/segment/latest/docs){:target="_blank"}. + +## Managing your Segment workspace with Terraform and Git Sync + +Segment supports one-way synchronization from Segment to Git, but you can set up two-way synchronization using the Segment Terraform provider. + +Terraform offers an open-source way to manage Segment resources through a Git repository as an alternative to a fully managed two-way sync. However, this method requires third-party tools like [Atlantis](https://www.runatlantis.io/){:target="_blank"} for CI integration, which Segment doesn’t officially support. + +To manage Segment resources using Git and Terraform, follow these steps: + +1. Copy the generated Terraform configuration for the resources you want to manage into a separate Git repository dedicated to Terraform. +2. Include the following provider configuration blocks: + + ```hcl + # providers.tf -## Working with Git syncs + terraform { + required_providers { + segment = { + source = "segmentio/segment" + version = "1.0.4" + } + } + } -The Git sync extension syncs [Connections](/docs/connections/) ([Sources](/docs/connections/sources/) and [Destinations](/docs/connections/destinations/)) from Segment to your Git repository. + provider "segment" { + # Provide the token directly or load it from an environment variable + } + ``` +3. Apply configuration changes by running Terraform locally or using a tool like Atlantis to run it directly from your Git provider. -After you set up the Git sync extension for the first time, Segment performs an initial sync that sends source and destination information in your Segment workspace to the Git repository you connected. -You can run syncs at any time by clicking **Full resync** on the Git sync page. To disable Git sync from the Git sync page, switch the **Enabled** toggle to off. \ No newline at end of file +For more information on using Terraform, visit [Terraform's documentation](https://developer.hashicorp.com/terraform/docs){:target="_blank"}. From 4de9e1cfe193ef55761a88ba7c7d4a98fe00c84b Mon Sep 17 00:00:00 2001 From: pwseg Date: Fri, 16 Aug 2024 12:13:39 -0400 Subject: [PATCH 3/3] final touches --- src/segment-app/extensions/git.md | 2 +- src/segment-app/extensions/index.md | 11 +++++------ 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/src/segment-app/extensions/git.md b/src/segment-app/extensions/git.md index 37a6b76ba5..7a2f920a7f 100644 --- a/src/segment-app/extensions/git.md +++ b/src/segment-app/extensions/git.md @@ -21,7 +21,7 @@ Follow these steps to set up Git Sync: The Git sync extension syncs the following resources from Segment to your Git repository: -- [Sources](/docs/connections/sources/) and [Destinations](/docs/connections/destinations/)) +- [Sources](/docs/connections/sources/) and [Destinations](/docs/connections/destinations/) - [Warehouses](/docs/connections/storage/warehouses/) - [Destination Filters and Mappings](/docs/connections/destinations/destination-filters/) - [Tracking Plans](/docs/protocols/tracking-plan/create/) diff --git a/src/segment-app/extensions/index.md b/src/segment-app/extensions/index.md index 1948727b45..5fd32c487d 100644 --- a/src/segment-app/extensions/index.md +++ b/src/segment-app/extensions/index.md @@ -2,12 +2,11 @@ title: Extensions --- -Extensions integrate third-party tools into your existing Segment workspace to help you automate tasks. +Extensions let you integrate third-party tools into your existing Segment workspace, helping you automate tasks, manage data flows, and maintain version control. -> info "" -> Extensions is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. During Public Beta, Extensions is available for Team and Developer plans only. [Reach out to Segment](mailto:friends@segment.com) if you're on a Business Tier plan and would like to participate in the Public Beta. +Segment offers the following extensions: -During public beta, Segment offers the following extensions: +- [dbt models and dbt Cloud](/docs/segment-app/extensions/dbt): Sync your dbt Labs models with Segment to streamline model management, versioning, and CI checks. This extension lets you securely connect Segment to a Git repository, making it easier to integrate and manage dbt models across different environments like testing, staging, and production. +- [Git Sync](/docs/segment-app/extensions/git): Manage versioning and track changes by syncing your Segment workspace a Git repository. The Git Sync extension helps maintain a clear and organized relationship between your workspace and its corresponding Git repository, ensuring that your resources are consistently managed and versioned across your environments. -- [dbt models and dbt Cloud](/docs/segment-app/extensions/dbt), which lets you sync models with dbt Labs. -- [Git sync](/docs/segment-app/extensions/git), which you can use to manage versioning and track changes in your Segment workspace. +Segment built Extensions to help you get the most out of your Segment workspace, allowing you to keep your projects organized, efficient, and aligned with best practices for data management and version control.