Skip to content

Commit 93f8208

Browse files
stayseesongstayseesong
committed
first draft [netlify-build]
1 parent a6b4083 commit 93f8208

File tree

5 files changed

+250
-1
lines changed

5 files changed

+250
-1
lines changed

‎src/_data/sidenav/main.yml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,14 @@ sections:
241241
title: Rate Limits
242242
- path: /connections/regional-segment
243243
title: Regional Segment
244+
- section_title: Reverse ETL
245+
section:
246+
- path: /reverse-etl/reverse-etl
247+
title: Reverse ETL Overview
248+
- path: /reverse-etl/bigquery-setup
249+
title: Bigquery Reverse ETL Setup
250+
- path: /reverse-etl/snowflake-setup
251+
title: Snowflake Reverse ETL Setup
244252
- section_title: Profiles
245253
section:
246254
- path: /profiles
@@ -397,7 +405,7 @@ sections:
397405
- path: /privacy/account-deletion
398406
title: Account & Data Deletion
399407
- path: /privacy/hipaa-eligible-segment
400-
title: HIPAA Eligible Segment
408+
title: HIPAA Eligible Segment
401409
- path: /privacy/faq
402410
title: Privacy FAQs
403411
- section_title: Protocols

‎src/_includes/icons/reverse-etl.svg

Lines changed: 1 addition & 0 deletions
Loading

‎src/reverse-etl/bigquery-setup.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
title: BigQuery Reverse ETL Setup
3+
---
4+
5+
Set up BigQuery as your Reverse ETL source. You can also choose to [set up Snowflake](/docs/reverse-etl/snowflake-setup/) as your source.
6+
7+
> warning ""
8+
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
9+
10+
To set up the Segment BigQuery connector:
11+
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
12+
2. Click **+ Create Service Account** to create a new service account
13+
3. Enter your **Service account name** and a description of what the account will do
14+
4. Click **Create and Continue**.
15+
5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add.
16+
6. Click **+ Add another role** and add the *BigQuery Job User* role.
17+
7. Click **Continue**.
18+
8. Click **Done**.
19+
9. Search for the service account you just created.
20+
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
21+
11. Click **Add Key > Create new key**.
22+
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
23+
13. Copy all the content within the file you just created and downloaded.
24+
14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section.
25+
19. Enter your **Data Location**.
26+
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
27+
6. Click **Create Source** if the test connection is successful.
28+
29+
Once you've added BigQuery as a source, you can [add a model](/docs/reverse-etl/reverse-etl/#step-2-add-a-model).
30+
31+
## Constructing your own role or policy
32+
When you construct your own role or policy, Segment needs the following permissions:
33+
34+
Permission | Details
35+
---------- | --------
36+
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
37+
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists
38+
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets/tables your model query references, in addition to managing our tables for tracking state
39+
40+
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. If you don't wish to grant `bigquery.datasets.create` access, you may create this dataset yourself, but Segment still needs `bigquery.datasets.get` access.

‎src/reverse-etl/reverse-etl.md

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
title: Reverse ETL
3+
beta: true
4+
---
5+
6+
> info ""
7+
> Reverse ETL is currently in beta and Segment is actively working on this feature. Segment's [First-Access and Beta terms](https://segment.com/legal/first-access-beta-preview/) govern this feature. If you’d like to join the beta, please reach out to your CSM, AE, or SE to learn more.
8+
9+
Reverse ETL (Extract, Transform, Load) is a directional version of ETL that extracts data from a data warehouse and loads it into a 3rd party destination. Reverse ETL allows you to connect your data warehouse to the tools that Marketing, Sales, Support, Product, Analytics, and other business teams use. For example, with Reverse ETL, you can sync rows from Snowflake to Salesforce. This enables your warehouse to remain the single source of truth in which all data is collected, operated on, and enriched.
10+
11+
As Segment is actively developing this feature, Segment welcomes your feedback on your experience with Reverse ETL. Click the button below to submit your feedback.
12+
13+
{% include components/button-fill.html modifier="expand" text="Submit feedback" href=" https://airtable.com/shrdu5cW06LsuZvXN" %}
14+
15+
## Use cases
16+
Use Reverse ETL when you want to:
17+
* Sync lead scores created in the warehouse to Salesforce
18+
* Connect Google Sheets to a view in the warehouse for up-to-date reports that can also be downloaded as CSVs.
19+
* Sync enriched data to Amplitude or Mixpanel for a more complete view than what Segment events provides.
20+
* Sync audiences and other data built in the warehouse to Braze, Iterable, or Hubspot or Salesforce Marketing Cloud for marketing campaigns.
21+
* Turn data in the warehouse from sources that can’t be instrumented by Segment back into Segment as events for Personas and other use cases.
22+
* Pass offline or enriched data to conversion APIs like Facebook, Google Ads, TikTok, Snapchat.
23+
24+
## Getting started
25+
There are four components to Reverse ETL: Sources, Models, Destinations, and Mappings.
26+
27+
![image from Charmaine to go here]
28+
29+
Follow these 4 steps to set up Reverse ETL and learn what each component is about:
30+
1. [Add a Source](#step-1-add-a-source)
31+
2. [Add a Model](#step-2-add-a-model)
32+
3. [Add a Destination](#step-3-add-a-destination)
33+
4. [Create Mappings](#step-4-create-mappings)
34+
35+
### Step 1: Add a Source
36+
A Source is where your data originates from. Traditionally in Segment, a [Source](/docs/connections/sources/#what-is-a-source) is a website, server library, mobile SDK, or cloud application which can send data into Segment. In Reverse ETL, your data warehouse acts as the Source.
37+
38+
> info ""
39+
> Reverse ETL currently only supports BigQuery and Snowflake as sources.
40+
41+
To add your warehouse as a source:
42+
43+
> warning ""
44+
> You need to be a user that has both read and write access to the warehouse.
45+
46+
1. Navigate to **Reverse ETL** in the Segment app.
47+
2. Click **Add Source**.
48+
3. Select the source you want to add. You can choose between **BigQuery** and **Snowflake**.
49+
* If you choose to use Snowflake, run the queries listed in the [Snowflake setup guide](/docs/reverse-etl/snowflake-setup/) to set up the Segment Snowflake connector. Segment recommends using the `ACCOUNTADMIN` role to execute all the commands.
50+
* If you choose to use BigQuery, use the permissions outlined in the [BigQuery setup guide](/docs/reverse-etl/bigquery-setup/), to create a Service Account and generate JSON credentials that will then be copied into the Segment UI when creating a Reverse ETL Source.
51+
4. Add in the account information for your source.
52+
5. Click **Test Connection** to test to see if the connection works.
53+
6. Click **Create Source** if the test connection is successful.
54+
55+
After you add your data warehouse as a source, you can [add a model](#step-2-add-a-model) to your source.
56+
57+
### Step 2: Add a Model
58+
Models are SQL queries that define sets of data you want to synchronize to your Reverse ETL destinations. After you add your source, you can add a model.
59+
60+
To add a model:
61+
1. Navigate to **Reverse ETL > Sources** and click **Add Model**.
62+
2. Click **SQL Editor** as your modeling method. (Note: Segment will add more modeling methods in the future)
63+
3. Enter the SQL query that’ll define your model. Your model is used to map data to your Reverse ETL destinations.
64+
4. Choose a column to use as the unique identifier for each row in the **Unique Identifier column** field. This column is used to detect new, updated, and deleted rows.
65+
5. Click **Preview** to see a preview of the results of your SQL query. The data from the preview is extracted from the first 10 rows of your warehouse.
66+
6. Click **Next**.
67+
7. Enter your **Model Name**.
68+
8. Select the Schedule type for the times you want the model’s data to be extracted from your warehouse. You can choose from:
69+
* **Interval**: Extractions perform based on a selected time cycle.
70+
* **Day and time**: Extractions perform at specific times on selected days of the week.
71+
9. Select how often you want the schedule to run in **Schedule configuration**.
72+
* For an **Interval** schedule type, you can choose from: 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours, 6 hours, 8 hours, 12 hours, 1 day.
73+
* 15 minutes is considered real-time for warehouse syncs
74+
* For a **Day and time** schedule type, you can choose the day(s) you’d like the schedule to run as well as the time. You can only choose to run the extraction at the top of the hour.
75+
10. Click **Create Model**.
76+
77+
### Step 3: Add a Destination
78+
Once you’ve added a model, you need to add a destination. In Reverse ETL, destinations are the business tools or apps you use that Segment syncs the data from your warehouse to.
79+
80+
> info ""
81+
> Reverse ETL only supports [Actions destinations](/docs/connections/destinations/actions/). Depending on the destination, you may need to know certain endpoints and have specific credentials to add the destination.
82+
83+
To add a destination:
84+
1. Navigate to **Reverse ETL > Destinations**.
85+
2. Click **Add Destination**.
86+
3. Select the destination you want to connect to.
87+
4. Select the source you want to connect the destination to.
88+
5. Enter the **Destination name** and click **Create Destination**.
89+
6. Enter the required information on the **Settings** tab of the destination.
90+
91+
### Step 4: Create Mappings
92+
After you’ve added a destination, you can create mappings from your warehouse to the destination. Mappings enable you to map the data you extract from your warehouse to the fields in your destination.
93+
94+
To create a mapping:
95+
1. Go to the **Mappings** tab of the destination and click **Add Mapping**.
96+
2. Select the model to sync from.
97+
3. Select the **Action** you want to sync and click **Next**.
98+
* Actions determine the information sent to the destination. You map the fields that come from your source, to fields that the destination expects to find. Fields on the destination side depend on the type of action selected.
99+
4. In the **Select record to map and send** section, select which records to send to your destination after Segment completes extracting data based on your model. You can choose from:
100+
* Added records
101+
* Updated records
102+
* Added or updated records
103+
* Deleted records
104+
5. Select a test record to preview the fields that you can map to your destination in the **Add test record** field.
105+
6. Define how to map the record columns from your model to your destination in the **Select Mappings** section.
106+
7. Click **Create Mapping**.
107+
8. Select the destination you’d like to enable on the **My Destinations** page under **Reverse ETL > Destinations**.
108+
9. Turn the toggle on for the **Mapping State** to enable the destination. Events that match the trigger condition in the mapping will be sent to the destination.
109+
* If you disable the mapping state to the destination, events that match the trigger condition in the mapping won’t be sent to the destination.
110+
111+
## Using Reverse ETL
112+
After you've followed [all four steps](/docs/reverse-etl/reverse-etl/#getting-started) and set up your source, model, destination, and mappings for Reverse ETL, your data will extract and will sync to your destination(s) right away if you chose an interval schedule. If you set your data to extract at a specific day and time, the extraction will take place then.
113+
114+
### Run statuses
115+
You can check the status of your data extractions and see details of your runs.
116+
117+
To check the status of your extractions:
118+
1. Navigate to **Reverse ETL > Destinations**.
119+
2. Select the destination you want to view.
120+
3. Select the mapping you want to view.
121+
4. Click the run you want to view to get details of the run. You can view:
122+
* The status of the run
123+
* How long it took for the run to complete
124+
* The load results - how many successful records were synced as well as how many records were updated, deleted, or are new.
125+
126+
127+
### Edit your model
128+
129+
To edit your model:
130+
1. Navigate to **Reverse ETL > Sources**.
131+
2. Select the source with the model you want to edit.
132+
3. On the overview tab, click **Edit** to edit your query.
133+
4. Click the **Settings** tab to edit the model name or change the schedule settings.
134+
135+
### Edit your mapping
136+
137+
To edit your mapping:
138+
1. Navigate to **Reverse ETL > Destinations**.
139+
2. Select the destination with the mapping you want to edit.
140+
3. Select the **...*** three dots and click **Edit mapping**. If you want to delete your mapping, select **Delete**.

‎src/reverse-etl/snowflake-setup.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: Snowflake Reverse ETL Setup
3+
beta: true
4+
---
5+
6+
Set up Snowflake as your Reverse ETL source. You can also choose to [set up BigQuery](/docs/reverse-etl/bigquery-setup/) as your source.
7+
8+
At a high level, when you set up Snowflake for Reverse ETL, the configured user/role needs read permissions for any resources (databases, schemas, tables) the query needs to access. Segment keeps track of changes to your query results via a managed schema (`__SEGMENT_REVERSE_ETL`), which requires the configured user to allow write permissions for that schema.
9+
10+
## Set up guide
11+
Follow the instructions below to set up the Segment Snowflake connector. Segment recommends you use the `ACCOUNTADMIN` role to execute all the commands below.
12+
13+
1. Log in to your Snowflake account.
14+
2. Navigate to *Worksheets*.
15+
3. Enter and run the code below to create a database.
16+
Segment uses the database specified in your connection settings to create a schema called `__segment_reverse_etl` to avoid collision with your data. The schema is used for tracking changes to your model query results between syncs.
17+
An existing database can be reused, if desired. Segment recommends you to use the same database across all your models attached to this source to keep all the state tracking tables in 1 place.
18+
19+
```sql
20+
-- not required if another database is being reused
21+
CREATE DATABASE segment_reverse_etl;
22+
```
23+
4. Enter and run the code below to create a virtual warehouse.
24+
Segment Reverse ETL needs to execute queries on your Snowflake account, which requires a Virtual Warehouse to handle the compute. You can also reuse an existing warehouse.
25+
26+
```sql
27+
-- not required if reusing another warehouse
28+
CREATE WAREHOUSE segment_reverse_etl
29+
WITH WAREHOUSE_SIZE = 'XSMALL'
30+
WAREHOUSE_TYPE = 'STANDARD'
31+
AUTO_SUSPEND = 600 -- 5 minutes
32+
AUTO_RESUME = TRUE;
33+
```
34+
5. Enter and run the code below to create specific roles for Reverse ETL.
35+
All Snowflake access is specified through roles, which are then assigned to the user you’ll create later.
36+
37+
```sql
38+
-- create role
39+
CREATE ROLE segment_reverse_etl;
40+
41+
-- warehouse access
42+
GRANT USAGE ON WAREHOUSE segment_reverse_etl TO ROLE segment_reverse_etl;
43+
44+
-- database access
45+
GRANT USAGE ON DATABASE segment_reverse_etl TO ROLE segment_reverse_etl;
46+
GRANT CREATE SCHEMA ON DATABASE segment_reverse_etl TO ROLE segment_reverse_etl;
47+
```
48+
6. Enter and run the code below to create the username and password combination that will be used to execute queries. Make sure to enter your password where it says `my_strong_password`.
49+
50+
```sql
51+
-- create user
52+
CREATE USER segment_reverse_etl_user
53+
MUST_CHANGE_PASSWORD = FALSE
54+
DEFAULT_ROLE = segment_reverse_etl
55+
PASSWORD = 'my_strong_password'; -- Do not use this password
56+
57+
-- role access
58+
GRANT ROLE segment_reverse_etl TO USER segment_reverse_etl_user;
59+
```
60+
7. Follow the steps listed in the [Add a Source](/docs/reverse-etl/reverse-etl/#step-1-add-a-source) section to finish adding Snowflake as a source.

0 commit comments

Comments
 (0)