Reverse ETL 2.0 docs

src/_data/sidenav/main.yml

@@ -182,6 +182,14 @@ sections:
     section:
     - path: /connections/reverse-etl
       title: Reverse ETL Overview
+    - path: /connections/reverse-etl/setup
+      title: Set up Reverse ETL
+    - path: /connections/reverse-etl/mappings
+      title: Reverse ETL Mappings
+    - path: /connections/reverse-etl/observability
+      title: Reverse ETL Observability
+    - path: /connections/reverse-etl/system
+      title: Reverse ETL System
     - path: /connections/reverse-etl/reverse-etl-catalog
       title: Reverse ETL Catalog
     - section_title: Reverse ETL Source Setup Guides

src/connections/reverse-etl/faq.md

@@ -0,0 +1,25 @@
+---
+title: Reverse ETL FAQ
+beta: false
+---
+
+Get answers to some frequently asked Reverse ETL questions. 
+
+## Why do my sync results show *No records extracted* when I select *Updated records* after I enable the mapping? 
+It's expected that when you select **Updated records** the records do not change after the first sync. During the first sync, the reverse ETL system calculates a snapshot of all the results and creates records in the `_segment_reverse_etl` schema. All the records are considered as *Added records* instead of *Updated records* at this time. The records can only meet the *Updated records* condition when the underlying values change after the first sync completes.
+
+## Can I be notified when Reverse ETL syncs fail?
+Yes, you can sign up for Reverse ETL sync notifications.
+
+To receive Reverse ETL sync notifications: 
+1. Navigate to **Settings > User Preferences**.
+2. Select **Reverse ETL** In the **Activity Notifications** section.
+3. Enable the toggle for **Reverse ETL Sync Failed**.
+
+In case of consecutive failures, Segment sends notifications for every sync failure. Segment doesn't send notifications for partial failures.
+
+## Does Segment use Transport Layer Security (TLS) for the connection between Snowflake and Segment?
+Segment uses the [gosnowflake library](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#pkg-variables){:target="_blank"} to connect with Snowflake, which internally uses TLS for the HTTP transport.
+
+## Can I have multiple queries in the Query Builder?
+No. In Reverse ETL, Segment executes queries in a [common table expression](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#with_clause){:target="_blank”}, which can only bind the results from **one single** subquery. If there are multiple semicolons `;` in the query, they'll be treated as several subqueries (even if the second part is only an inline comment) and cause syntax errors.

src/connections/reverse-etl/mappings.md

@@ -0,0 +1,83 @@
+---
+title: Reverse ETL Mappings
+beta: false 
+---
+
+Learn which mapping fields support object and array values in your mappings and how you can reset or replay your syncs. 
+
+## Supported object and arrays 
+
+When you set up destination actions in Reverse ETL, depending on the destination, some [mapping fields](#step-4-create-mappings) may require data to be in the form of an [object](#object-mapping) or [array](#array-mapping). 
+
+### Object mapping
+You can send data to a mapping field that requires object data. An example of object mapping is an `Order completed` model with a `Products` column that’s in object format. 
+
+Example: 
+
+    {
+        "product": {
+            "id": 0001,
+            "color": "pink",
+            "name": "tshirt",
+            "revenue": 20,
+            "inventory": 500
+        }
+    }
+
+To send data to a mapping field that requires object data, you can choose between these two options: 
+
+Option | Details
+------ | --------
+Customize object | This enables you to manually set up the mapping fields with any data from the model. If the model contains some object data, you can select properties within the object to set up the mappings as well.
+Select object | This enables you to send all nested properties within an object. The model needs to provide data in the format of the object. 
+
+> success ""
+> Certain object mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your object don't match with the destination properties, the data won't send. Segment recommends you to use **Customize Object** to ensure your mapping is successful.
+
+
+### Array mapping
+To send data to a mapping field that requires array data, the model must provide data in the format of an array of objects. An example is an `Order completed` model with a `Product purchased` column that’s in an array format.
+
+Example: 
+
+
+    [
+    {
+        "currency": "USD",
+        "price": 40,
+        "productName": "jacket",
+        "purchaseTime": "2021-12-17 23:43:47.102",
+        "quantity": 1
+    },
+    {
+        "currency": "USD",
+        "price": 5,
+        "productName": "socks",
+        "quantity": 2
+    }
+    ]
+
+
+To send data to a mapping field that requires array data, you can choose between these two options: 
+
+Option | Details
+------ | --------
+Customize array | This enables you to select the specific nested properties to send to the destination. 
+Select array | This enables you to send all nested properties within the array.
+
+> success ""
+> Certain array mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your array don't match the destination properties, the data won't send. Segment recommends you to use the **Customize array** option to ensure your mapping is successful.
+
+Objects in an array don't need to have the same properties. If a user selects a missing property in the input object for a mapping field, the output object will miss the property. 
+
+## Reset syncs
+You can reset your syncs so that your data is synced from the beginning. This means that Segment resyncs your entire dataset for the model.
+
+To reset a sync:
+1. Select the three dots next to **Sync now**.
+2. Select **Reset sync**. 
+3. Select the checkbox that you understand what happens when a sync is reset.
+4. Click **Reset sync**.
+
+## Replays
+You can choose to replay syncs. To replay a specific sync, contact [[email protected]](mailto:[email protected]). Keep in mind that triggering a replay resyncs all records for a given sync.

src/connections/reverse-etl/observability.md

@@ -0,0 +1,33 @@
+---
+title: Reverse ETL Observability
+beta: false 
+---
+
+Use Segment's sync history and email alert features to get better insights about the status of your Reverse ETL syncs. 
+
+## Sync history
+Check the status of your data extractions and see details of your syncs. Click into failed records to view additional details on the error, sample payloads to help you debug the issue, and recommended actions.
+
+To check the status of your extractions:
+1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
+2. Select the destination you want to view.
+3. Select the mapping you want to view.  
+4. Click the sync you want to view to get details of the sync. You can view:
+    * The status of the sync.
+    * Details of how long it took for the sync to complete.
+    * How many total records were extracted, as well as a breakdown of the number of records added, updated, and deleted.
+    * The load results - how many successful records were synced as well as how many records were updated, deleted, or are new.
+5. If your sync failed, click the failed reason to get more details on the error and view sample payloads to help troubleshoot the issue.
+
+## Email alerts
+You can opt in to receive email alerts regarding notifications for Reverse ETL. 
+
+To subscribe to email alerts: 
+1. Navigate to **Settings > User Preferences**. 
+2. Select **Reverse ETL** in the **Activity Notifications** section.
+3. Click the toggle on for the notifications you want to receive. You can choose from:
+
+    Notification | Details
+    ------ | -------
+    Reverse ETL Sync Failed | Set toggle on to receive notification when your Reverse ETL sync fails. 
+    Reverse ETL Sync Partial Success | Set toggle on to receive notification when your Reverse ETL sync is partially successful. 

src/connections/reverse-etl/reverse-etl-catalog.md

@@ -1,8 +1,10 @@
 ---
 title: Reverse ETL Catalog
-hidden: true
+beta: false
 ---
 
+Reverse ETL supports most of the Segment destination catalog - 30+ Actions destinations are natively supported, Segment Classic destinations are supported through the [Segment Connections](#segment-connections-destination) destination, and Twilio Engage Premier Subscriptions users can use the Segment Profiles destination to sync subscription data from warehouses to destinations. 
+
 These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).  
 
 <div class="destinations-catalog">
@@ -38,3 +40,28 @@ These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you
       </div>
     </div>
 
+## Segment Connections destination
+If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).  
+
+The Segment Connections destination enables you to mold data extracted from your warehouse in [Segment Spec](/docs/connections/spec/) API calls that are then processed by [Segment’s HTTP Tracking API](/docs/connections/sources/catalog/libraries/server/http-api/). The requests hit Segment’s servers, and then Segment routes your data to any destination you want. Get started with the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/). 	
+
+> warning ""
+> The Segment Connections destination sends data to Segment’s Tracking API, which has cost implications. New users count as new MTUs and each call counts as an API call. For information on how Segment calculates MTUs and API calls, please see [MTUs, Throughput and Billing](/docs/guides/usage-and-billing/mtus-and-throughput/).
+
+## Send data to Engage with Segment Profiles
+Engage Premier Subscriptions users can use Reverse ETL to sync subscription data from warehouses to destinations.
+
+To get started with using Reverse ETL for subscriptions:
+1. Navigate to Engage > Audiences and select the Profile explorer tab.
+2. Click Manage subscription statuses and select Update subscription statuses.
+3. Select Sync with RETL as the method to update your subscription statuses.
+4. Click Configure.
+5. In the Reverse ETL catalog, select the Reverse ETL source you want to use.
+6. Set up the source. Refer to the add a source section for more details on how to set up the source.
+7. Add the Segment Profiles destination as your Reverse ETL destination. Refer to add a destination for more details to set up the destination.
+8. Once your destination is set, go to the Mappings tab of your destination and click Add Mapping.
+9. Select the model you want to use and then select Send Subscriptions.
+10. Click Create Mapping.
+11. Follow the steps in the Create Mappings section to set your mappings.
+
+<!--- TODO: Add link ^^ --->

src/connections/reverse-etl/reverse-etl-source-setup-guides/azure-setup.md

@@ -72,5 +72,5 @@ To set up Azure as your Reverse ETL source:
 9. Click **Test Connection** to see if the connection works. If the connection fails, make sure you have the right permissions and credentials, then try again.
 10. Click **Add source** if the test connection is successful. 
 
-After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 
+After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 
 

src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md

@@ -30,6 +30,9 @@ To set up the Segment BigQuery connector:
 
 After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl#step-2-add-a-model).
 
+> info "BigQuery Reverse ETL sources support Segment's dbt extension"
+> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
+
 ## Constructing your own role or policy
 When you construct your own role or policy, Segment needs the following permissions:
 
@@ -48,3 +51,5 @@ Permission | Details
 `bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
 
 The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. 
+
+After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 

src/connections/reverse-etl/reverse-etl-source-setup-guides/databricks-setup.md

@@ -4,11 +4,10 @@ title: Databricks Reverse ETL Setup
 
 Set up Databricks as your Reverse ETL source. 
 
-At a high level, when you set up Databricks for Reverse ETL, the configured service-principal needs read permissions for any resources (databases, schemas, tables) the query needs to access. Segment keeps track of changes to your query results with a managed schema (`__SEGMENT_REVERSE_ETL`), which requires the configured service-principal to allow write permissions for that schema.
-
-> info ""
-> Segment supports only OAuth (M2M) authentication. To generate a client ID and Secret, follow the steps listed in Databricks' [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html){:target="_blank"} documentation.
+At a high level, when you set up Databricks for Reverse ETL, the configured service-principal needs read permissions for any resources (databases, schemas, tables) the query needs to access. Segment keeps track of changes to your query results with a managed schema (`__SEGMENT_REVERSE_ETL`), which requires the configured service-principal to allow write permissions for that schema. Segment supports only OAuth (M2M) authentication. To generate a client ID and Secret, follow the steps listed in Databricks' [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html){:target="_blank"} documentation.
 
+> info "Databricks Reverse ETL sources support Segment's dbt extension"
+> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
 
 ## Required permissions
 * Make sure the service principal you use to connect to Segment has permissions to use that warehouse. In the Databricks console go to **SQL warehouses** and select the warehouse you're using. Navigate to **Overview > Permissions** and make sure the service principal you use to connect to Segment has *can use* permissions.
@@ -60,4 +59,4 @@ To set up Databricks as your Reverse ETL source:
 > Segment previously supported token-based authentication, but now uses OAuth (M2M) authentication at the recommendation of Databricks.
 > If you previously set up your source using token-based authentication, Segment will continue to support it. If you want to create a new source or update the connection settings of an existing source, Segment only supports [OAuth machine-to-machine (M2M) authentication](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html){:target="_blank"}.
 
-Once you've succesfully added your Databricks source, [add a model](/docs/connections/reverse-etl/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 
+Once you've successfully added your Databricks source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 

src/connections/reverse-etl/reverse-etl-source-setup-guides/postgres-setup.md

@@ -6,6 +6,9 @@ Set up Postgres as your Reverse ETL source.
 
 At a high level, when you set up Postgres for Reverse ETL, the configured user/role needs read permissions for any resources (databases, schemas, tables) the query needs to access. Segment keeps track of changes to your query results with a managed schema (`__SEGMENT_REVERSE_ETL`), which requires the configured user to allow write permissions for that schema.
 
+> info "Postgres Reverse ETL sources support Segment's dbt extension"
+> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
+
 Segment supports the following Postgres database providers:
 - Heroku
 - RDS
@@ -36,3 +39,5 @@ To set up Postgres with Reverse ETL:
 * Give the `segment` user read permissions for any resources (databases, schemas, tables) the query needs to access. 
 
 * Give the `segment` user write permissions for the Segment managed schema (`__SEGMENT_REVERSE_ETL`), which keeps track of changes to the query results.  
+
+After you've successfully added your Postgres source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.

src/connections/reverse-etl/reverse-etl-source-setup-guides/redshift-setup.md

@@ -5,7 +5,10 @@ redirect_from:
   - '/reverse-etl/redshift-setup/'
 ---
 
-Set up Redshift as your Reverse ETL source. 
+Set up Redshift as your Reverse ETL source.
+
+> info "Redshift Reverse ETL sources support Segment's dbt extension"
+> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
 
 To set up Redshift with Reverse ETL: 
 1. Log in to Redshift and select the Redshift cluster you want to connect with Reverse ETL.
@@ -32,3 +35,5 @@ If you are able to run the query in the Query Builder, but the sync fails with t
 ```ts
 SELECT id FROM <schema_name>.<table_name>
 ```
+
+After you've successfully added your Redshift source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.