Skip to content

Commit ad73c1f

Browse files
committed
Creating include, adding it to warehouse overview/sync pages
1 parent 27bf441 commit ad73c1f

File tree

3 files changed

+17
-7
lines changed

3 files changed

+17
-7
lines changed
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
When Segment loads data into your warehouse, each sync goes through the following steps:
2+
1. **Ping:** Segment servers try to connect to your warehouse.
3+
2. **Scan:** Segment finds new events and updated objects.
4+
3. **Download:** Segment pulls events and objects from AWS S3 or Dynamo into a staging area.
5+
4. **Process:** The raw Segment event and object archive files are pulled from the staging area and transformed into database-specific formats. The [warehouse schema](schema/) is also defined in this step.
6+
5. **Load:** Segment de-duplicates the transformed data and loads it into your warehouse. If you have queries set up in your warehouse, they run after the data was loaded into your warehouse.

src/connections/storage/warehouses/index.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ Relational databases are great when you know and predefine the information colle
2020

2121
Examples of data warehouses include Amazon Redshift, Google BigQuery, and Postgres.
2222

23+
{% include content/how-a-sync-works.md %}
24+
2325
<div data-headings-anchors id="warehouse-schemas"></div>
2426
> info "Looking for the Warehouse Schemas docs?"
2527
> They've moved! Check them out [here](schema/).
@@ -56,19 +58,19 @@ Check out our [Frequently Asked Questions about Warehouses](/docs/connections/st
5658

5759
[Can I load in my own data into my warehouse?](/docs/connections/storage/warehouses/faq/#what-if-i-want-to-add-custom-data-to-my-warehouse)
5860

59-
[Can I control what data is sent to my warehouse?](/docs/connections/storage/warehouses/faq/)
61+
[Can I control what data is sent to my warehouse?](/docs/connections/storage/warehouses/faq/#can-i-control-what-data-is-sent-to-my-warehouse)
6062

6163
### Managing a warehouse
6264

63-
[How fresh is the data in my warehouse?](/docs/connections/storage/warehouses/faq/)
65+
[How fresh is the data in my warehouse?](/docs/connections/storage/warehouses/faq/#how-fresh-is-the-data-in-segment-warehouses)
6466

65-
[Can I add, tweak, or delete some of the tables?](/docs/connections/storage/warehouses/faq/)
67+
[Can I add, tweak, or delete some of the tables?](/docs/connections/storage/warehouses/faq/#can-we-add-tweak-or-delete-some-of-the-tables)
6668

67-
[Can I transform or clean up old data to new formats or specs?](/docs/connections/storage/warehouses/faq/)
69+
[Can I transform or clean up old data to new formats or specs?](/docs/connections/storage/warehouses/faq/#can-we-transform-or-clean-up-old-data-to-new-formats-or-specs)
6870

6971
[What are common errors and how do I debug them?](/docs/connections/storage/warehouses/warehouse-errors/)
7072

71-
[How do I speed up my queries?](/docs/connections/storage/warehouses/redshift-tuning/)
73+
[How do I speed up my Redshift queries?](/docs/connections/storage/warehouses/redshift-tuning/)
7274

7375
### Analyzing with SQL
7476

src/connections/storage/warehouses/warehouse-syncs.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,14 @@ title: Warehouse Syncs
33
redirect_from: '/connections/warehouses/selective-sync/'
44
---
55

6-
The Warehouse Sync process prepares the raw data coming from a source and loads it into a warehouse destination. There are two phases to the sync process:
6+
<!--- The Warehouse Sync process prepares the raw data coming from a source and loads it into a warehouse destination. There are two phases to the sync process:
77
1. **Preparation phase**: This is where Segment prepares the data coming from a source so that it's in the right format for the loading phase.
8-
2. **Loading phase**: This is where Segment deduplicates data and the data loads into the warehouse destination. Any sync issues that occur in this phase can be traced back to your warehouse.
8+
2. **Loading phase**: This is where Segment deduplicates data and the data loads into the warehouse destination. Any sync issues that occur in this phase can be traced back to your warehouse. -->
99

1010
Instead of constantly streaming data to the warehouse destination, Segment loads data to the warehouse in bulk at regular intervals. Before the data loads, Segment inserts and updates events and objects, and automatically adjusts the schema to make sure the data in the warehouse is inline with the data in Segment.
1111

12+
{% include content/how-a-sync-works.md %}
13+
1214
Warehouses sync with all data coming from your source and your data is available in your warehouse within 24-48 hours. If you'd like to manage the data you send to your warehouse, use [Warehouse Selective Sync](#warehouse-selective-sync).
1315

1416
## Sync History

0 commit comments

Comments
 (0)