You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When Segment loads data into your warehouse, each sync goes through the following steps:
2
+
1.**Ping:** Segment servers try to connect to your warehouse.
3
+
2.**Scan:** Segment finds new events and updated objects.
4
+
3.**Download:** Segment pulls events and objects from AWS S3 or Dynamo into a staging area.
5
+
4.**Process:** The raw Segment event and object archive files are pulled from the staging area and transformed into database-specific formats. The [warehouse schema](schema/) is also defined in this step.
6
+
5.**Load:** Segment de-duplicates the transformed data and loads it into your warehouse. If you have queries set up in your warehouse, they run after the data was loaded into your warehouse.
@@ -56,19 +58,19 @@ Check out our [Frequently Asked Questions about Warehouses](/docs/connections/st
56
58
57
59
[Can I load in my own data into my warehouse?](/docs/connections/storage/warehouses/faq/#what-if-i-want-to-add-custom-data-to-my-warehouse)
58
60
59
-
[Can I control what data is sent to my warehouse?](/docs/connections/storage/warehouses/faq/)
61
+
[Can I control what data is sent to my warehouse?](/docs/connections/storage/warehouses/faq/#can-i-control-what-data-is-sent-to-my-warehouse)
60
62
61
63
### Managing a warehouse
62
64
63
-
[How fresh is the data in my warehouse?](/docs/connections/storage/warehouses/faq/)
65
+
[How fresh is the data in my warehouse?](/docs/connections/storage/warehouses/faq/#how-fresh-is-the-data-in-segment-warehouses)
64
66
65
-
[Can I add, tweak, or delete some of the tables?](/docs/connections/storage/warehouses/faq/)
67
+
[Can I add, tweak, or delete some of the tables?](/docs/connections/storage/warehouses/faq/#can-we-add-tweak-or-delete-some-of-the-tables)
66
68
67
-
[Can I transform or clean up old data to new formats or specs?](/docs/connections/storage/warehouses/faq/)
69
+
[Can I transform or clean up old data to new formats or specs?](/docs/connections/storage/warehouses/faq/#can-we-transform-or-clean-up-old-data-to-new-formats-or-specs)
68
70
69
71
[What are common errors and how do I debug them?](/docs/connections/storage/warehouses/warehouse-errors/)
70
72
71
-
[How do I speed up my queries?](/docs/connections/storage/warehouses/redshift-tuning/)
73
+
[How do I speed up my Redshift queries?](/docs/connections/storage/warehouses/redshift-tuning/)
The Warehouse Sync process prepares the raw data coming from a source and loads it into a warehouse destination. There are two phases to the sync process:
6
+
<!--- The Warehouse Sync process prepares the raw data coming from a source and loads it into a warehouse destination. There are two phases to the sync process:
7
7
1. **Preparation phase**: This is where Segment prepares the data coming from a source so that it's in the right format for the loading phase.
8
-
2.**Loading phase**: This is where Segment deduplicates data and the data loads into the warehouse destination. Any sync issues that occur in this phase can be traced back to your warehouse.
8
+
2. **Loading phase**: This is where Segment deduplicates data and the data loads into the warehouse destination. Any sync issues that occur in this phase can be traced back to your warehouse. -->
9
9
10
10
Instead of constantly streaming data to the warehouse destination, Segment loads data to the warehouse in bulk at regular intervals. Before the data loads, Segment inserts and updates events and objects, and automatically adjusts the schema to make sure the data in the warehouse is inline with the data in Segment.
11
11
12
+
{% include content/how-a-sync-works.md %}
13
+
12
14
Warehouses sync with all data coming from your source and your data is available in your warehouse within 24-48 hours. If you'd like to manage the data you send to your warehouse, use [Warehouse Selective Sync](#warehouse-selective-sync).
0 commit comments