Skip to content

Commit 0fd668f

Browse files
Add BQ dataset's partition expiry
add debugging information about partition expiration which can be set on a dataset level for BQ
1 parent f0c3b24 commit 0fd668f

File tree

1 file changed

+7
-1
lines changed
  • src/connections/storage/catalog/bigquery

1 file changed

+7
-1
lines changed

src/connections/storage/catalog/bigquery/index.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -220,4 +220,10 @@ a need for streaming data into BigQuery, [contact Segment support](https://segme
220220

221221
### I see duplicates in my tables.
222222

223-
This behavior is expected. Segment only de-duplicates data in your views. Refer to the [schema section](#schema) for more details.
223+
This behavior is expected. Segment only de-duplicates data in your views. Refer to the [schema section](#schema) for more details.
224+
225+
### BigQuery Default Partition Expiration
226+
227+
If you notice that your BigQuery data is getting deleted after a specific period of time, then it might be due to a [dataset's default table expiration](https://cloud.google.com/bigquery/docs/updating-datasets#partition-expiration) in BigQuery that sets a 90-day (or similar) expiration on all partitioned tables that are created.
228+
229+
However, you can safely remove these expirations from the tables/dataset and change them to ‘Never’, and change the dataset's default table expiration as needed. We will then be able to run a backfill for you to send all the historical data to your warehouse.

0 commit comments

Comments
 (0)