Skip to content

update retl bigquery setup with storage-api #7945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 29, 2025
Merged

update retl bigquery setup with storage-api #7945

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
40d65b0
update retl bigquery setup with storage-api
gilsegment Sep 16, 2025
7c222eb
change order [netlify-build]
gilsegment Sep 26, 2025
e8b3090
Apply suggestions from code review
stayseesong Sep 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 18 additions & 11 deletions src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,17 +70,24 @@ You can find the location of your BigQuery resources using the following method:
3. The Location of the dataset (like US or EU) is displayed in the Dataset Info.

## Set up BigQuery as your Reverse ETL source
1. In the BigQuery console, search for the service account you created.
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
3. Click **Add Key > Create new key**.
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
5. Copy all the content in the JSON file you created in the previous step.
6. Open the Segment app and navigate to **Connections > Sources**.
7. On the My sources page, click **+ Add source**.
8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**.
9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section.
10. Enter the location of your BigQuery warehouse in the **Data Location** field.
11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
1. In the BigQuery console, search for the service account you created.
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
3. Click **Add Key > Create new key**.
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
5. Copy all the content in the JSON file you created in the previous step.
6. Open the Segment app and navigate to **Connections > Sources**.
7. On the My sources page, click **+ Add source**.
8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**.
9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section.
10. Enter the location of your BigQuery warehouse in the **Data Location** field.
11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
12. If the test connection completes successfully, click **Add source** to complete the setup process.

When setting up a BigQuery Reverse ETL source, you can choose which API Segment uses to read from BigQuery. You can make this selection during the initial setup, or later from Sources > Reverse ETL > BigQuery Source > Settings > Connection Settings.
- **REST API**
Recommended for most tables. REST is generally more cost-efficient, but syncs may be slower for very large datasets.
- **Storage API**
Recommended for large tables. The Storage API can significantly improve throughput and reduce sync times, but may incur higher costs. See [BigQuery Storage API client libraries](https://cloud.google.com/bigquery/docs/reference/storage/libraries) and [BigQuery pricing for data extraction](https://cloud.google.com/bigquery/pricing?hl=en#data_extraction_pricing) for more information
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gilsegment is there a reason why you want this info added after step 10? the placement of this info is very weird as step 10 is about the data location, unless you're saying that choosing which API Segment uses is related to the data location?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the order. @stayseesong



After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.