Skip to content

Commit baea5ce

Browse files
committed
updating bigquery source setup
1 parent 7533105 commit baea5ce

File tree

1 file changed

+48
-36
lines changed

1 file changed

+48
-36
lines changed

src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md

Lines changed: 48 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -4,56 +4,68 @@ redirect_from:
44
- '/reverse-etl/bigquery-setup/'
55
---
66
## Constructing your own role or policy
7-
When you construct your own role or policy, Segment needs the following permissions:
7+
> warning ""
8+
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
89
9-
Permission | Details
10-
---------- | --------
11-
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
12-
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists.
13-
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
10+
When constructing a role and service-account we provide guidance for two approaches:
11+
1. Grant full access - this approach gives Segment all the required permissions and is slightly faster as it also gives us to permission to automatically complete the setup for you.
12+
2. Grant limited access - this approach is more secured, but because of the limited access we have, we require few additional one time setup steps from you.
1413

15-
If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions:
14+
You can choose whatever works best for you and skip the other step.
15+
16+
17+
### Grant full access
18+
With this approach we will use BigQuery predefined roles:
19+
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
20+
2. Click **+ Create Service Account** to create a new service account.
21+
3. Enter your **Service account name** and a description of what the account will do.
22+
4. Click **Create and Continue**.
23+
5. Click **+ Add another role** and add the *BigQuery User* role.
24+
6. Click **+ Add another role** and add the *BigQuery Data Editor* role.
25+
7. Click **Continue**.
26+
8. Click **Done**.
27+
28+
### Grant limited access
29+
With this approach we will use costume role with the below permissions:
1630

1731
Permission | Details
1832
---------- | --------
1933
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists.
2034
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists.
2135
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
2236

23-
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.
2437

25-
To create a new role in BigQuery console:
2638
1. Navigate to **IAM & Admin > Roles** in BigQuery.
27-
2. Click **+ CREATE ROLE** to create a new role.
39+
2. Click **+ CREATE ROLE** to create a new role.
2840
3. Add **title** and **Description** as you like.
29-
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
30-
5. Click **CREATE**.
31-
32-
## Create service account
33-
34-
> warning ""
35-
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
36-
37-
To set up the Segment BigQuery connector:
38-
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
39-
2. Click **+ Create Service Account** to create a new service account.
40-
3. Enter your **Service account name** and a description of what the account will do.
41-
4. Click **Create and Continue**.
42-
5. In the **Grant this service account access to project** section, select the role you created in the previous section.
43-
6. Click **+ Add another role** and add the *BigQuery Job User* role.
44-
7. Click **Continue**.
45-
8. Click **Done**.
46-
9. Search for the service account you just created.
47-
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
48-
11. Click **Add Key > Create new key**.
49-
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
41+
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
42+
5. Click **CREATE**.
43+
6. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
44+
7. Click **+ Create Service Account** to create a new service account.
45+
8. Enter your **Service account name** and a description of what the account will do.
46+
9. Click **Create and Continue**.
47+
10. In the **Grant this service account access to project** section, select the role you just created.
48+
11. Click **Continue**.
49+
12. Click **Done**.
50+
13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment:
51+
```
52+
CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`;
53+
```
54+
14. Grant additional permissions just on the newly created dataset:
55+
```
56+
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>";
57+
```
5058
5159
## Set up BigQuery as your Reverse ETL source
52-
1. Copy all the content within the **JSON** file created and downloaded in previous section.
53-
2. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section.
54-
3. Enter your **Data Location**.
55-
4. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
56-
5. Click **Add source** if the test connection is successful.
60+
1. In the BigQuery console, search for the service account you just created.
61+
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
62+
3. Click **Add Key > Create new key**.
63+
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
64+
5. Copy all the content within the **JSON** file created and downloaded in previous section.
65+
6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section.
66+
7. Enter your **Data Location**.
67+
8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
68+
9. Click **Add source** if the test connection is successful.
5769
5870
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
5971

0 commit comments

Comments
 (0)