You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md
+48-36Lines changed: 48 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,56 +4,68 @@ redirect_from:
4
4
- '/reverse-etl/bigquery-setup/'
5
5
---
6
6
## Constructing your own role or policy
7
-
When you construct your own role or policy, Segment needs the following permissions:
7
+
> warning ""
8
+
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
8
9
9
-
Permission | Details
10
-
---------- | --------
11
-
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
12
-
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists.
13
-
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
10
+
When constructing a role and service-account we provide guidance for two approaches:
11
+
1. Grant full access - this approach gives Segment all the required permissions and is slightly faster as it also gives us to permission to automatically complete the setup for you.
12
+
2. Grant limited access - this approach is more secured, but because of the limited access we have, we require few additional one time setup steps from you.
14
13
15
-
If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions:
14
+
You can choose whatever works best for you and skip the other step.
15
+
16
+
17
+
### Grant full access
18
+
With this approach we will use BigQuery predefined roles:
19
+
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
20
+
2. Click **+ Create Service Account** to create a new service account.
21
+
3. Enter your **Service account name** and a description of what the account will do.
22
+
4. Click **Create and Continue**.
23
+
5. Click **+ Add another role** and add the *BigQuery User* role.
24
+
6. Click **+ Add another role** and add the *BigQuery Data Editor* role.
25
+
7. Click **Continue**.
26
+
8. Click **Done**.
27
+
28
+
### Grant limited access
29
+
With this approach we will use costume role with the below permissions:
16
30
17
31
Permission | Details
18
32
---------- | --------
19
33
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists.
20
34
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists.
21
35
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
22
36
23
-
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.
24
37
25
-
To create a new role in BigQuery console:
26
38
1. Navigate to **IAM & Admin > Roles** in BigQuery.
27
-
2. Click **+ CREATE ROLE** to create a new role.
39
+
2. Click **+ CREATE ROLE** to create a new role.
28
40
3. Add **title** and **Description** as you like.
29
-
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
30
-
5. Click **CREATE**.
31
-
32
-
## Create service account
33
-
34
-
> warning ""
35
-
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
36
-
37
-
To set up the Segment BigQuery connector:
38
-
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
39
-
2. Click **+ Create Service Account** to create a new service account.
40
-
3. Enter your **Service account name** and a description of what the account will do.
41
-
4. Click **Create and Continue**.
42
-
5. In the **Grant this service account access to project** section, select the role you created in the previous section.
43
-
6. Click **+ Add another role** and add the *BigQuery Job User* role.
44
-
7. Click **Continue**.
45
-
8. Click **Done**.
46
-
9. Search for the service account you just created.
47
-
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
48
-
11. Click **Add Key > Create new key**.
49
-
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
41
+
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
42
+
5. Click **CREATE**.
43
+
6. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
44
+
7. Click **+ Create Service Account** to create a new service account.
45
+
8. Enter your **Service account name** and a description of what the account will do.
46
+
9. Click **Create and Continue**.
47
+
10. In the **Grant this service account access to project** section, select the role you just created.
48
+
11. Click **Continue**.
49
+
12. Click **Done**.
50
+
13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment:
51
+
```
52
+
CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`;
53
+
```
54
+
14. Grant additional permissions just on the newly created dataset:
55
+
```
56
+
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>";
57
+
```
50
58
51
59
## Set up BigQuery as your Reverse ETL source
52
-
1. Copy all the content within the **JSON** file created and downloaded in previous section.
53
-
2. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section.
54
-
3. Enter your **Data Location**.
55
-
4. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
56
-
5. Click **Add source** if the test connection is successful.
60
+
1. In the BigQuery console, search for the service account you just created.
61
+
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
62
+
3. Click **Add Key > Create new key**.
63
+
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
64
+
5. Copy all the content within the **JSON** file created and downloaded in previous section.
65
+
6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section.
66
+
7. Enter your **Data Location**.
67
+
8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
68
+
9. Click **Add source** if the test connection is successful.
57
69
58
70
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
0 commit comments