Skip to content

Commit ac30935

Browse files
haritamarharitamar
committed
Update databricks docs
1 parent dd03322 commit ac30935

File tree

5 files changed

+73
-24
lines changed

5 files changed

+73
-24
lines changed

‎docs/changelog.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -249,7 +249,7 @@ Learn more about connecting Elementary to [Jira](https://docs.elementary-data.co
249249

250250
**Known Issues:**
251251

252-
* dbt-databricks must be <1.10.2 (See issue 1931 for more details)
252+
* dbt-databricks must be below 1.10.2 (See issue 1931 for more details)
253253

254254
**Full Changelog**: [v0.18.3...v0.19.0](https://github.com/elementary-data/elementary/compare/v0.18.3...v0.19.0)
255255

‎docs/oss/release-notes/releases/0.19.0.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ This update includes:
1010

1111
**Known Issues:**
1212

13-
- dbt-databricks must be <1.10.2 (See [issue 1931](https://github.com/elementary-data/elementary/issues/1931) for more details)
13+
- dbt-databricks must be below 1.10.2 (See [issue 1931](https://github.com/elementary-data/elementary/issues/1931) for more details)
1414

1515
Check out the full details here: https://github.com/elementary-data/elementary/releases/tag/v0.19.0
1616

‎docs/snippets/cloud/integrations/databricks.mdx‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import CreateServicePrincipal from '/snippets/dwh/databricks/create_service_principal.mdx';
2-
import PermissionsAndSecurity from '/snippets/cloud/integrations/permissions-and-security.mdx';
2+
import PermissionsAndSecurity from '/snippets/dwh/databricks/databricks_permissions_and_security.mdx';
33
import IpAllowlist from '/snippets/cloud/integrations/ip-allowlist.mdx';
44

55
You will connect Elementary Cloud to Databricks for syncing the Elementary schema (created by the [Elementary dbt package](/cloud/onboarding/quickstart-dbt-package)).
@@ -8,14 +8,17 @@ You will connect Elementary Cloud to Databricks for syncing the Elementary schem
88

99
<PermissionsAndSecurity />
1010

11-
### Fill the connection form
11+
### Add an environment in Elementary (requires an admin user)
1212

13-
Provide the following fields:
13+
In the Elementary platform, go to Environments in the left menu, and click on the "Create Environment" button.
14+
Choose a name for your environment, and then choose Databricks as your data warehouse type.
1415

15-
- **Host**: The hostname of your Databricks account to connect to.
16+
Provide the following fields in the form:
17+
18+
- **Server Host**: The hostname of your Databricks account to connect to.
1619
- **Http path**: The path to the Databricks cluster or SQL warehouse.
17-
- **Token**: The token you generated for Elementary. For more information, see [Generate a token](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-service-principals) in the Databricks docs.
20+
- **Access token**: The token you generated for the Elementary service principal (see step 7 under "Create service principal" above)
1821
- **Catalog (optional)**: The name of the Databricks Catalog.
19-
- **Elementary schema**: The name of your Elementary schema. Usually `[schema name]_elementary`.
22+
- **Elementary schema**: The name of your Elementary schema. Usually `[your dbt target schema]_elementary`.
2023

2124
<IpAllowlist />

‎docs/snippets/dwh/databricks/create_service_principal.mdx‎

Lines changed: 25 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,46 @@
11
### Create service principal
22

3-
1. In your Databrick console, go to the admin settings by clicking your username in the to right corner -> Admin settings (add photo)
3+
1. Open your Databricks console, and then open your relevant workspace.
4+
5+
2. Click on your Profile icon on the right and choose Settings.
46

57
<img
6-
src="https://res.cloudinary.com/diuctyblm/image/upload/f_auto,q_auto/v1/dwh/databricks/admin_settings"
7-
alt="Admin settings"
8+
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763312536/databricks_01_choose_settings.png"
9+
alt="Choose settings"
810
/>
911

10-
2. Go to the Service principals tab, then click Add service principal (add photo)
12+
3. On the sidebar, click on *Identity and access*, and then under the *Service Principals* row click on *Manage*.
1113

1214
<img
13-
src="https://res.cloudinary.com/diuctyblm/image/upload/f_auto,q_auto/v1/dwh/databricks/service_principals_settings"
14-
alt="Service principal settings"
15+
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763313131/databricks_02_manage_service_principal_b5gc72.png"
16+
alt="Choose settings"
1517
/>
1618

17-
3. Give the service principal a good name (e.g elementary) and click Add (add photo)
19+
4. Click on the *Add service principal* button, choose "Add new" and give a name to the service principal. This will be used by Elementary Cloud
20+
to access your Databricks instance.
1821

1922
<img
20-
src="https://res.cloudinary.com/diuctyblm/image/upload/f_auto,q_auto/v1/dwh/databricks/add_service_principal"
23+
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763313361/databricks_04_add_service_principal_e54raz.png"
2124
alt="Add service principal"
2225
/>
2326

24-
4. Then, from the service principal configuration view, copy the Application Id (add photo)
27+
5. Click on your newly created service principal, add the "Databricks SQL access" entitlement, and click Update. Also, please copy the
28+
"Application ID" field as it will be used later in the permissions section.
2529

2630
<img
27-
src="https://res.cloudinary.com/diuctyblm/image/upload/f_auto,q_auto/v1/dwh/databricks/service_principal_id"
28-
alt="Service principal ID"
31+
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763313542/databricks_05_add_databricks_sql_access_zzdcf7.png"
32+
alt="Add databricks SQL access"
2933
/>
3034

31-
4. Finally, run the following query:
35+
6. Next, you may also need to allow Token Usage for this service principal (if it is not allowed for all users). To do so, under the settings menu choose Advanced -> Personal Access Tokens -> Permission Settings.
36+
Then add the service principal there.
37+
38+
<img
39+
src="https://res.cloudinary.com/dgpojk42n/image/upload/v1763316575/databricks_06_token_usage_eufjwv.png"
40+
alt="Add databricks SQL access"
41+
/>
3242

33-
```
34-
GRANT SELECT ON SCHEMA <elementary_schema> TO `<service_principal_id>`;
35-
```
43+
7. Create a personal access token for your service principal. For more details, please click [here](https://docs.databricks.com/aws/en/dev-tools/auth/pat#databricks-personal-access-tokens-for-service-principals)
3644

37-
Make sure to replace the `<elementary_schema>` and `<service_principal_id>` placeholders with the correct values
45+
8. Finally, in order to enable Elementary's automated monitors feature, please ensure [predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-or-disable-predictive-optimization-for-your-account) is enabled in your account.
46+
This is required for table statistics to be updated (specifically Elementary needs row count)

‎docs/snippets/dwh/databricks/databricks_permissions_and_security.mdx‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
### Permissions and security
2+
3+
#### Required permissions
4+
5+
Elementary cloud requires the following permissions:
6+
7+
- **Elementary schema read-only access** - This is required by Elementary to read dbt metadata generated by the Elementary dbt package as a part of your pipeline runs.
8+
This permission does not give access to your data.
9+
10+
- **Information schema metadata access** - Elementary needs access to the `system.information_schema.tables` and `system.information_schema.columns` system tables, to get metadata
11+
about existing tables and columns in your data warehouse.
12+
13+
- **Read access needed for some metadata operations (optional)** - In order to enable Elementary's automated volume & freshness monitors, Elementary needs access to query history, as well
14+
as Databricks APIs to obtain table statistics.
15+
These operations require granting SELECT access on your tables. This is a Databricks limitation - Elementary **never** reads any data from your tables, only metadata. However, there isn't
16+
today any table-level metadata-only permission available in Databricks, so SELECT is required.
17+
18+
19+
#### Grants SQL template
20+
21+
Please use the following SQL statements to grant the permissions specified above (you should replace the placeholders with the correct values):
22+
23+
```sql
24+
-- Grant read access on the elementary schema (usually [your dbt target schema]_elementary)
25+
GRANT USE CATALOG ON CATALOG <catalog> TO `<service_principal_app_id>`;
26+
GRANT USE SCHEMA, SELECT ON SCHEMA <elementary_schema> TO `<service_principal_app_id>`;
27+
28+
-- Grant access to information schema tables
29+
GRANT USE CATALOG ON CATALOG system TO `<service_principal_app_id>`;
30+
GRANT USE SCHEMA ON SCHEMA system.information_schema TO `<service_principal_app_id>`;
31+
GRANT SELECT ON TABLE system.information_schema.tables TO `<service_principal_app_id>`;
32+
GRANT SELECT ON TABLE system.information_schema.columns TO `<service_principal_app_id>`;
33+
34+
-- Grant select on tables for history & statistics access
35+
-- (Optional, required for automated volume & freshness tests - see explanation above. You can also limit to specific schemas used by dbt instead of granting on the full catalog)
36+
GRANT USE CATALOG, USE SCHEMA, SELECT ON catalog <catalog> to `<service_principal_app_id>`;
37+
```

0 commit comments

Comments
 (0)