Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 67 additions & 11 deletions azure/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- Microsoft.ManagedIdentity

2. **Service Principal Permissions**
- Owner - Azure Subscription
- Contributor - Azure Subscription
- Account Admin - Databricks Account Portal

3. **Existing Resources**
Expand All @@ -17,22 +17,33 @@

## 1. **Configure TFVAR file**

Create a tfvar file and name it `db.tfvar`
Copy the file `template.tfvars.example` and name it `db.tffvar`

Copy below configurations and replace with relevant entries
Update the configuration with the relavant entries:

```
databricks_resource_id = "https://XX.com"
azure_client_id = "XX"
azure_client_secret = "XX"
azure_tenant_id = "XX"
# Databricks Environment Variables
databricks_account_id = "" // Your Azure Databricks Account ID
databricks_workspace_id = "" // The Azure Databricks workspace ID e.g. "adb-{workspace_id}.x.azuredatabricks.net"

# Azure RG to deploy assets
resource_group = "XX"
# Common Authentication Variables
databricks_host = "" // The URL of the workspace e.g "adb-xxxxxxxxxxxxxxx.x.azuredatabricks.net"

# PAT Token Authentication Credentials
databricks_token = "" // The personal access token to provision the resources in the Databricks workspace

# Azure-Managed Service Principal credentials
databricks_resource_id = "" // Specifies the resource ID of the Databricks workspace e.g /subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Databricks/workspaces/{workspace_name}
azure_client_id = "" // Specifies the Application ID of the service principal
azure_client_secret = "" // Specifies a client secret associated with the service principal
azure_tenant_id = "" // The Azure Tenant ID the service principal resides in

# Azure Resources
resource_group = "" // The resource group where storage accounts & access connectors will be deployed to

```

Note: Azure and Databricks creds can be injected as environment variables if required. Approach adopted here is for ease of switching environments while testing.
Note: Azure and Databricks credentials can be injected as environment variables if required. Approach adopted here is for ease of switching environments while testing.

## 2. **Provide expected catalog names as input**

Expand All @@ -56,8 +67,12 @@ By default 3 catalogs (and associated DB and Azure entities) will get deployed:
- sandbox
- dev
- prod
and 3 groups will be created:
- production_sp
- developers
- sandbox users

If you need to change the catalog names, navigate to the `variables.tf` in the root directory and update the values given against catalog_1, catalog_2 or catalog_3.
If you need to change the catalog or group names, navigate to the `variables.tf` in the root directory and update the values given against catalog_1, catalog_2 or catalog_3.

```
variable catalog_1 {
Expand All @@ -71,6 +86,47 @@ variable catalog_2 {
variable catalog_3 {
default = "prod"
}

variable "group_1" {
default = "production_sp"
}

variable "group_2" {
default = "developers"
}

variable "group_3" {
default = "sandbox_users"
}



variable "catalog_1_permissions" {
type = map(list(string))
default = {
group_1 = ["ALL_PRIVILEGES"]
group_2 = ["USE_CATALOG", "SELECT"]
group_3 = []
}
}

variable "catalog_2_permissions" {
type = map(list(string))
default = {
group_1 = ["ALL_PRIVILEGES"]
group_2 = ["ALL_PRIVILEGES"]
group_3 = []
}
}

variable "catalog_3_permissions" {
type = map(list(string))
default = {
group_1 = ["ALL_PRIVILEGES"]
group_2 = ["ALL_PRIVILEGES"]
group_3 = ["ALL_PRIVILEGES"]
}
}
```

## 3. **Deploy**
Expand Down
10 changes: 10 additions & 0 deletions azure/modules/compute/cluster.tf
Original file line number Diff line number Diff line change
Expand Up @@ -56,4 +56,14 @@ resource "databricks_cluster" "example" {
depends_on = [
databricks_cluster_policy.uc_qs_policy
]
}

// Cluster Access Control
resource "databricks_permissions" "cluster_usage" {
cluster_id = databricks_cluster.example.id

access_control {
group_name = var.group_name
permission_level = "CAN_MANAGE"
}
}
20 changes: 20 additions & 0 deletions azure/modules/grants/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,23 @@ resource "databricks_grants" "this" {
}
}

data "databricks_catalog" "system_catalog" {
name = "system"
}

data "databricks_schemas" "system_schemas" {
catalog_name = "system"
}

resource "databricks_grant" "system_catalog" {
catalog = data.databricks_catalog.system_catalog.name
principal = var.group_1_name
privileges = ["USE_CATALOG"]
}

resource "databricks_grant" "system_schemas" {
for_each = toset(data.databricks_schemas.system_schemas.ids)
schema = "${each.key}"
principal = var.group_1_name
privileges = ["USE_SCHEMA", "SELECT"]
}