[ISSUE] databricks_grants Resource Revokes Catalog-Level Grants When Table-Level Grants Are Applied

[asasidharan1]

Summary

When managing grants at different hierarchy levels (catalog vs table) using separate databricks_grants resources, applying table-level grants silently revokes catalog-level grants that were previously configured. The Terraform plan does not show these revocations, leading to unexpected permission loss.

Provider Version

• Provider: databrickslabs/databricks
• Version: >= 1.47.0 (observed in latest versions)

Terraform Version

• Terraform: >= 1.0

Description

The databricks_grants resource uses a "replace all" strategy when managing grants. When you have:

1. A databricks_grants resource managing grants at the catalog level (already configured and working)
2. A separate databricks_grants resource managing grants at the table level (for specific tables in that catalog)

Applying the table-level grants resource will revoke all catalog-level grants, even though:

• The Terraform plan shows no changes to catalog-level grants
• The catalog-level grants resource is still in the configuration
• The catalog-level grants were working correctly before

Steps to Reproduce

Note: This issue was observed in a production environment. The steps below are based on the actual production scenario that occurred.

1. Initial Setup - Create Catalog with Catalog-Level Grants

resource "databricks_catalog" "example" {
  name         = "example_catalog"
  metastore_id = var.metastore_id
}

# Catalog-level grants (configured first)
resource "databricks_grants" "catalog_grant" {
  catalog = databricks_catalog.example.name

  grant {
    principal  = "data_analysts"
    privileges = ["USE_CATALOG", "USE_SCHEMA"]
  }

  grant {
    principal  = "data_engineers"
    privileges = ["USE_CATALOG", "SELECT"]
  }
}

Result: Catalog-level grants are created successfully. Users in data_analysts and data_engineers groups have the appropriate permissions.

2. Add Table-Level Grants

# Add table-level grants for specific tables
resource "databricks_grants" "table_grant" {
  catalog = databricks_catalog.example.name
  schema  = "default"
  table   = "my_table"

  grant {
    principal  = "data_engineers"
    privileges = ["SELECT"]
  }
}

3. Run Terraform Plan

terraform plan

Expected: Plan should show:

• Creation of table-level grants for data_engineers on my_table
• No changes to catalog-level grants

Actual: Plan shows:

• Creation of table-level grants for data_engineers on my_table
• No changes to catalog-level grants (incorrect - they will be revoked)

4. Run Terraform Apply

terraform apply

Expected:

• Table-level grants are created
• Catalog-level grants remain intact

Actual:

• Table-level grants are created
• Catalog-level grants are silently revoked
• Users in data_analysts group lose USE_CATALOG and USE_SCHEMA permissions
• Users in data_engineers group lose catalog-level USE_CATALOG and SELECT permissions (only table-level SELECT remains)

Production Observation

This issue was discovered in a production environment where:

1. Catalog-level grants were already configured and working
2. Table-level grants were added to the Terraform configuration
3. After applying, catalog-level grants were silently revoked
4. The Terraform plan showed no indication that catalog-level grants would be affected
5. Access issues were discovered only after users reported problems

Expected Behavior

1. Plan should detect conflicts: When table-level grants would affect catalog-level grants, the plan should:

   • Show a warning about potential grant conflicts
   • Show that catalog-level grants will be affected
   • Optionally, show what grants will be revoked

2. Grants should be additive: Managing grants at different hierarchy levels should be additive, not destructive. Catalog-level grants should be preserved when table-level grants are applied.

3. Read logic should account for all levels: The provider's read logic should:

   • Read grants at all hierarchy levels (catalog, schema, table)
   • Understand the relationship between grants at different levels
   • Show accurate diffs in the plan

Actual Behavior

1. Plan shows no changes: The plan does not indicate that catalog-level grants will be affected
2. Grants are silently revoked: Catalog-level grants are removed during apply
3. No warning or error: No indication that grants at higher hierarchy levels will be affected

Code Example

Here's a complete example demonstrating the issue (based on actual production code):

# Catalog resource
resource "databricks_catalog" "test_catalog" {
  name         = "test_catalog_bug"
  metastore_id = var.metastore_id
}

# Catalog-level grants (created first)
resource "databricks_grants" "catalog_grants" {
  catalog = databricks_catalog.test_catalog.name

  grant {
    principal  = "data_analysts"
    privileges = ["USE_CATALOG", "USE_SCHEMA"]
  }

  grant {
    principal  = "data_engineers"
    privileges = ["USE_CATALOG", "SELECT"]
  }
}

# Table-level grants (added later)
resource "databricks_grants" "table_grants" {
  for_each = {
    "schema1.table1" = {
      schema     = "schema1"
      table      = "table1"
      principal  = "data_engineers"
      privileges = ["SELECT"]
    }
  }

  catalog = databricks_catalog.test_catalog.name
  schema  = each.value.schema
  table   = each.value.table

  grant {
    principal  = each.value.principal
    privileges = each.value.privileges
  }
}

Sequence of events:

1. Apply with only catalog_grants → Works correctly, all groups have catalog-level permissions
2. Add table_grants and run plan → Shows creation of table grants, no changes to catalog_grants
3. Apply → catalog_grants are silently revoked, breaking access for data_analysts and other groups

Impact

• Severity: High
• Impact:
  • Production permissions can be accidentally revoked
  • No visibility into what will be affected
  • Requires manual verification after each apply
  • Breaks the principle of least surprise

Workaround

Currently, the only reliable workaround is to:

1. Avoid mixing hierarchy levels: Only manage grants at one level (either catalog OR table, not both)
2. Use explicit dependencies: Add depends_on to ensure order, though this doesn't prevent the issue
3. Manual verification: Check grants after each apply to ensure nothing was revoked
4. Use separate catalogs: If you need both catalog and table-level grants, use separate catalogs

Additional Context

Databricks Grant Hierarchy

In Databricks Unity Catalog, grants follow a hierarchy:

• Metastore (highest level)
• Catalog
• Schema
• Table (most specific)

More specific grants should take precedence, but the Terraform provider doesn't respect this hierarchy when managing grants.

Related Issues

This appears to be related to how the provider implements the databricks_grants resource:

• It uses a "replace all" strategy rather than "merge/additive"
• The read logic only reads grants at the specified level
• No cross-level grant awareness

Suggested Fix

1. Improve read logic: When reading grants at catalog level, also read grants at schema and table levels to understand the full grant state
2. Add plan warnings: Detect when applying grants at one level will affect grants at another level
3. Implement additive grants: Consider making grant management additive rather than replace-all, or at least provide an option for this behavior
4. Better documentation: Document that managing grants at different levels can cause conflicts

Environment

• OS: Linux/macOS (production environment)
• Terraform Version: >= 1.0
• Provider Version: >= 1.47.0 (observed in latest versions)
• Databricks Version: Unity Catalog enabled workspace

Additional Notes

This issue was discovered in a production environment where:

• Catalog-level grants were already configured and working for multiple groups
• Table-level grants were added to the Terraform configuration for specific tables
• The catalog-level grants were silently revoked during apply
• The Terraform plan showed no indication that catalog-level grants would be affected
• Access issues were discovered only after users reported problems (groups lost catalog-level permissions)
• Manual verification confirmed that catalog-level grants were removed after applying table-level grants

Reproducibility: While we cannot easily reproduce this in a test environment due to resource constraints, the production incident clearly demonstrates the issue. The behavior is consistent with the provider's "replace all" strategy for grant management, where managing grants at a more specific level (table) appears to affect grants at a higher level (catalog).