[ISSUE] Issue with `databricks_group` resource. When adding an external group (i.e. Entra group) to the workspace level it does not create the group at the account level

[haveyoumetcp]

Configuration

provider "databricks" {
  alias = "workspace"
  host = azurerm_databricks_workspace.databricks_workspace.workspace_url
}

data "azuread_group" "test_cdp_user_group" {
  for_each       = toset(var.test_databricks_workspace_user_groups)
  display_name   = each.key
  security_enabled = true
}

resource "databricks_group" "test_cdp_workspace_user_databricks_groups" {
  for_each     = data.azuread_group.test_cdp_user_group
  display_name = each.value.display_name
  external_id  = each.value.object_id
  provider     = databricks.workspace
  force        = true
}

Expected Behavior

The external group is added to the account level.

When adding a group via the UI in the workspace it automatically adds it the account.

Actual Behavior

The external group is not added to the account level.

Steps to Reproduce

1. `terraform apply

Terraform and provider versions

v1.96.0

Is it a regression?

Not sure it ever worked, but mostly an issue now with the Automated Identity Federation and not importing groups via SCIM.

Debug Output

Important Factoids

I can add the group when using the account level connection with this provider:

provider "databricks" {
  alias      = "account"
  host       = "https://accounts.azuredatabricks.net"
  account_id = var.databricks_account_id
}

However, there's a huge issue with this. If a group is deleted (i.e. in a development process), then the group is deleted at the account level and will impact production processes.

Would you like to implement a fix?

No

[alexott]

Please collect debug logs with both creation and deletion of the group

[alexott]

in workspace context we're using workspace-level API, so deletion shouldn't affect account. If it happens, then it's a SCIM backend problem that should be reported via support ticket.

[haveyoumetcp]

This issue isn't about deletion. This issue is specifically about the posted Expected and Actual behavior:

Expected Behavior
The external group is added to the account level.

When adding a group via the UI in the workspace it automatically adds it the account.

Actual Behavior
The external group is not added to the account level.

[alexott]

There is no public API yet for adding an account-level group in the workspace context. Otherwise, it would have been done a long time ago

[haveyoumetcp]

There is no public API yet for adding an account-level group in the workspace context. Otherwise, it would have been done a long time ago

So, that's to say this isn't possible and won't be fixed/enhanced?

I'd assume this wasn't done previously because groups (and users) were added to the account via SCIM. But, that's no longer needed; which I think makes this valid.

[alexott]

It will be fixed eventually, but we need API for that.

[haveyoumetcp]

It will be fixed eventually, but we need API for that.

Awesome sounds good. I'll work on getting the debug logs for the creation of the group. Thanks.

[alexott]

No need for logs right now - I misunderstood the description. The problem with workspace-level groups is well known (unfortunately)

[haveyoumetcp]

Thank you for your quick and helpful responses. We'll move forward in another way and monitor for new functionality around this.

[haveyoumetcp]

So, I found this GitHub issue:
#5142

And @alexott I saw your comment on that:
#5142 (comment)

That took me to this API (In Beta):
https://docs.databricks.com/api/account/iamv2/resolvegroup

But, there's also one for the workspace:
https://docs.databricks.com/api/workspace/iamv2/resolvegroupproxy

And going by the description for that API it sounds like what's needed. So, maybe future state would use this API. But, for the time being I added this Terraform that calls this API and everything works so far:

resource "null_resource" "resolve_group_by_external_id" {
  for_each = data.azuread_group.test_cdp_user_group
  
  provisioner "local-exec" {
    command = <<-EOT
      curl -X POST "https://${azurerm_databricks_workspace.databricks_workspace.workspace_url}/api/2.0/identity/groups/resolveByExternalId" \
        -H "Authorization: Bearer ${databricks_token.azdo_api_token.token_value}" \
        -H "Content-Type: application/json" \
        -d '{"external_id": "${each.value.object_id}"}'
    EOT
  }
  
  triggers = {
    external_id = each.value.object_id
    workspace_url = azurerm_databricks_workspace.databricks_workspace.workspace_url
  }
}