CSP Tags are not visible in some of the subcriptions.

[Harsha5445]

Describe the bug

User attempted to enable the Compliance Profile Security (CSP) tag on PCI workspaces. They have an existing automation process designed to uplift workspaces and enable CSP. However, when running this process on certain workspaces, the enhanced security compliance profile is not available. The user's uplift automation, which works well in QA and Englab subscriptions, struggles with some subscriptions. While the CSP tag updates correctly in a few subscriptions, it fails to do so in others, particularly in Dev, PAT, and Prod environments.

Related command

az resource update --name {adb_workspace_name} --resource-group {adb_resource_group_name} --resource-type Microsoft.Databricks/workspaces --set properties.enhancedSecurityCompliance.complianceSecurityProfile.value=Enabled properties.enhancedSecurityCompliance.complianceSecurityProfile.complianceStandards="['PCI_DSS']" properties.enhancedSecurityCompliance.automaticClusterUpdate.value=Enabled properties.enhancedSecurityCompliance.enhancedSecurityMonitoring.value=Enabled --api-version 2024-09-01-preview

Errors

N/A

Issue script & Debug output

N/A

Expected behavior

CSP tag is updated.

Environment Summary

Databricks CLI version - 1.0.1 and azure CLI version - 2.65.0

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @edwinc, @mvvsubbu.

[Harsha5445]

Updated the related command as:

az resource update --name {adb_workspace_name} --resource-group {adb_resource_group_name} --resource-type Microsoft.Databricks/workspaces --set properties.enhancedSecurityCompliance.complianceSecurityProfile.value=Enabled properties.enhancedSecurityCompliance.complianceSecurityProfile.complianceStandards="['PCI_DSS']" properties.enhancedSecurityCompliance.automaticClusterUpdate.value=Enabled properties.enhancedSecurityCompliance.enhancedSecurityMonitoring.value=Enabled --api-version 2024-09-01-preview

[scottmacchia-msft]

Hello @Harsha5445 ,

It's hard to pinpoint the errors for your user exactly without some debug logs but I've seen similar issues with users updating ESC properties via az resource update. The issue is that the command is trying to set a nested value without its parent objects existing.

The workaround while still using the az resource module is to initialize the parent objects first like this:

az resource update --name {adb_workspace_name} --resource-group {adb_resource_group_name} --resource-type Microsoft.Databricks/workspaces --set properties.enhancedSecurityCompliance={} properties.enhancedSecurityCompliance.complianceSecurityProfile={} properties.enhancedSecurityCompliance.complianceSecurityProfile.value=Enabled properties.enhancedSecurityCompliance.complianceSecurityProfile.complianceStandards="['PCI_DSS']" properties.enhancedSecurityCompliance.automaticClusterUpdate={} properties.enhancedSecurityCompliance.automaticClusterUpdate.value=Enabled properties.enhancedSecurityCompliance.enhancedSecurityMonitoring={} properties.enhancedSecurityCompliance.enhancedSecurityMonitoring.value=Enabled --api-version 2024-09-01-preview

The easier workaround is to use the az databricks module which includes flags for ESC features (https://learn.microsoft.com/en-us/cli/azure/databricks/workspace?view=azure-cli-latest#az-databricks-workspace-update)

[evelyn-ys]

Close as @windoze asked. It should be fixed already