Add Azure configuration options for tiered storage

[andrewstucki]

#1099 points out that we're missing the ability to set azure_vm_instance_metadata in our rendering code/helm chart. Following our docs for configuring tiered storage (make sure you're selecting the "Azure" option), looks like we're missing a few other fields as well. It looks like these are already fields in our Redpanda CRD, so my guess is this is mainly affecting helm chart users due to the schema validation that happens during a helm install.

[KoomeKiriinya]

Thanks @andrewstucki

[andrewstucki]

FYI too, here is a list of all of the cluster configuration options related to tiered storage and azure pulled from a recent-ish schema just to validate the typing (we should really leverage the published schema to autogen some of this code):

➜ curl https://dl.redpanda.com/public/redpanda/raw/names/redpanda-configuration-schema/versions/25.1.7/configuration_schema.json.gz --silent -o - | gunzip - | jq '.properties | with_entries(select(.key | startswith("cloud_storage_azure"))) | with_entries(select(.value.visibility == "user"))'
{
  "cloud_storage_azure_adls_endpoint": {
    "description": "Azure Data Lake Storage v2 endpoint override. Use when hierarchical namespaces are enabled on your storage account and you have set up a custom endpoint.",
    "nullable": true,
    "needs_restart": true,
    "visibility": "user",
    "is_secret": false,
    "type": "string"
  },
  "cloud_storage_azure_adls_port": {
    "description": "Azure Data Lake Storage v2 port override. See also `cloud_storage_azure_adls_endpoint`. Use when Hierarchical Namespaces are enabled on your storage account and you have set up a custom endpoint.",
    "nullable": true,
    "needs_restart": true,
    "visibility": "user",
    "is_secret": false,
    "type": "integer"
  },
  "cloud_storage_azure_container": {
    "description": "The name of the Azure container to use with Tiered Storage. If `null`, the property is disabled. The container must belong to cloud_storage_azure_storage_account.",
    "nullable": true,
    "needs_restart": true,
    "visibility": "user",
    "is_secret": false,
    "type": "string"
  },
  "cloud_storage_azure_managed_identity_id": {
    "description": "The managed identity ID to use for access to the Azure storage account. To use Azure managed identities, you must set `cloud_storage_credentials_source` to `azure_vm_instance_metadata`.",
    "nullable": true,
    "needs_restart": false,
    "visibility": "user",
    "is_secret": false,
    "type": "string"
  },
  "cloud_storage_azure_shared_key": {
    "description": "The shared key to be used for Azure Shared Key authentication with the Azure storage account configured by `cloud_storage_azure_storage_account`.  If `null`, the property is disabled. Redpanda expects this key string to be Base64 encoded.",
    "nullable": true,
    "needs_restart": false,
    "visibility": "user",
    "is_secret": true,
    "type": "string"
  },
  "cloud_storage_azure_storage_account": {
    "description": "The name of the Azure storage account to use with Tiered Storage. If `null`, the property is disabled.",
    "nullable": true,
    "needs_restart": true,
    "visibility": "user",
    "is_secret": false,
    "type": "string"
  }
}

[RafalKorepta]

LGTM, but I would rather change our documentation to points out that config.cluster is more flexible.

[chrisseto]

LGTM. I agree with @RafalKorepta.

[david-yu]

@andrewstucki Should we back port to at least 25.1.x?

[github-actions]

💚 All backports created successfully

Status	Branch	Result
✅	release/v25.1.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details