[Bug] Key Vault Requests Fail After Re-Creating Key Vault in New Tenant

[dockoneal]

Library name and version

Azure.Identity 1.12.0, Azure.Security.KeyVault.Keys 4.6.0

Describe the bug

Azure.Security.KeyVault maintains a dictionary of key vault URIs to ChallengeParameters [1]. If a request has already been made to key vault, the challenge parameters determined by the SDK are stored in the cache and used for future requests. The tenant ID determined by the SDK is prioritized over the one provided by the caller [2].

The ChallengeParameter cache is never invalidated. If the key vault tenant ID changes due to subscription migration, then key vault requests will fail until the process restarts.

This behavior is affecting Ultra Disk storage. Customers cannot associate their key vault with a migrated disk due to this caching behavior. Restarting the service is not an ideal solution due to performance and availability SLAs.

[1]

azure-sdk-for-net/sdk/keyvault/Azure.Security.KeyVault.Shared/src/ChallengeBasedAuthenticationPolicy.cs

Line 22 in fca5abb

private static readonly ConcurrentDictionary<string, ChallengeParameters> s_challengeCache = new();

[2] https://github.com/Azure/azure-sdk-for-net/blob/fca5abb395e85278232b81eeb04d256c55d62c06/sdk/identity/Azure.Identity/src/TenantIdResolver.cs#L40C17-L40C58

Expected behavior

Should be able to make key vault requests after creating a new key vault with the same name in a different tenant without restarting the application.

Actual behavior

Creating a key vault in a new tenant with the same name as the old tenant causes a conflict with the cached tenant ID making key vault requests impossible.

Reproduction Steps

1. Have a key vault with a managed identity so you can make wrap/unwrap requests
2. Delete the key vault and re-create the key vault with the same name in another tenant and create a new identity to access the key vault.
3. Update the key vault, client ID, tenant ID, and credential without restarting your process
4. Key vault requests fail

Environment

• Microsoft Windows Server 2022 Datacenter 10.0.20348
• .net framework 4.7.2

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @adarce @alukovenko-ms @cheathamb36 @chlowell @dhanvikapila @dragav @fponzi-ms @FRL200 @jackrichins @kasturi3k @ksemenov-gh @kurtzeborn @mancycakv @marekkarbarz @melissamserv @ncamillucci @osmuller @rahulalapati43 @seanbamsft @sebansal @vakuncha @vickm.

[github-actions]

Hi @dockoneal, I'm an AI Support assistant here to help with your issue. While the Azure SDK team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

• The behavior described is due to how the SDK caches challenge parameters per key vault URI (source), which means that once a challenge is cached (including its tenant ID), subsequent requests continue to use it.
• One suggestion is to avoid reusing an existing client instance when you know the vault has been re-created in a different tenant. Recreating your Key Vault client (for example, a KeyClient or SecretClient) forces the SDK to perform a new challenge and acquire updated parameters.
• Ensure that any client-level caching in your application is refreshed when configuration updates occur (for example, when tenant or identity changes) so that the new instance correctly picks up the correct tenant from the challenge (details).

The Azure SDK team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!