Create azuread_application periodically fails - unexpected status 409 (409 Conflict) with error: Error_MsaAppDoesNotExist: The corresponding MSA application does not exist.

[hkrazure]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritise this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritise the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureAD Provider) Version

I have tried this with multiple version of terraform and multiple versions of the azuread provider - they are all unstable:

terraform: 1.11.0, 1.12.0, 1.12.2, 1.13.0-rc1, 1.13.0
azuread: 2.51.0, 3.0.0, 3.1.0, 3.5.0

Affected Resource(s)

• azuread_application

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "4.0.0"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "3.5.0"
    }
  }
  required_version = ">= 1.1.0"
}
 
provider "azurerm" {
  features {}
  resource_provider_registrations = "none"
  subscription_id                 = "XXXXXXXXXXX"
}
 
provider "azuread" {
  tenant_id     = "XXXXXXXXXXX"
  client_id     = "XXXXXXXXXXX"
  client_secret = "XXXXXXXXXXX"
}
 
data "azuread_application_published_app_ids" "well_known" {}
 
resource "azuread_application" "service-application" {
  display_name = "test"
  sign_in_audience = "AzureADandPersonalMicrosoftAccount"
 
  api {
    requested_access_token_version = 2
  } 
}

Debug Output

https://gist.github.com/hkrazure/fbb09c916b38a1d7789199873318b68f

Panic Output

Expected Behavior

The app registration should be created every time instead of failing some of the time.

Actual Behavior

It fails with the following message:

│ unexpected status 409 (409 Conflict) with error: Error_MsaAppDoesNotExist:
│ The corresponding MSA application does not exist.

Apparently creating an app registration resource results in two calls from terraform - a post and a patch.

It appears that the patch request fails because azure schedules creating some resources asynchronously - but this is not awaited before the patch request is executed which results in the failure.

Steps to Reproduce

1. Initialize terraform
2. Run terraform plan
3. Run terraform apply

1. terraform apply

Important Factoids

This is an issue with sign_in_audience = "AzureADandPersonalMicrosoftAccount".

It seems to work with other sign_in_audience values.

But we need this specific one in our setup.

References

I can see there is an issue for microsoft graph that fails with the samme message:

microsoftgraph/msgraph-sdk-powershell#3380

• #0000

[pkaluckimirion]

Got the same error, application registration gets created but apply fails and terraform wants to recreate the resource, after manually importing the registration into state it's working.

[tlaukkanen]

Have started to get the same error also with azurerm 4.35.0 and azuread 3.4.0:

│ Error: Failed to patch application with object ID "6e45...355f" after creating
│ 
│   with module.application.azuread_application.sampleapp,
│   on modules/application/main.tf line 300, in resource "azuread_application" "sampleapp":
│  300: resource "azuread_application" "sampleapp" {
│ 
│ unexpected status 409 (409 Conflict) with error: Error_MsaAppDoesNotExist:
│ The corresponding MSA application does not exist.

[Dev0psDavid]

Also happening to me for the last three days:
Error: ‌Failed to patch application with object ID "..." after creating‌ with azuread_application.aad_ky...r_client,‌ on application-aad.tf line 648, in resource "azuread_application" "aad_ky...r_client":‌ 648: resource "azuread_application" "aad_ky...r_client" ‌{‌ApplicationsClient.BaseClient.Patch(): unexpected status 409 with OData‌
‌error: Error_MsaAppDoesNotExist: The corresponding MSA application does not‌ ‌exist.‌

[hkrazure]

It worked for me if I switched to version 2.8.0 of azuread. But it's not exactly a long term solution.

[brian-sqq]

We had created a ticket with Azure, and after a week of investigation and baton passing, here is their official response 🙄.

The 409 error occurs due to a conflict with the current state of the object. You need to add back-off retry logic to handle this. For example, when you create an app and try to update it immediately, the app may not be ready yet. This should be managed on your end. For more details on graph 409 errors, refer to: https://learn.microsoft.com/en-us/graph/errors#:~:text=409,delay%20between%20retries.

Looks like this provider will need a change.

[kornelkalnai]

For some reason, I can see that it is already working (with 3.5.0 version) on the Azure side, as if it had been fixed.
I can't find any release notes or anything from Azure about this.
Could someone try it out and confirm (or refute) my statement?

Edit: Either it works or it doesn't... I tried it 5 times, and the first 4 times, 3/3 apps were created. The 5th time, only 1/3.

[tlaukkanen]

Edit: Either it works or it doesn't... I tried it 5 times, and the first 4 times, 3/3 apps were created. The 5th time, only 1/3.

Yes, we are seeing the same - not yet fixed. Works randomly but fails more often than works in our case.

[brian-sqq]

It makes sense that it works sporadically. Here is the final word from the Azure support team.

This is not an issue specific to the Microsoft Graph AAD Dev team, but rather a limitation common to distributed systems. Similar challenges are faced by AWS and Google, and customers working with distributed systems may experience this as well. For further details, please refer to the following resource:

https://learn.microsoft.com/en-us/troubleshoot/entra/entra-id/app-integration/404-not-found-error-manage-objects-microsoft-graph

We're going to have to work around it, so hopefully my PR will be accepted, or some alternative implementation will be proposed.