Skip to content

Conversation

@josielsouzanordcloud
Copy link
Contributor

Description

Resolves intermittent 403 authorisation errors during Terraform apply when creating storage queues and blobs. The issue was caused by Azure RBAC permission propagation delays between role assignment creation and storage queue validation.

Root Cause:

  • Storage queues were being validated immediately after RBAC role assignments
  • Azure RBAC permissions can take time to propagate across the platform
  • Terraform validation failed due to insufficient permissions during the propagation window
  • Pipeline would succeed on retry after permissions fully propagated

Solution:

  • Added time_sleep resource with 30-second delay after RBAC assignments
  • Ensures RBAC permissions have time to propagate before downstream operations
  • Maintains proper dependency chain: Storage Account → RBAC Assignments → Propagation Delay
  • No circular dependencies introduced

Changes:

  • Added time_sleep resource with 30s duration after RBAC role assignments
  • Proper dependency management for managed identity permissions
  • Eliminates race condition between permission assignment and validation

Jira link

[Link to Jira ticket]

Review notes

  • Addresses the intermittent 403 errors requiring pipeline retries
  • 30-second delay is conservative but ensures reliable deployment
  • No functional changes to infrastructure resources
  • Testing: Verify clean pipeline execution without authorization errors on first run
  • Follow-up: Monitor if 30s delay can be reduced based on propagation times

@github-actions
Copy link

The review app is available at this URL:
https://pr-420.manage-breast-screening.non-live.screening.nhs.uk
You must authenticate with Entra ID

@josielsouzanordcloud
Copy link
Contributor Author

Investigation will continue on a different PR

@josielsouzanordcloud josielsouzanordcloud deleted the DTOSS-11034-investigating-race-condition branch October 22, 2025 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant