Add retry to logic in reset_iks_api_key.sh ?

[ocofaigh]

Can we add a retry anywhere in our code to handle intermittent issues like this?

module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Creating...
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Provisioning with 'local-exec'...
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): Executing: ["/bin/bash" "-c" ".terraform/modules/landing_zone.landing_zone.cluster/scripts/reset_iks_api_key.sh us-south 62fd2a41c7fd4e5fa60d0d16af886c98 false default"]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [10s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [10s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [20s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [30s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [40s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key: Still creating... [50s elapsed]
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): ERROR:: FAILED TO RESET THE IAM API KEY
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): HTTP/2 500
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): content-type: application/json; charset=utf-8
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): cross-origin-resource-policy: cross-origin
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-carrier: prod-dal10-carrier105
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-correlation-id: 5615502e-c3c1-44d9-b1b6-2a0d86790f9d
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-hostname: armada-global-api-6fc8fc7bcd-pkk7c
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-region: us-south
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-request-id: b7ab3d98-6d78-424c-a02b-582fe14fbfa6
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): content-length: 159
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): content-security-policy: default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; font-src 'self'
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): strict-transport-security: max-age=31536000; includeSubDomains
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-content-type-options: nosniff
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-frame-options: DENY
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-permitted-cross-domain-policies: master-only
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): x-xss-protection: 1; mode=block
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): expires: Thu, 03 Apr 2025 23:12:30 GMT
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): cache-control: max-age=0, no-cache, no-store
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): pragma: no-cache
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): date: Thu, 03 Apr 2025 23:12:30 GMT
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): server-timing: cdn-cache; desc=MISS
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): server-timing: edge; dur=120
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): server-timing: origin; dur=50070
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): server-timing: ak_p; desc="1743721900013_399640271_1881643625_5010228_4736_1_70_15";dur=1
module.landing_zone.module.landing_zone.module.cluster["slz-d560-management-cluster"].null_resource.reset_api_key (local-exec): {"incidentID":"b7ab3d98-6d78-424c-a02b-582fe14fbfa6","code":"G0002","description":"An internal error occurred while processing your request.","type":"General"}

[ocofaigh]

Another example where retry would of helped:

│ Error: local-exec provisioner error
│ 
│   with module.ocp_base.null_resource.reset_api_key,
│   on .terraform/modules/ocp_base/main.tf line 309, in resource "null_resource" "reset_api_key":
│  309:   provisioner "local-exec" {
│ 
│ Error running command
│ '.terraform/modules/ocp_base/scripts/reset_iks_api_key.sh us-south
│ 93abc958196f41099600d6c7c2025105 false default': exit status 1. Output:
│ ERROR:: FAILED TO RESET THE IAM API KEY
│ HTTP/2 400 
│ content-type: application/json; charset=utf-8
│ cross-origin-resource-policy: cross-origin
│ x-carrier: prod-dal10-carrier105
│ x-correlation-id: a24b084d-e7c0-4a51-9380-a0e7e4d2681d
│ x-hostname: armada-global-api-66d4495df9-xbm2z
│ x-region: us-south
│ x-request-id: 14469a6b-4fb9-464f-b68a-0ba28ea452be
│ content-length: 182
│ content-security-policy: default-src 'none'; script-src 'self'
│ 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self' data:;
│ style-src 'self' 'unsafe-inline'; font-src 'self'
│ strict-transport-security: max-age=31536000; includeSubDomains
│ x-content-type-options: nosniff
│ x-frame-options: DENY
│ x-permitted-cross-domain-policies: master-only
│ x-xss-protection: 1; mode=block
│ expires: Tue, 13 May 2025 23:37:33 GMT
│ cache-control: max-age=0, no-cache, no-store
│ pragma: no-cache
│ date: Tue, 13 May 2025 23:37:33 GMT
│ server-timing: cdn-cache; desc=MISS
│ server-timing: edge; dur=242
│ server-timing: origin; dur=884
│ server-timing: ak_p;
│ desc="1747179452567_389287748_18978981_112463_8141_30_52_15";dur=1
│ 
│ {"incidentID":"14469a6b-4fb9-a64f-b68a-0ba28ea452be","code":"A03e9b","description":"Failed
│ to create an API key with IAM. Revise your request and try
│ again.","type":"Authentication"}

[Aashiq-J]

PR merged. Closing issue.