retry logic not working in enable_disable_ocp_console.sh

[ocofaigh]

The following error was seen (which can sometimes happen when RBAC synchronisation is not complete on a cluster yet):

│ Error: local-exec provisioner error
│ 
│   with module.ocp_base.null_resource.ocp_console_management,
│   on .terraform/modules/ocp_base/main.tf line 468, in resource "null_resource" "ocp_console_management":
│  468:   provisioner "local-exec" {
│ 
│ Error running command
│ '.terraform/modules/ocp_base/scripts/enable_disable_ocp_console.sh': exit
│ status 1. Output: =========================================
│ Enabling the OpenShift Console
│ Attempt 1 of 10: Applying OpenShift Console patch...
│ Error from server (Forbidden): consoles.operator.openshift.io "cluster" is
│ forbidden: User "IAM#goldeneye.development@ibm.com" cannot get resource
│ "consoles" in API group "operator.openshift.io" at the cluster scope
│ Failed to apply patch. Retrying in 5s...
│ 
╵}

As you can see the error says "Retrying in 5s..." however it didnt seem to do any retry attempt (full log here).
I also think we should maybe update the depends_on value to be depends_on = [null_resource.confirm_network_healthy] as this might reduce the chance of the script executing before the RBAC synchronisation is complete.

[imprateeksh]

@ocofaigh - I think there is no issue with the retry logic. I verified this earlier as well.
To verify this, I replaced the command with a wrong command (i.e. ech) and retry attempts can be seen.

The error in provisioner is RBAC related as you rightly mentioned (cluster service forbidden..)
Will verify further.

Separate Issue I found in the script is related to waiting time in retries but not the retry logic:

Using existing changes	When removed exponential backoff (faster)

I have removed the exponential backoff and kept the retry attempts with wait time always to be 60.
Raised PR here

[imprateeksh]

Closing the issue as changed are now merged.