No longer possible to deploy OCP DA using trusted profile

[ocofaigh]

After migrating from a null_resource script to using ibm_container_api_key_reset in #795 it has been reported that cluster provisioning fails with below error when they deploy the DA using a trusted profile:

2025/10/20 14:28:57 Terraform apply |   with module.ocp_base.ibm_container_vpc_cluster.cluster[0],
 2025/10/20 14:28:57 Terraform apply |   on ../../main.tf line 140, in resource "ibm_container_vpc_cluster" "cluster":
 2025/10/20 14:28:57 Terraform apply |  140: resource "ibm_container_vpc_cluster" "cluster" {
 2025/10/20 14:28:57 Terraform apply | 
 2025/10/20 14:28:57 Terraform apply | ---
 2025/10/20 14:28:57 Terraform apply | id: terraform-40a3a1fe
 2025/10/20 14:28:57 Terraform apply | summary: 'Request failed with status code: 400, ServerErrorResponse:
 2025/10/20 14:28:57 Terraform apply | {"incidentID":"a0701484-1dcf-a82e-90b1-455464f7064c","code":"E3595","description":"The
 2025/10/20 14:28:57 Terraform apply |   entitlement ''cloud_pak'' was not found. Specify ''ocp_entitled'' to search for
 2025/10/20 14:28:57 Terraform apply |   any supported license or entitlement. If no match is found, then you do not have
 2025/10/20 14:28:57 Terraform apply |   a supported license or entitlement.","type":"BadRequest"}'
 2025/10/20 14:28:57 Terraform apply | severity: error
 2025/10/20 14:28:57 Terraform apply | resource: ibm_container_vpc_cluster
 2025/10/20 14:28:57 Terraform apply | operation: create
 2025/10/20 14:28:57 Terraform apply | component:
 2025/10/20 14:28:57 Terraform apply |   name: github.com/IBM-Cloud/terraform-provider-ibm
 2025/10/20 14:28:57 Terraform apply |   version: 1.83.3
 2025/10/20 14:28:57 Terraform apply | ---

The error seems to indicate that there may be an issue using 'cloud_pak' as an entitlement when deploying using a Trusted Profile. We should test this out.

We should also expose a boolean to allow advanced users to skip the ibm_container_api_key_reset as a workaround until the backend race condition is addressed by IKS or until IBM-Cloud/terraform-provider-ibm#6468 is addressed by provider

[ocofaigh]

From consumer who reported the issue:

I do not think the issue is with the entitlement - that's just a red herring. After that error I did a reset via CLI using my own user identity and redeployment works fine. On the second run the ibm_container_api_key_reset resource actually is not triggering (since there is no change in parameters), so it just keeps the apikey and cluster provisioning works fine - with the same cloud_pak entitlement.
I did not have a chance to fiddle with other entitlement options, but based on the fact that it works with TP when the apikey is not messed up by the reset, I think entitlement is not the root cause

[ocofaigh]

@Aashiq-J Another issue when using trusted profile. The un-deploy fails with:

 2025/10/23 05:38:31 Terraform destroy | Error: local-exec provisioner error
 2025/10/23 05:38:31 Terraform destroy | 
 2025/10/23 05:38:31 Terraform destroy |   with terraform_data.delete_secrets[0],
 2025/10/23 05:38:31 Terraform destroy |   on main.tf line 263, in resource "terraform_data" "delete_secrets":
 2025/10/23 05:38:31 Terraform destroy |  263:   provisioner "local-exec" {
 2025/10/23 05:38:31 Terraform destroy | 
 2025/10/23 05:38:31 Terraform destroy | Error running command './scripts/delete_secrets.sh
 2025/10/23 05:38:31 Terraform destroy | 76ed4f0f-3f02-89d2-39b8-a14031aab329 private
 2025/10/23 05:38:31 Terraform destroy | 3d04d5a3-940a-4d62-8693-df20039a78a4 us-south private': exit status 1.
 2025/10/23 05:38:31 Terraform destroy | Output: "Provided API key could not be found."
 2025/10/23 05:38:31 Terraform destroy | Could not obtain an IAM access token

Are you able to reproduce this?
From the consumer:

I suspect that the apikey parameter to the delete_secrets resource gets stored in the state from an initial execution of the terraform. Then days later you try to run undeploy, but that original apikey is already recycled by the Projects trusted profile cleanup process.
So probably if you switch the script to using a token and make sure it's not get cached in the state, it might work.