CloudlogsWriteTest write test has failed with bad status code 500

[ocofaigh]

The Observability DA has been failing with this error:

2025/02/11 17:52:59 Terraform apply | Error: CreateTargetWithContext failed Your request to test a target has failed because the target details are not valid. The service could not write to the IBM Cloud Logs instance that is configured, because of CloudlogsWriteTest write test has failed with bad status code 500.
 2025/02/11 17:52:59 Terraform apply | {
 2025/02/11 17:52:59 Terraform apply |     "StatusCode": 400,
 2025/02/11 17:52:59 Terraform apply |     "Headers": {
 2025/02/11 17:52:59 Terraform apply |         "Cache-Control": [
 2025/02/11 17:52:59 Terraform apply |             "no-store, no-cache, must-revalidate, private, max-age=0"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Cf-Cache-Status": [
 2025/02/11 17:52:59 Terraform apply |             "DYNAMIC"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Cf-Ray": [
 2025/02/11 17:52:59 Terraform apply |             "9106320da8de477f-DFW"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Content-Length": [
 2025/02/11 17:52:59 Terraform apply |             "368"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Content-Security-Policy": [
 2025/02/11 17:52:59 Terraform apply |             "default-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; font-src 'self'"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Content-Type": [
 2025/02/11 17:52:59 Terraform apply |             "application/json; charset=utf-8"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Cross-Origin-Opener-Policy": [
 2025/02/11 17:52:59 Terraform apply |             "same-origin"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Cross-Origin-Resource-Policy": [
 2025/02/11 17:52:59 Terraform apply |             "same-origin"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Date": [
 2025/02/11 17:52:59 Terraform apply |             "Tue, 11 Feb 2025 17:52:59 GMT"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Expires": [
 2025/02/11 17:52:59 Terraform apply |             "Tue, 11 Feb 2025 17:52:55 GMT"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Referrer-Policy": [
 2025/02/11 17:52:59 Terraform apply |             "no-referrer"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Server": [
 2025/02/11 17:52:59 Terraform apply |             "cloudflare"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Set-Cookie": [
 2025/02/11 17:52:59 Terraform apply |             "__cfruid=a1bb6009789af104d9f4ddb6d57a0e756813df3e-1739296379; path=/; domain=.atracker.cloud.ibm.com; HttpOnly; Secure; SameSite=None"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "Strict-Transport-Security": [
 2025/02/11 17:52:59 Terraform apply |             "max-age=31557600; includeSubDomains"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "X-Content-Type-Options": [
 2025/02/11 17:52:59 Terraform apply |             "nosniff"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "X-Correlation-Id": [
 2025/02/11 17:52:59 Terraform apply |             "9c3bff20-d2de-4f9a-aa87-d1fd79d6bc28"
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "X-Request-Id": [
 2025/02/11 17:52:59 Terraform apply |             "51055ae0-21de-4cbc-b795-cd77569232ad"
 2025/02/11 17:52:59 Terraform apply |         ]
 2025/02/11 17:52:59 Terraform apply |     },
 2025/02/11 17:52:59 Terraform apply |     "Result": {
 2025/02/11 17:52:59 Terraform apply |         "errors": [
 2025/02/11 17:52:59 Terraform apply |             {
 2025/02/11 17:52:59 Terraform apply |                 "code": "cloudlogs_write_test_failed",
 2025/02/11 17:52:59 Terraform apply |                 "message": "Your request to test a target has failed because the target details are not valid. The service could not write to the IBM Cloud Logs instance that is configured, because of CloudlogsWriteTest write test has failed with bad status code 500."
 2025/02/11 17:52:59 Terraform apply |             }
 2025/02/11 17:52:59 Terraform apply |         ],
 2025/02/11 17:52:59 Terraform apply |         "status_code": 400,
 2025/02/11 17:52:59 Terraform apply |         "trace": "51055ae0-21de-4cbc-b795-cd77569232ad"
 2025/02/11 17:52:59 Terraform apply |     },
 2025/02/11 17:52:59 Terraform apply |     "RawResult": null
 2025/02/11 17:52:59 Terraform apply | }
 2025/02/11 17:52:59 Terraform apply | 
 2025/02/11 17:52:59 Terraform apply | 
 2025/02/11 17:52:59 Terraform apply |   with module.observability_instance.module.activity_tracker.ibm_atracker_target.atracker_cloud_logs_targets["observab-8e4d-cloud-logs-target"],
 2025/02/11 17:52:59 Terraform apply |   on .terraform/modules/observability_instance/modules/activity_tracker/main.tf line 84, in resource "ibm_atracker_target" "atracker_cloud_logs_targets":
 2025/02/11 17:52:59 Terraform apply |   84: resource "ibm_atracker_target" "atracker_cloud_logs_targets" {
 2025/02/11 17:52:59 Terraform apply | 
 2025/02/11 17:52:59 �[1m�[31mTerraform APPLY error: Terraform APPLY errorexit status 1�[39m�[0m

The DA is using the following inputs along with the rest of the defaults:

{
  "ibmcloud_api_key": $VALIDATION_APIKEY,
  "resource_group_name": $PREFIX,
  "existing_kms_instance_crn": $HPCS_US_SOUTH_CRN,
  "region": "us-south",
  "enable_platform_logs": false,
  "enable_platform_metrics": false,
  "prefix": $PREFIX
}

[ocofaigh]

Looks like its a timing issue - it passes on retry. I have reached out to ICL team

[ocofaigh]

Still seeing this intermittently. A slightly different error message this time:

 2025/02/12 15:14:59 Terraform apply |     "Result": {
 2025/02/12 15:14:59 Terraform apply |         "errors": [
 2025/02/12 15:14:59 Terraform apply |             {
 2025/02/12 15:14:59 Terraform apply |                 "code": "cloudlogs_write_test_failed",
 2025/02/12 15:14:59 Terraform apply |                 "message": "Your request to test a target has failed because the target details are not valid. The service could not write to the IBM Cloud Logs instance that is configured, because of Post \"https://373bd476-29e3-40f4-a721-74c6495a6f77.ingress.us-south.logs.cloud.ibm.com/logs/v1/singles\": net/http: TLS handshake timeout."
 2025/02/12 15:14:59 Terraform apply |             }
 2025/02/12 15:14:59 Terraform apply |         ],
 2025/02/12 15:14:59 Terraform apply |         "status_code": 400,
 2025/02/12 15:14:59 Terraform apply |         "trace": "d59b72e0-aca8-49df-bb2b-383fffe17a78"
 2025/02/12 15:14:59 Terraform apply |     },

[iamar7]

@ocofaigh Do we want to keep this open since the support case was closed related to this issue?

[ocofaigh]

@iamar7 Issue was reproduced today, so I opened a new support case. I will move this to the cloud logs repo