[Bug] elasticstack_fleet_integration_policy unable to handle IDs on isSecretRef

[kenwaikinwongfwd]

Describe the bug
resource "elasticstack_fleet_agent_policy" unable to handle field "ids" on "isSecretRef" for some integration policies that have a list of IDs instead of single ID.

Here is my example of SQL input integration policy from API showing secret ref of "ids" instead of "id":

{
  "package": {
    "name": "sql",
    "version": "1.1.0"
  },
  "name": "test-name",
  "namespace": "test-namespace",
  "description": "Test SQL integration",
  "policy_ids": [
    "my_agent_policy_id"
  ],
  "vars": {},
  "inputs": {
    "sql-sql/metrics": {
      "enabled": true,
      "streams": {
        "sql.sql": {
          "enabled": true,
          "vars": {
            "hosts": {
              "ids": [
                "Zgr3ZJgBz9OcmZgWR1vo"
              ],
              "isSecretRef": true
            },
            "period": "5m",
            "driver": "mssql",
            "sql_queries": "- query: SHOW GLOBAL STATUS LIKE 'Innodb_system%'\n  response_format: variables\n        \n",
            "merge_results": false,
            "ssl": "#",
            "processors": "#",
            "data_stream.dataset": "sql"
          }
        }
      }
    }
  }
}

Error when running terraform apply

╷│ Error: Plugin did not respond
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin6.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵

Stack trace from the terraform-provider-elasticstack_v0.11.15 plugin:

panic: interface conversion: interface {} is nil, not string

github.com/elastic/terraform-provider-elasticstack/internal/fleet/integration_policy.HandleReqRespSecrets({0x1624568, 0x4000750a80}, {0x4000774b78, 0x0, 0x0, 0x400043ad88, {0x4000604d98, 0x11}, 0x4000774b90, 0x0, ...}, ...)
   github.com/elastic/terraform-provider-elasticstack/internal/fleet/integration_policy/secrets.go:172 +0x1390
github.com/elastic/terraform-provider-elasticstack/internal/fleet/integration_policy.(*integrationPolicyResource).Create(0x400043a9d0, {0x1624568, 0x4000750a80}, {{{{0x162b578, 0x4000770120}, {0x11d7e00, 0x40007602d0}}, {0x162e9c8, 0x40005dc3c0}}, {{{0x162b578, ...}, ...}, ...}, ...}, ...)
   github.com/elastic/terraform-provider-elasticstack/internal/fleet/integration_policy/create.go:37 +0x380
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).CreateResource(0x4000226488, {0x1624568, 0x4000750a80}, 0x40005ab458, 0x40005ab430)
   github.com/hashicorp/terraform-plugin-framework@v1.14.1/internal/fwserver/server_createresource.go:101 +0x3fc
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ApplyResourceChange(0x4000226488, {0x1624568, 0x4000750a80}, 0x4000536780, 0x40005ab550)
   github.com/hashicorp/terraform-plugin-framework@v1.14.1/internal/fwserver/server_applyresourcechange.go:57 +0x380
github.com/hashicorp/terraform-plugin-framework/internal/proto6server.(*Server).ApplyResourceChange(0x4000226488, {0x1624568?, 0x4000750990?}, 0x40005366e0)
   github.com/hashicorp/terraform-plugin-framework@v1.14.1/internal/proto6server/server_applyresourcechange.go:55 +0x2dc
github.com/hashicorp/terraform-plugin-mux/tf6muxserver.(*muxServer).ApplyResourceChange(0x400022b080, {0x1624568?, 0x40007506c0?}, 0x40005366e0)
   github.com/hashicorp/terraform-plugin-mux@v0.18.0/tf6muxserver/mux_server_ApplyResourceChange.go:36 +0x17c
github.com/hashicorp/terraform-plugin-go/tfprotov6/tf6server.(*server).ApplyResourceChange(0x40004080a0, {0x1624568?, 0x4000695b60?}, 0x4000426cb0)
   github.com/hashicorp/terraform-plugin-go@v0.26.0/tfprotov6/tf6server/server.go:866 +0x294
github.com/hashicorp/terraform-plugin-go/tfprotov6/internal/tfplugin6._Provider_ApplyResourceChange_Handler({0x13870c0, 0x40004080a0}, {0x1624568, 0x4000695b60}, 0x4000434f00, 0x0)
   github.com/hashicorp/terraform-plugin-go@v0.26.0/tfprotov6/internal/tfplugin6/tfplugin6_grpc.pb.go:611 +0x1c0
google.golang.org/grpc.(*Server).processUnaryRPC(0x4000440000, {0x1624568, 0x4000695b00}, 0x400007ed80, 0x40002f7b90, 0x2379c78, 0x0)
   google.golang.org/grpc@v1.69.4/server.go:1392 +0xc38
google.golang.org/grpc.(*Server).handleStream(0x4000440000, {0x1625488, 0x4000363860}, 0x400007ed80)
   google.golang.org/grpc@v1.69.4/server.go:1802 +0x900
google.golang.org/grpc.(*Server).serveStreams.func2.1()
   google.golang.org/grpc@v1.69.4/server.go:1030 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 23
   google.golang.org/grpc@v1.69.4/server.go:1041 +0x138

Error: The terraform-provider-elasticstack_v0.11.15 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

From ./internal/fleet/integration_policy/secrets.go in the terraform-provider-elasticstack repo, there is no handle for "ids" instead only handle"id".

To Reproduce

1. Create an integration policy of SQL Input integration.

resource "elasticstack_fleet_integration_policy" "sample_sql_input" {
  name                = "sample-sql-input-policy"
  namespace           = "default"
  description         = "A sample SQL input integration policy"
  agent_policy_id     = var.agent_policy_id
  integration_name    = "sql"
  integration_version = "1.1.0"
 
  input {
    input_id  = "sql-sql/metrics",
      enabled   = true,
      vars_json = "{}",
      streams_json = jsonencode({
        "sql.sql" : {
          "enabled" : var.enabled,
          "vars" : {
            "hosts" : ["root:test@tcp(127.0.0.1:3306)/"],
            "period" : "1m",
            "driver" : "mysql",
            "sql_queries" : "- query: SHOW GLOBAL STATUS LIKE 'Innodb_system%'\n  response_format: variables\n        \n",
            "merge_results" : false,
            "ssl" : "",
            "data_stream.dataset" : "sql",
            "processors" : ""
          }
        }
      })
  }
}

2. Run terraform plan then terraform apply

Expected behavior
Integration policy created successfully.

Versions (please complete the following information):

• OS: Linux
• Terraform Version v1.11.3
• Provider version v0.11.15
• Elasticsearch Version v9.1.0

[aleks-ivk]

Is there any timeline on when the bug will be fixed? This is a rather critical integration for our Database team that uses it to gather insights into customers' databases. Being that they cannot create new policies atm, they are completely blocked from onboarding 50+ of our customers' DBs.