sysdig_secure_ml_policy: Using the resource may crash the Terraform process

[r2bit]

Use of sysdig_secure_ml_policy causes a perpetual diff that cannot be resolved w/o deleting the resource from Terraform state:

Error: Rules not found
 
   with module.main.sysdig_secure_ml_policy.base,
   on ../../runtime.tf line 16, in resource "sysdig_secure_ml_policy" "base":
   16: resource "sysdig_secure_ml_policy" "base" {
 
Operation failed: failed running terraform plan (exit 1)

Once deleted from Sysdig, the plugin crashes:

Stack trace from the terraform-provider-sysdig_v1.37.2 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdf1946]

goroutine 213 [running]:
github.com/draios/terraform-provider-sysdig/sysdig.setTFResourceBaseAttrs(0xc0000fa780, {0x0?, {0x0?, 0x3100000000eb4520?, 0xc000c14930?}})
github.com/draios/terraform-provider-sysdig/sysdig/tfresource.go:75 +0x26
github.com/draios/terraform-provider-sysdig/sysdig.Reduce[...](0xc0000fa780?, {0x0, {0x0?, 0xfd5eed?, 0x2?}}, {0xc000010f00?, 0x3, 0xc0000fa780})
github.com/draios/terraform-provider-sysdig/sysdig/tfresource.go:46 +0x74
github.com/draios/terraform-provider-sysdig/sysdig.init.Reducer[...].func10({0x0?, {0x0?, 0x9f03d2?, 0xc000807278?}})
github.com/draios/terraform-provider-sysdig/sysdig/tfresource.go:39 +0x45
github.com/draios/terraform-provider-sysdig/sysdig.mlPolicyToResourceData(...)
github.com/draios/terraform-provider-sysdig/sysdig/resource_sysdig_secure_ml_policy.go:88
github.com/draios/terraform-provider-sysdig/sysdig.resourceSysdigMLPolicyRead({0x1245368, 0xc000404bd0}, 0xc0000fa780, {0xf85ba0?, 0xc0005ae140?})
github.com/draios/terraform-provider-sysdig/sysdig/resource_sysdig_secure_ml_policy.go:155 +0x26c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0003d4380, {0x12452c0, 0xc00072fef0}, 0xc0000fa780, {0xf85ba0, 0xc0005ae140})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:724 +0x119
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0003d4380, {0x12452c0, 0xc00072fef0}, 0xc0007dc340, {0xf85ba0, 0xc0005ae140})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:1015 +0x51a
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00020bec0, {0x12452c0?, 0xc00072fe30?}, 0xc00012f840)
github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:613 +0x4aa
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc0005af2c0, {0x12452c0?, 0xc00072f6b0?}, 0xc000a4df20)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:748 +0x46f
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0xf9c000, 0xc0005af2c0}, {0x12452c0, 0xc00072f6b0}, 0xc000404690, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00021e000, {0x124a6e0, 0xc000718000}, 0xc0007a2480, 0xc00062acf0, 0x19353f0, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1340 +0xd16
google.golang.org/grpc.(*Server).handleStream(0xc00021e000, {0x124a6e0, 0xc000718000}, 0xc0007a2480, 0x0)
google.golang.org/grpc@v1.51.0/server.go:1713 +0x9da
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.51.0/server.go:965 +0x87
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 22
google.golang.org/grpc@v1.51.0/server.go:963 +0x226

Error: The terraform-provider-sysdig_v1.37.2 plugin crashed!

Terraform code:

resource "sysdig_secure_ml_policy" "base" {
  name        = "Crypto Mining detection"
  description = "Policy that enables Crypto Mining detection using advanced ML capabilities"

  severity = 3
  enabled  = true

  scope = "kubernetes.cluster.name in (\"X\")"

  rule {
    description = "Crypto Mining detection"

    cryptomining_trigger {
      enabled   = true
      threshold = 1
    }
  }

  notification_channels = [Y]
}

[dk-teranishis]

I am also facing this issue.
I'm happy to provide information if you need it.

Thank you!

[IgorEulalio]

Hey @r2bit @dk-teranishis, could you please provide more details on how you're reproducing that?

I have built the plugin in your same version, v1.37.2 and so far, I can't replicate it.

After deploying it I've triggered plans and applies, and there are no changes on the infrastructure.

[raserma]

Hey @r2bit @dk-teranishis, could you please provide more details on how you're reproducing that?

I have built the plugin in your same version, v1.37.2 and so far, I can't replicate it.

After deploying it I've triggered plans and applies, and there are no changes on the infrastructure.

Hi @IgorEulalio
I'm also experiencing this issue consistently with the Sysdig Terraform provider (version 1.41.0).

The only workaround I've found is to:

• Remove the resource from the Terraform state.
• Manually delete the Runtime Policy from the Sysdig UI.
• Run terraform apply again to recreate the policy.

This resolves the issue temporarily, but it reoccurs the next day, causing Terraform to fail during plan or apply.

Reproduction Steps

• Sysdig provider: 1.41.0
• Triggered Terraform plans and applies, resulting in no infrastructure changes on the first day.
• The "Rules not found" error occurs the next day during the next Terraform plan/apply.

Terraform Code

resource "sysdig_secure_aws_ml_policy" "sysdig_runtime_aws_ml" {
  name        = "Anomalous Console Login Detection"
  description = "The Anomalous Console Login Detection feature triggers an event when a login seems unusual compared to normal patterns. Setting a higher confidence level decreases false alarms but may miss actual anomalous logins."
  enabled     = true
  severity    = 1

  rule {
    description = "Test ML Rule Description"

    anomalous_console_login {
      threshold = 1
      enabled   = false
    }
  }

  notification_channels = [data.sysdig_secure_notification_channel_slack.critical_notifications.id]
}

Relevant Logs

I've attached anonymized logs for reference (trimmed for clarity):

TF_LOG=debug terraform plan
2024-12-30T10:57:01.855+0100 [INFO]  Terraform version: 1.4.6
2024-12-30T10:57:01.855+0100 [DEBUG] using github.com/hashicorp/go-tfe v1.21.0
2024-12-30T10:57:01.855+0100 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2024-12-30T10:57:01.855+0100 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2024-12-30T10:57:01.855+0100 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2024-12-30T10:57:01.855+0100 [DEBUG] using github.com/zclconf/go-cty v1.12.1
2024-12-30T10:57:01.855+0100 [INFO]  Go runtime version: go1.19.6
2024-12-30T10:57:01.855+0100 [INFO]  CLI args: []string{"/Users/user/.asdf/installs/terraform/1.4.6/bin/terraform", "plan"}
2024-12-30T10:57:01.856+0100 [DEBUG] Attempting to open CLI config file: /Users/user/.terraformrc
2024-12-30T10:57:01.856+0100 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-12-30T10:57:01.856+0100 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-12-30T10:57:01.856+0100 [DEBUG] ignoring non-existing provider search directory /Users/user/.terraform.d/plugins
2024-12-30T10:57:01.856+0100 [DEBUG] ignoring non-existing provider search directory /Users/user/Library/Application Support/io.terraform/plugins
2024-12-30T10:57:01.856+0100 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2024-12-30T10:57:01.856+0100 [INFO]  CLI command args: []string{"plan"}
2024-12-30T10:57:01.863+0100 [INFO]  Attempting to use session-derived credentials
2024-12-30T10:57:02.282+0100 [INFO]  Successfully derived credentials from session
2024-12-30T10:57:02.282+0100 [INFO]  AWS Auth provider used: "SSOProvider"
2024-12-30T10:57:02.282+0100 [INFO]  Attempting to AssumeRole arn:aws:iam::account-id:role/role (SessionName: "", ExternalId: "")
[...]

sysdig_secure_aws_ml_policy.sysdig_runtime_aws_ml: Refreshing state... [id=10320992]
2024-12-30T10:57:04.652+0100 [INFO]  provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:04 [DEBUG] GET /api/v2/policies/10320992 HTTP/1.1
Host: eu1.app.sysdig.com
User-Agent: SysdigTerraform/1.41.0
Authorization: Bearer HASHED
Content-Type: application/json
Sysdig-Provider: Terraform
Accept-Encoding: gzip
: timestamp=2024-12-30T10:57:04.652+0100
2024-12-30T10:57:04.652+0100 [DEBUG] provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:04 [DEBUG] GET https://eu1.app.sysdig.com/api/v2/policies/10320992
2024-12-30T10:57:04.654+0100 [INFO]  provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:04 [DEBUG] HTTP/1.1 200 OK
Content-Length: 741
Cache-Control: private
Content-Type: application/json
Date: Mon, 30 Dec 2024 09:57:04 GMT
Server: envoy
Set-Cookie: INGRESSCOOKIEAPI="HASHED"; Max-Age=86400; HttpOnly
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 969
X-Ratelimit-Reset: 1735552667
X-Request-Id: 6903add2-5fdc-4a27-a5f0-022b45fe2590
[...]

{"id":10320992,"name":"Anomalous Console Login Detection","version":1,"type":"aws_machine_learning","description":"The Anomalous Console Login Detection feature triggers an event when a login seems unusual compared to normal patterns. Setting a higher confidence level decreases false alarms but may miss actual anomalous logins.","severity":1,"enabled":true,"notificationChannelIds":[43751],"actions":null,"scope":null,"ruleNames":["10320992_AWSMLRule"],"rules":[{"ruleName":"10320992_AWSMLRule","enabled":true,"modifiedOn":"2024-12-24T09:28:15.531822Z","createdOn":"2024-12-24T09:28:15.531822Z"}],"origin":"Secure UI","versionId":"0.0.0","createdOn":1735032495308,"modifiedOn":1735032495308,"alertId":1346190,"alertVersion":1,"teamId":20009944,"templateId":0,"templateVersion":"","isDefault":false}: timestamp=2024-12-30T10:57:04.745+0100
2024-12-30T10:57:04.746+0100 [INFO]  provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:04 [DEBUG] GET /api/policies/v3/rules/groups?names=10320992_AWSMLRule HTTP/1.1
Host: eu1.app.sysdig.com
User-Agent: SysdigTerraform/1.41.0
Authorization: Bearer HASHED
Content-Type: application/json
Sysdig-Provider: Terraform
Accept-Encoding: gzip
: timestamp=2024-12-30T10:57:04.745+0100
2024-12-30T10:57:04.746+0100 [DEBUG] provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:04 [DEBUG] GET https://eu1.app.sysdig.com/api/policies/v3/rules/groups?names=10320992_AWSMLRule
2024-12-30T10:57:04.747+0100 [WARN]  Provider "registry.terraform.io/sysdiglabs/sysdig" produced an invalid plan for sysdig_secure_managed_policy.sysdig_runtime_aws_threat_detection, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .disabled_rules: planned value cty.SetValEmpty(cty.String) for a non-computed attribute
      - .runbook: planned value cty.StringVal("") for a non-computed attribute
      - .scope: planned value cty.StringVal("") for a non-computed attribute
[...]

[[{"id":27860171,"name":"10320992_AWSMLRule","origin":"Secure UI","versionId":"0.0.0","filename":"fast_engine_rules.yaml","description":"Test ML Rule Description","details":{"ruleType":"AWS_MACHINE_LEARNING","anomalousConsoleLogin":{"enabled":false,"threshold":1,"severity":0}},"tags":["machine_learning"],"version":1,"createdOn":1735032495308,"modifiedOn":1735032495308}]]: timestamp=2024-12-30T10:57:05.003+0100
2024-12-30T10:57:05.004+0100 [WARN]  Provider "registry.terraform.io/sysdiglabs/sysdig" produced an invalid plan for sysdig_secure_managed_policy.sysdig_runtime_threat_detection, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .disabled_rules: planned value cty.SetValEmpty(cty.String) for a non-computed attribute
      - .runbook: planned value cty.StringVal("") for a non-computed attribute
      - .scope: planned value cty.StringVal("") for a non-computed attribute
2024-12-30T10:57:05.006+0100 [WARN]  Provider "registry.terraform.io/sysdiglabs/sysdig" produced an invalid plan for sysdig_secure_aws_ml_policy.sysdig_runtime_aws_ml, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .type: planned value cty.StringVal("aws_machine_learning") for a non-computed attribute
      - .runbook: planned value cty.StringVal("") for a non-computed attribute
      - .scope: planned value cty.StringVal("") for a non-computed attribute
2024-12-30T10:57:05.008+0100 [INFO]  provider.terraform-provider-sysdig_v1.41.0: 2024/12/30 10:57:05 [DEBUG] HTTP/1.1 200 OK
Content-Length: 399
Cache-Control: private
Content-Type: application/json
Date: Mon, 30 Dec 2024 09:57:05 GMT
Server: envoy
Set-Cookie: INGRESSCOOKIEAPI="2269581dba21abbe"; Max-Age=86400; HttpOnly
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 954
X-Ratelimit-Reset: 1735552667
X-Request-Id: e7fb98ac-2d87-49f1-ad87-4a330f6b144d

[[{"id":27860173,"name":"10320995_MalwareRule","origin":"Secure UI","versionId":"0.0.0","filename":"fast_engine_rules.yaml","description":"Malware Rule","details":{"ruleType":"MALWARE","useManagedHashes":true,"usePolymorphicRules":false,"additionalHashes":{},"ignoreHashes":{},"ignorePaths":{},"useRegex":false},"tags":["malware"],"version":1,"createdOn":1735032495314,"modifiedOn":1735032495314}]]: timestamp=2024-12-30T10:57:05.008+0100
2024-12-30T10:57:05.012+0100 [WARN]  Provider "registry.terraform.io/sysdiglabs/sysdig" produced an invalid plan for sysdig_secure_malware_policy.sysdig_runtime_malware, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .scope: planned value cty.StringVal("") for a non-computed attribute
      - .runbook: planned value cty.StringVal("") for a non-computed attribute
      - .type: planned value cty.StringVal("malware") for a non-computed attribute
2024-12-30T10:57:05.018+0100 [INFO]  backend/local: plan operation completed

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Rules not found
│
│   with sysdig_secure_ml_policy.sysdig_runtime_workload_ml,
│   on workload-ml.tf line 3, in resource "sysdig_secure_ml_policy" "sysdig_runtime_workload_ml":
│    3: resource "sysdig_secure_ml_policy" "sysdig_runtime_workload_ml" {
│
╵

Let me know if you need anything else to help clarify this.

[raserma]

Any update on this @tembleking ? 🙏

[tembleking]

No news yet @raserma, is this still happening to you in the latest version?

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

This issue has been closed due to inactivity.