Reconciler error: Large size VulnerabilityReports will failed create and remains scan jobs

[mrtc0]

What steps did you take and what happened:

Creation of VulnerabilityReports fails if the number of detected vulnerabilities is too large.

Steps to reproduce

1. Create a Pod with python:3.5.10-buster image on a cluster with trivy-operator installed

(python:3.5.10-buster has 1252 vulnerabilities.)

$ kubectl create deployment python -n default --image python:3.5.10-buster

3. Scan finishes successfully, but creation of VulnerabilityReports fails

# Scan is completed successfully, but no VulnerabilityAlerts are created and no jobs are deleted
❯ kubectl get jobs -n trivy-system
NAME                                  COMPLETIONS   DURATION   AGE
scan-vulnerabilityreport-7d8db4fc84   1/1           73s        5m42s

❯ kubectl get pods -n trivy-system
NAME                                        READY   STATUS      RESTARTS   AGE
scan-vulnerabilityreport-7d8db4fc84-gsztw   0/1     Completed   0          4m12s
trivy-operator-54bc4769db-5djnd             1/1     Running     0          85m

❯ kubectl -n trivy-system logs scan-vulnerabilityreport-7d8db4fc84-gsztw | tr -d "\n" | base64 -d | bunzip2 | jq .ArtifactName
Defaulted container "python" out of: python, 95164937-7089-491c-ab58-7f52bc8a8cce (init)
"python:3.5.10-buster"

❯ kubectl get vulnerabilityreports -n default
No resources found in default namespace.

# Reconciler error is logged in Operator
❯ kubectl logs -n trivy-system trivy-operator-54bc4769db-5djnd
...
{"level":"error","ts":1670231487.1708708,"msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7d8db4fc84","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-7d8db4fc84","reconcileID":"7e1ccc8e-f6eb-4080-b1dd-25dcdb150e7c","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (2633583 vs. 2097152)","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}

I am not familiar with the Kubernetes Operator implementation, but from the error message (trying to send message larger than max (2633583 vs. 2097152)) I would guess that it is due to the VulnerabilityAlerts being too large.

Normally, reconcileJobs() should delete the Job, but it gets stuck because it fails to create VulnerabilityReports.
Therefore, when the number of similar jobs reaches the scanJobsConcurrentLimit, no new scans will be performed.

What did you expect to happen:

There are several possible approaches to solving this error:

• Failure to create VulnerabilityReports Job to be deleted
• VulnerabilityReports are successfully created

Environment:

• Trivy-Operator version (use trivy-operator version): 0.7.1
• Kubernetes version (use kubectl version): v1.25.0
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc):

[chen-keinan]

@mrtc0 thank you for reporting it.
Are you running the operator with trivy client / server mode ?

[mrtc0]

I am running trivy.mode with Standalone specified.

[chen-keinan]

I am running trivy.mode with Standalone specified.

thanks I'll take a look at it.
is this happen for every scan job or few ?

[mrtc0]

I've tried several times and it fails every time.
We have confirmed reproduction with python:3.5.10-buster as well as redash/redash:8.0.2.b37747.

[chen-keinan]

I've tried several times and it fails every time. We have confirmed reproduction with python:3.5.10-buster as well as redash/redash:8.0.2.b37747.

thanks , I will try to reproduce it myself

[chen-keinan]

Thanks , its look like it is related to following issues :
#445 #442 #443 #441
these item are under investigation already , I will be able to update later on

[df-cgdm]

Hello I've the same kind of problems

The trivy operator report a lot of error like the one below and the job remains so the operator stop working I have to delete the job manually
{"level":"error","ts":1678879621.0102816,"msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-cb64fcddb","namespace":"lens-security"},"namespace":"lens-security","name":"scan-vulnerabilityreport-cb64fcddb","reconcileID":"df646de4-7b9a-4a95-90ab-fcfebe5f0e14","error":"json: cannot unmarshal number into Go value of type trivy.ScanReport","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:234"}

[HW-Jeremy]

Hello,

I've the same problem

[chen-keinan]

@df-cgdm are you running with scanJobCompressLogs = false flag ?

[srikwang]

Hello I've the same kind of problems,The difference is that my compliance report happened，See Error:

{"level":"error","ts":"2023-04-26T06:59:17Z","logger":"reconciler.clustercompliancereport","msg":"failed to generate compliance report","compliance report":"/cis","error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (2273222 vs. 2097152)","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:68\nk8s.io/client-go/util/retry.OnError.func1\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.3/util/retry/util.go:51\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.3/pkg/util/wait/wait.go:222\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.3/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.3/pkg/util/wait/wait.go:228\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.26.3/pkg/util/wait/wait.go:423\nk8s.io/client-go/util/retry.OnError\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.3/util/retry/util.go:50\nk8s.io/client-go/util/retry.RetryOnConflict\n\t/home/runner/go/pkg/mod/k8s.io/client-go@v0.26.3/util/retry/util.go:104\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).generateComplianceReport\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:50\ngithub.com/aquasecurity/trivy-operator/pkg/compliance.(*ClusterComplianceReportReconciler).reconcileComplianceReport.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/compliance/clustercompliancereport.go:44\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}