-
Notifications
You must be signed in to change notification settings - Fork 262
Description
What steps did you take and what happened:
Running trivy-operator in a kubernetes cluster. Noticed that some vulnerability scan jobs were not being triggered. When I added debug logging and turned on LOG_DEV_MODE, noticed that the issue was that custom volume mounts configured in scanJob.customVolumesMount were being duplicated for some containers causing the creation of the job to fail.
What did you expect to happen:
No duplicate custom volume mounts and job created succesfully.
Anything else you would like to add:
After investigation, it looks like the volumeMounts slice is shared among containers which then causes customVolumesMount slice to be appended twice for some containers.
I have some debug logging which might help.
I'll be refering to these lines of code:
trivy-operator/pkg/plugins/trivy/image.go
Line 559 in ac67105
containers = append(containers, corev1.Container{ templateSpec.Containers[i].VolumeMounts = append(templateSpec.Containers[i].VolumeMounts, s.customVolumesMount...)
I've added debug logging of the relevant volumeMounts slice header before line 1 and before and after line 2, tested with a pod with two containers, and these are the results.
-
Case with the bug but where it is not noticeable:
line 1: (printed volumeMounts)before container append container1: `en=2 cap=2 ptr=0xc001194d10 before container append container2: len=2 cap=2 ptr=0xc001194d10line 2: (printed templateSpec.Containers[i].VolumeMounts)
before custom volume mounts append container1: len=2 cap=2 ptr=0xc001194d10 after custom append container1: len=3 cap=4 ptr=0xc005c9e000 before custom volume mounts append container2: len=2 cap=2 ptr=0xc001194d10 after custom append container2: len=3 cap=4 ptr=0xc005c9e160 -
Case with the bug where it is noticeable:
line 1: (printed volumeMounts)before container append container1: len=3 cap=4 ptr=0xc004945080 before container append container2: len=4 cap=4 ptr=0xc004945080line 2: (printed templateSpec.Containers[i].VolumeMounts)
before custom volume mounts append container1: len=3 cap=4 ptr=0xc004945080 after custom append container1: len=4 cap=4 ptr=0xc004945080 before custom volume mounts append container2: len=4 cap=4 ptr=0xc004945080 after custom append container2: len=5 cap=8 ptr=0xc001585208 -
Case with the bug fixed:
line 1: (printed volumeMounts (per container))before container append container1: len=3 cap=4 ptr=0xc000bbf8c0 before container append container2: len=3 cap=4 ptr=0xc0019438c0line 2: (printed templateSpec.Containers[i].VolumeMounts)
before custom volume mounts append container1: len=3 cap=4 ptr=0xc000bbf8c0 after custom append container1: len=4 cap=4 ptr=0xc000bbf8c0 before custom volume mounts append container2: len=3 cap=4 ptr=0xc0019438c0 after custom append container2: len=4 cap=4 ptr=0xc0019438c0
From the results we can see that volumeMounts slice is currently shared between containers. The bug is not always noticeable because when appending the customVolumesMount sometimes the array is reallocated right after the first append causing each container's reference to diverge. When this does not happen, however, the second container reallocates while having the full first container's array in it's reference, causing duplicates.
My fix was to allocate a new slice for each container and pass it on to its creation.
Environment:
- Trivy-Operator version (use
trivy-operator version): 0.28.0 - Kubernetes version (use
kubectl version): v1.32.7 - OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Debian 12