New protect rules don't have effect when 38 protect rules are already applied

[holyspectral]

Upstream reference: cilium/tetragon#4244

---

Is there an existing issue for this?

• I have searched the existing issues

Environment

- OS: Ubuntu
- Architecture: amd64
- Cluster: kind

Issue Description

• Expected behavior: the max limit we can support is to be defined, but 38 policies is way too low.
• Current behavior: when more than 38 protect policies are assigned, tetragon would return below errors:

[tetragon] level=warn msg="adding tracing policy failed" error="sensor generic_kprobe from collection deploy-ubuntu-deployment-86 failed to load: failed prog /var/lib/tetragon/bpf_generic_kprobe_v612.o kern_version 396811 loadInstance: attaching 'generic_fmodret_override' failed: create raw tracepoint: argument list too long"

When LSM is enabled, it returns a different error:

[tetragon] level=warn msg="adding tracing policy failed" error="sensor generic_lsm from collection deploy-ubuntu-deployment-26 failed to load: failed prog /var/lib/tetragon/bpf_generic_lsm_output_v612.o kern_version 396811 loadInstance: attaching 'generic_lsm_output' failed: create tracing link: argument list too long"

• Steps to reproduce:

ubuntu-policy.yaml:

apiVersion: security.rancher.io/v1alpha1
kind: WorkloadSecurityPolicy
metadata:
  name: deploy-ubuntu-deployment
  namespace: default
spec:
  mode: protect
  rules:
    executables:
      allowed:
      - /usr/bin/sleep
  selector:
    matchLabels:
      app: ubuntu

for i in {1..200}
do
    cat ~/events/ubuntu-policy.yaml | sed "s/deploy-ubuntu-deployment/deploy-ubuntu-deployment-$i/g" | sed "s/app: ubuntu/app: ubuntu-$i/g" | kubectl apply -f -
done

[holyspectral]

This issue was discovered during the scaling test about #20 . After investigation, it's likely caused by BPF_MAX_TRAMP_LINKS because internally Tetragon depends on eBPF trampoline and fmod_ret to implement override.

https://elixir.bootlin.com/linux/v6.14.11/source/kernel/bpf/trampoline.c#L579

[holyspectral]

Given that the number is hardcoded in Linux kernel, I don't think it's possible to make it configurable in Tetragon or other places.

I think we need to work with upstream to figure out a solution. Other than that, I think having a policy optimization logic in Tetragon would help. Take kprobe for example, it installs a generic_fmodret_override program for each Tetragon policy. However, all of our policies are basically the same, so we don't really need separate generic_fmodret_override for each policy.

We should be able to let each kprobe program to share the same override_tasks map and the same generic_fmodret_override.

[holyspectral]

For LSM, it would need more investigation.

1. Tetragon installs two eBPF LSM programs on the same hook point in this PR: IMA hashes in LSM events cilium/tetragon#2818
2. In the current Tetragon design, LSM program runs as a whole, which means for each different pod selector we would need one collection of ebpf programs.
3. Per this issue, the bpf program needs the BTF ID when loaded, which means we can't have a common LSM program for all.
4. A quick check using cat /proc/kallsyms | grep ' security_' | wc -l returns 365 LSM hooks.
5. It doesn't look like in the recent versions, bpf_override_return is still used. Maybe it's worth a revisit.

[holyspectral]

It turned out that our usage doesn't require bpf_override_return, so we can create a PR upstream to remove the constraint. This will simplify the design A LOT in both neuvector/runtime-enforcement#190 and this case. We probably don't need neuvector/runtime-enforcement#190 for a LSM version of hooks anymore too.

[holyspectral]

This POC allows us to have more than 38 enforcement policies: https://github.com/holyspectral/tetragon/tree/experimental-remove-38-limits

[Andreagit97]

See the upstream issue cilium/tetragon#4204

[holyspectral]

A rough plan:

In createKprobeSensorFromEntry() or createSingleKprobeSensor(), which is called by PolicyHandler, we:

1. Create a new program using program.Builder() to load the generic_fmodret_override program.
2. Define another type of program, e.g., generic_fmodret_override, to implement its attach flow.
   • This is necessary because we want to manage kprobe program and fmodret's lifecycle separately.
3. For this new type of program, we we should be able to see program.Program.LoadState. So we know if we should attach again or not. prog.Attach can be used.
4. Maintain the reference count of the generic_fmodret_override prog, so we can destroy it when needed.
5. Redirect the map access. We can consider symlink.

[holyspectral]

A draft PR is created to gather feedback from upstream. cilium/tetragon#4244

[holyspectral]

I got feedback from upstream and I think the most significant one is that they want to include kprobe_override's override_tasks into the scope. Will continue discussing with them.