Adding support for privilege delegation using BPF Token

[anryko]

I'm trying to load ebpf program in unprivileged container with bpffs privilege delegation enabled. It works with bpftool but fails while using cilium/ebpf. Is this functionality supported?

BPFFS mount:

none on /sys/fs/bpf type bpf (rw,relatime,delegate_cmds=any,delegate_maps=any,delegate_progs=any,delegate_attachs=any)

Successful map creation while using bpftool:

bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=860, max_entries=1, map_flags=BPF_F_RDONLY_PROG|0x10000, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=3, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0, ...}, 92) = 13

Failure loading the same object file using cilium/ebpf:

bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=860, max_entries=1, map_flags=BPF_F_RDONLY_PROG, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 80) = -1 EPERM (Operation not permitted)

[dylandreimerink]

Is this functionality supported?

Not yet. Though I see you are actively working on this (thank you).

Please help out and advice on how to proceed with moving forward with this feature.

• feat: poc of bpf token delegation #1926

We should probably think about the abstraction level at which we want to implement it and the architecture for this more broadly. What you have done so far in the POC is to essentially do it at the syscall level and to make a "token per process" model.

I took a quick peek at how libbpf approached this, and I think they have got it by the right end in this case. They went with a token per Collection model. They allow a user to pass the path of the bpffs they would like to use. If no path is specified they attempt to use the default /sys/fs/bpf. They also have a environment variable as path input which I think we can do away with. For us that would be a string field in the collection spec.

Then when they load the collection they attempt to get a token, with is allowed to fail if the user did not specifically input a path. Libbpf also allows the user to input a blank string meaning "don't attempt to use a token" which would also be good to have in some form. The token fd becomes part of the Collection and is collected on Collection.Close(). Any operation that can use a token fd (btf load, prog load, map create) is passed the FD if we ended up getting one. So this would simply be a change to the attributes passed to the syscall layer, no need for retries.

I also think there are some other parts of the token feature to consider.

Something to discuss further and to think about is that perhaps it would make sense for us to also provide the facilities for the BPFFS creation, FS transmission and delegation such as done in the test code. We would have to figure out what that API should look like. But if tokens are going to be a think we want to work with in the library, it also makes sense that we at least help with obtaining delegated privileges given that seems involved as well.

Lastly, tokens have some implications for feature probing. That is because every BPFFS mount point from which we derive tokens now may have a different set of capabilities (map types, program types) it can use. Currently all of our feature probing functions do not take tokens and use caches that are process wide. So we will have to think about how we want to integrate tokens into the feature probing as well.
Libbpf took the approach of doing feature probing per Collection. This seems inconvenient to me though. Perhaps it would make sense to have some sort of object that represents a BPFFS from which we we get the tokens. That way we could store feature cache info in that object as well as have the object for the above mentioned API. And pass it to the collection spec instead of just a path.

@rgo3 for input on the feature probing side of this.

[anryko]

@dylandreimerink, thanks a lot for your detailed feedback. I pushed a new commit to the POC draft PR trying to address some of the points you raised. Specifically, I moved BPF token setup to CollectionOptions and then pass it around to the functions that can make use of it (with the exception of btf load, will handle it if you like what you see so far).

I'll have a look at bpffs testing tooling part as you suggested and see what I can do there.

Could you elaborate regarding your thoughts on the feature probing caching and passing around the token related btf cache object to the collection? I struggle to sort out how it supposed to look like in code.

[dylandreimerink]

Could you elaborate regarding your thoughts on the feature probing caching and passing around the token related btf cache object to the collection? I struggle to sort out how it supposed to look like in code.

Yea, so take for example HaveProgramType. This is currently a global function, and when you call it it will not use a token.

I thought we were caching more than we were. It appears we only cache helper functions per program type at the moment.

So the first, and perhaps obvious option is to make a ...WithToken variant for example, such as HaveProgramTypeWithToken(pt ebpf.ProgramType, bpffsPath string).

If you want to do caching, such as for the helper then we need to include the path in the cache key. However, that is technically incorrect, since unmounting the bpffs and remounting after getting different delegated privileges could be possible, which would invalidate the cache. So I was thinking that it would make more sense to bind such a cache to an object that represents an instance of the bpffs or a specific token.

Also still thinking about the actual delegation process. As I understand it the interaction looks something like this:

Privileged process                     | Unprivileged process
---------------------------------------|----------------------------
Creates UNIX socket                    |
Spawns Unprivileged process    -> Spawns, with other end of socket
                                       | // Create a FS FD, nothing is mounted yet
                                       | fsfd = fsopen("bpf", 0) 
Receive FS                            <-- Send FS FD over unix socket
fsconfig(fsfd, opts)                   |
send ack                              --> Receive ack
                                       | bpffsfd = fsmount(fsfd) // Mount the FS, with the delegated priviliges
                                       | tokenfd = bpf(BPF_TOKEN_CREATE, {bpffsfd, ...})

In the above interaction we just end up with file descriptor representing the mounted FS, but its not actually present at a path.

You can then move it to an actual path so it will persist with move_mount(fsfd, AT_FDCWD, "/sys/fs/bpf", MOVE_MOUNT_F_EMPTY_PATH).

Once that is done you can use open or openat on the path to also obtain a BPFFS FD.

So there are essentially 3 different usage scenarios:

1. fsopen -> IPC -> fsmount -> bpf token
2. fsopen -> IPC -> fsmount -> move_mount (-> bpf_token)
3. open/openat -> bpf_token

In scenario one you do the delegation and get an ephemeral BPFFS which gets unmounted on process exit or FD close. In scenario two you move it so that other processes can re-use the BPFFS, a libbpf program for example. And in scenario three there already exists a BPFFS which we want to reuse.

There is also not a spec or standard for the IPC bit itself. The example simply transfers the FD without preamble, but I can imagine more complex setups where a container runtime or systemd might have a more complex conversation on the socket before FDs are exchanged.

Here is the pseudo code I came up with:

type BPFFS struct {
    path string
    fsfd *sys.FD
    bpffsfd *sys.FD
    tokenfd *sys.FD
}

func NewBPFFS() (*BPFFS, error) {
    var bpffs BPFFS

     bpffs.fsfd, err = unix.fsopen(bf.path)
    return
}

func NewBPFFSFromPath(path string) (*BPFFS, error) {
    var bpffs BPFFS

    bpffs.bpffsfd, err := unix.Open(path, syscall.O_DIRECTORY|syscall.O_RDONLY, 0)
	if err != nil {
		return nil, err
	}

    return &bpffs, nil
}

func (bf *BPFFS) FD() *sys.FD {
    return bf.fsfd
}

func (bf *BPFFS) Mount() error {
    var err error
    // Mount the FS, but just give the file descriptor, don't put it on a path
    bf.bpffsfd, err = unix.Fsmount(bf.fsfd.Uint(), ...)
    bf.fsfd.Close()
    bf.fsfd = nil
}

func (bf *BPFFS) MountAt(path string) error {
    var err error
    bf.bpffsfd, err = unix.Fsmount(bf.fsfd.Uint(), ...)
    // Move the mount to actually be on a path for other processes to use
    unix.MoveMount(bf.fsfd.Int(), "", unix.AT_FDCWD, path, MOVE_MOUNT_F_EMPTY_PATH)
    bf.fsfd.Close()
    bf.fsfd = nil
}

func (bf *BPFFS) Close() error {
    if fs.fsid != nil {
        os.Close(fs.fsid)
    }
    if fs.tokenfd != nil {
        os.Close(fs.fsid)
    }
    if fs.bpffsfd != nil {
        os.Close(fs.fsid)
    }
}

// Used internally by 
func (bf *BPFFS) token() (*sys.FD, error) {
    if bf.tokenfd != nil {
        return bf.tokenfd, nil
    }

    if bf.bpffsfd == nil {
        return nil, fmt.Errorf("BPFFS is not mounted")
    }

    var err error
    tokenAttr := sys.TokenCreateAttr{BpffsFd: bf.bpffsfd.Uint()}
	tok, err = sys.TokenCreate(&tokenAttr)
    return bf.tokenfd, err
}

So we have an object that represents the BPFFS. In scenario one and two you use NewBPFFS, then FD to get the descriptor for unix socket transmission which is left to the user. In scenario one the user uses Mount after the unix ack to get a FD FD. In scenario two you use MountAt to also mount at a path.
Scenario three would use NewBPFFSFromPath and be ready at once.

After each of these you have an object that can be passed to the collection spec or LoadCollection, and later gets embedded in the Collection. The BPFFS object holds onto the mountfs file descriptor so it sticks around in the case of scenario one.

We can also put any feature caches in this object, and pass it to the feature detection function for both the token and the cache. Like HaveProgramTypeWithToken(pt ebpf.ProgramType, bpffs *BPFFS)

Does this make sense to you?

I am aware this is a lot. I would be happy to help with this part, so it in an initial PR so you can make use of it during the loading part.

[anryko]

So there are essentially 3 different usage scenarios:

1. fsopen -> IPC -> fsmount -> bpf token
2. fsopen -> IPC -> fsmount -> move_mount (-> bpf_token)
3. open/openat -> bpf_token

@dylandreimerink, do we really need scenarios 1 and 2 in the library? The library code will be run inside the unprivileged container with the middleware (like LXD/Incus) taking care of the BPFFS setup. At the stage I load the program and create maps I already have the mount ready.

$ mount | grep bpf
none on /sys/fs/bpf type bpf (rw,relatime,delegate_cmds=map_create:map_update_elem:prog_load:obj_pin:obj_get:btf_load:map_freeze,delegate_maps=hash:array:percpu_array:ringbuf,delegate_progs=sched_cls:lwt_xmit,delegate_attachs=cgroup_inet_ingress)

All I need for the library to do is support BPF_TOKEN_CREATE, and a way to enable its usage. I think I'm missing the usecase for scenarios 1 and 2.

[anryko]

@dylandreimerink, I updated the draft PR to use ebpf.BPFFS as you suggested. Covered only the 3rd scenario for now, and no test fixes. I'll proceed with other cases when I have a better understanding of usecases for the other two scenarios.

[dylandreimerink]

do we really need scenarios 1 and 2 in the library?

We don't need it, especially if there is no use case for it right now. But we should design the API to allow for it in the future.

Covered only the 3rd scenario for now, and no test fixes. I'll proceed with other cases when I have a better understanding of usecases for the other two scenarios.

This is great. Lets just keep it like this for now, lets not add the logic if we are not going to have people using it. By structuring it like this we can always add that later if anyone comes along who wishes to have scenario 1 or 2.

[PrachiJha-404]

Hello! I would like to work on this issue. @anryko are you still actively working on this? Happy to collaborate!