Pretty Verifier

Pretty Verifier is a tool designed to enhance and simplify the error messages generated by the Linux Kernel's eBPF verifier.

When developing eBPF programs, the verifier can produce cryptic or low-level error outputs that are hard to interpret. Pretty Verifier improves the developer's experience by combining information extracted from:

• the original .c source code (which is statically analyzed),
• the compiled .o object file (containing BPF bytecode and DWARF debug symbols),
• the verifier's output,
• an internal knowledge base derived from the verifier's internal logic and from external public sources.

Pretty Verifier produces a clearer, source-level report that highlights the exact locations in the C code where issues occur and provides insightful explanations and fix suggestions.

Here is an overview of where Pretty Verifier fits into the eBPF development workflow:

By analyzing the .c file, the compiled .o file, and the eBPF verifier output, and by leveraging its internal knowledge base, Pretty Verifier returns enhanced error messages that make debugging easier and faster.

Requirements

• Python3
• eBPF developement tools

If you are on Debian/Ubuntu, you can run:

sudo apt install linux-headers-$(uname -r) \
                 libbpfcc-dev \
                 libbpf-dev \
                 llvm \
                 clang \
                 gcc-multilib \
                 build-essential \
                 linux-tools-$(uname -r) \
                 linux-tools-common \
                 linux-tools-generic \
                 cmake

Kernel Version

The current version of Pretty Verifier works best with distributions using the kernel version 6.8, but it partially works also for older and newer kernel versions.

Installation

Pretty Verifier can be installed system-wide, providing both a CLI tool and a C library for integration.

To install everything (CLI tool + C Library):

make install

If you only need the CLI tool:

make install-cli

To uninstall:

make uninstall

Usage

1. CLI Usage (Command Line)

Once installed, you can use the pretty-verifier command directly from your terminal.

Always compile your eBPF C code with Clang, using the -g option.

Pipe pretty-verifier when loading the eBPF program, specifying the source and object files through the -c and -o options.

Here are examples for the two main scenarios:

Pipe with stderr output (e.g., bpftool)

bpftool prog load your_bpf_object.o /sys/fs/bpf/your_bpf 2>&1 | pretty-verifier -c your_bpf_source.c -o your_bpf_object.o

Pipe with stdin output (Custom Loaders printing the eBPF Verifier log)

./your_program | pretty-verifier -c your_bpf_source.c -o your_bpf_object.o

If your eBPF program is split into multiple source files:

bpftool prog load your_bpf_object.o /sys/fs/bpf/your_bpf 2>&1 | pretty-verifier -c your_bpf_source.c your_bpf_library.h -o your_bpf_object.o

If you've saved the verifier log into a file (e.g., verifier.log), use the --logfile (-l) option:

pretty-verifier -l verifier.log -c your_bpf_source.c -o your_bpf_object.o

Loader Script Generator

The genloader utility creates a Bash script to automate the loading of eBPF programs and integration with Pretty Verifier.

To generate a loader script:

pretty-verifier geneloader \ 
    [--output-dir <output_directory> -d] \ 
    [--script-name <script_name> -n] \
    [--load-command "<custom_load_command>" -l] \
    [--test -t]
    [--help -h]

2. C Library Usage

You can integrate Pretty Verifier directly into your C userspace loader to directly obtain the enhanced eBPF verifier log.

Instructions

1. Include the header file: #include <pretty_verifier.h>
2. Load the eBPF program and store the eBPF verifier log.
3. Call pretty_verifier.
4. Compile with -lpretty-verifier.

Code Example

#include <stdio.h>
#include <bpf/libbpf.h>
#include <pretty_verifier.h>

int main() {
    // Buffer to capture the raw kernel verifier log
    char log_buf[64 * 1024]; 
    log_buf[0] = '\0';

    // Configure libbpf to store verifier logs in our buffer
    struct bpf_object_open_opts open_opts = {
        .sz = sizeof(struct bpf_object_open_opts),
        .kernel_log_buf = log_buf,
        .kernel_log_size = sizeof(log_buf),
        .kernel_log_level = 1, 
    };

    struct bpf_object *obj = bpf_object__open_file("test.bpf.o", &open_opts);
    if (!obj) {
        fprintf(stderr, "Failed to open BPF object\n");
        return 1;
    }

    // Try to load the program
    int err = bpf_object__load(obj);

    if (err) {
        char formatted_output[8192];
        struct pretty_verifier_opts pv_opts = {
            .source_paths = "test.bpf.c",
            .bytecode_path = "test.bpf.o",
            .enumerate = 0
        };

        // Pass the captured raw log to Pretty Verifier
        int res = pretty_verifier(log_buf, &pv_opts, formatted_output, sizeof(formatted_output));

    if (res >= PV_SUCCESS) {
            printf("%s\n", formatted_output);
        } 
        else if (res == PV_ERR_TRUNCATED) {
            printf("Output truncated:\n%s\n", formatted_output);
        } 
        else if (res == PV_ERR_NOT_FOUND) {
            fprintf(stderr, "Error: 'pretty-verifier' tool not found in PATH.\n");
        }
        else {
            fprintf(stderr, "Error formatting log (Code: %d)\n", res);
        }
    } else {
        // ... attach programs, create links, etc ...
        printf("Program loaded successfully.\n");
    }

    return 0;
}

Compilation

Link against the pretty-verifier library:

gcc my_loader.c -o my_loader -lpretty-verifier -lbpf -I/usr/local/include -L/usr/local/lib

3. Go Library Usage

You can integrate Pretty Verifier directly into your Go userspace loader to intercept and format verification errors returned by the cilium/ebpf library.

Instructions

After installing Pretty Verifier, import the package:

import (
    // ...
    "github.com/netgroup-polito/pretty-verifier/lib/go"
    // ...
)

Then pass the eBPF verifier log to Pretty Verifier.

Code Example

package main

import (
    "errors"
    "fmt"
    "log"

    "github.com/cilium/ebpf"
    pv "github.com/netgroup-polito/pretty-verifier/lib/go"
)

func main() {

    spec, err := ebpf.LoadCollectionSpec("test.bpf.o")
    if err != nil {
        log.Fatalf("Failed to load spec: %v", err)
    }

    opts := ebpf.CollectionOptions{
        Programs: ebpf.ProgramOptions{
            LogLevel: ebpf.LogLevelInstruction,
        },
    }

    // Try to load the program into the kernel
    coll, err := ebpf.NewCollectionWithOptions(spec, opts)

    if err != nil {
        var ve *ebpf.VerifierError
        if errors.As(err, &ve) {

            rawLog := fmt.Sprintf("%+v", ve)
            
            // Configure the Pretty Verifier options
            pvOpts := pv.Options{
                SourcePaths:  "test.bpf.c",
                BytecodePath: "test.bpf.o",
                Enumerate:    false,
            }

            // Pass the raw verifier log to Pretty Verifier
            formattedOutput, pvErr := pv.Format(ve.Log, pvOpts)
            
            if pvErr != nil {
                // In case of error, print original verifier log
                log.Printf("Error running pretty-verifier: %v", pvErr)
                fmt.Printf("Raw Verifier Log:\n%s\n", ve.Log)
            } else {
                // Print the enhanced output
                fmt.Println(formattedOutput)
            }
            return
        }
        
        log.Fatalf("Failed to create collection: %v", err)
    }
    defer coll.Close()

    fmt.Println("Program loaded successfully.")
}

4. Rust Library Usage

You can integrate Pretty Verifier directly into your Rust userspace loader to intercept and format verification errors returned by libraries like libbpf-rs.

Instructions

After installing Pretty Verifier, add the dependency to your Cargo.toml. Since libbpf-rs prints logs to stderr by default, you need to capture them using a callback.

Code Example

use anyhow::Result;
use libbpf_rs::ObjectBuilder;
use libbpf_rs::{PrintLevel, set_print};
use pretty_verifier::{self, Options};
use std::sync::Mutex;

static GLOBAL_LOG_BUFFER: Mutex<String> = Mutex::new(String::new());

fn logger_callback(_level: PrintLevel, msg: String) {
    if let Ok(mut guard) = GLOBAL_LOG_BUFFER.lock() {
        guard.push_str(&msg);
    }
}

fn main() -> anyhow::Result<()> {
    // Redirect the stderr of libbpf-rs to a global variable
    set_print(Some((
        PrintLevel::Debug,
        logger_callback 
    )));

    let filename = "test.bpf.o";
    let source_filename = "test.bpf.c";

    let open_object = ObjectBuilder::default().open_file(filename)?;

    // Try to load the eBPF program
    match open_object.load() {
        Ok(_) => {
            println!("Program loaded successfully.");
        }
        Err(_err) => {
            // Retrieve the eBPF verifier log from the global variable
            let captured_log = GLOBAL_LOG_BUFFER.lock().unwrap().clone();

            // Configure Pretty Verifier options
            let pv_opts = Options {
                source_paths: source_filename,
                bytecode_path: filename,
                enumerate: false,
                ..Default::default()
            };

            // Pass the captured log to Pretty Verifier
            match pretty_verifier::format(&captured_log, pv_opts) {
                Ok(formatted_output) => {
                    // Print the enhanced output
                    println!("{}", formatted_output);
                },
                // Manage possible errors
                Err(pretty_verifier::Error::Truncated(_, partial)) => {
                    println!("Output truncated:\n{}", partial);
                },
                Err(pretty_verifier::Error::NotFound) => {
                    eprintln!("Error: 'pretty-verifier' tool not found in PATH.");
                },
                Err(e) => {
                    eprintln!("Error formatting log: {}", e);
                }
            }
        }
    }

    Ok(())
}

Compilation

Ensure your Cargo.toml includes the necessary dependencies.

# ...
[dependencies]
pretty-verifier = { git = "https://github.com/netgroup-polito/pretty-verifier", subdir = "lib/rust" }
# ...

Then build and run your project (usually requires root privileges for eBPF):

cargo build
sudo ./target/debug/{program_name}

Development mode

In order to run Pretty Verifier without installation, into the current folder, run

PYTHONPATH=src python3 -m pretty_verifier.main

Acknowledgements

This work has been partially supported by the ELASTIC project https://elasticproject.eu/, which received funding from the Smart Networks and Services Joint Undertaking https://smart-networks.europa.eu/ (SNS JU) under the European Union’s Horizon Europe https://research-and-innovation.ec.europa.eu/funding/funding-opportunities/funding-programmes-and-open-calls/horizon-europe_en research and innovation programme under Grant Agreement No. 101139067 https://cordis.europa.eu/project/id/101139067. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them.