Skip to content

Generic Probe Support: Add kprobe Functionality Alongside exiting uprobes#9

Merged
dalehamel merged 1 commit intoshopify-omnibusfrom
kprobe-support
Nov 10, 2025
Merged

Generic Probe Support: Add kprobe Functionality Alongside exiting uprobes#9
dalehamel merged 1 commit intoshopify-omnibusfrom
kprobe-support

Conversation

@manuelfelipe
Copy link
Copy Markdown
Member

@manuelfelipe manuelfelipe commented Nov 10, 2025

Summary

Refactors probe infrastructure introduced in open-telemetry#651 to support kprobes, kretprobes, uprobes, and uretprobes through a unified interface. Previously only uprobes were supported.

Motivation

We wanted to understand what causes shared memory pages to turn into private memory pages (copy-on-write) in our Ruby workers. Pitchfork workers start with everything shared post-fork but quickly grow private memory as requests come in.

Being able to attach kprobes to kernel functions like do_wp_page allows us to capture stack traces when CoW events occur, showing exactly which Ruby code is causing pages to be copied. This data could help us identify memory hotspots and optimization opportunities.

Previous uprobe-only support limited profiling to user space functions. Adding kprobe support enables profiling kernel-level events (page faults, syscalls, etc.) alongside regular CPU profiling, giving a complete picture of system behavior.

What Changed

Renamed uprobe-specific code to generic probe terminology:

  • --uprobe-link--probe-link flag
  • UProbeLinksProbeLinks config
  • TraceOriginUProbeTraceOriginProbe
  • support/ebpf/uprobe.ebpf.cgeneric_probe.ebpf.c

Added kprobe support:

  • New kprobe__generic eBPF entry point
  • Unified probe attachment logic in tracer/probe.go
  • Support for kprobe/kretprobe/uprobe/uretprobe types

Enhanced probe-ctrl tool:

  • Works with all probe types
  • Added --list flag to show loaded eBPF programs
  • Better error messages

Usage


# Attach to kernel function
./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -probe-link "kprobe:do_wp_page`

# Attach to user space function  
./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls -probe-link "uprobe:/root/ebpf-profiler:runtime.(*mheap).alloc"

Running ^ locally, with traces going to devfiler:

kprobe-do_wp_page Screenshot 2025-11-10 at 3 19 27 PM

Copy link
Copy Markdown
Member

@dalehamel dalehamel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor nitpick, this looks great overall.

Curious what caused all the mod changes?

It would be good for the upstream PR description to also include a real world example we are using this for, describing the CoW debugging that was the motivation for facilitating this

Rename uprobe-specific code to generic probe terminology and add
support for kprobes alongside uprobes. Update probe-ctrl tool to
handle both probe types.
@manuelfelipe
Copy link
Copy Markdown
Member Author

Curious what caused all the mod changes?

before https://github.com/Shopify/opentelemetry-ebpf-profiler/pull/9/files#diff-411cc25513d351cd98dba8360c35b0cec01c0271246e3b15c71e3626eb63edc6R14, probe-ctrl was not actually importing the ebpf-profiler package and had its own code to load the program. Changed it to have the parsing and loading logic in the tracer pkg and using that from probe-ctrl side

@dalehamel dalehamel merged commit 4290889 into shopify-omnibus Nov 10, 2025
15 of 23 checks passed
@dalehamel dalehamel deleted the kprobe-support branch November 10, 2025 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants