proposal: runtime/pprof: add individual goroutine profile with labels #74964
Description
Background
The goroutine profile currently supports:
debug=0/debug=1: Includes pprof labels, but not individual goroutines.debug=2: Lists individual goroutines and their states, but excludes pprof labels.
There is currently no format that includes both, though this has been requested.
Proposal
Introduce debug=3, which lists individual goroutines and their stack traces and states in a format almost identical to debug=2, but additionally includes any associated pprof.Labels in the per-goroutine header after the [state] segment, before the :, e.g.:
goroutine 1234 [select, 2 minutes]{"service": "users", "priority": "background"}:This would be added as a new, opt-in format rather than altering debug=2's output, to preserve backward compatibility with parsers of the existing debug=2 output.
Additionally, for implementation reasons (see below and #74954), this new format could specify that it always omits function argument values when printed, using the same (...) sometimes used in (and thus alread expected by tools which parse) debug=2.
Motivation
Combining goroutine-level detail with labels provides a powerful tool for debugging and profiling. Labels are useful for identifying the purpose or context of goroutines, while full stack traces and states are critical for diagnosing issues like stalls or deadlocks. Today, users must choose between one or the other.
Implementation Sketch
Extend pprof.Lookup("goroutine").WriteTo() to support debug=3:
- Output full goroutine stack traces and states.
- Include associated labels in a parseable format (e.g.,
labels: key=value).
This could potentially be incorporated into #74954, to use the lower-STW collection approach that already collects labels, however this likely implies this new format also diverges from debug=2 in that it would consistently omit function argument values from the printed stack traces as well, further motivating the choice to offer it as a new, distinct format.
Compatibility
Existing formats remain unchanged. debug=3 would be a new, additive option.
Alternatives
- Modify
debug=2to include labels: risks breaking existing tools. - Export labels separately: harder to correlate with goroutines.