Skip to content

[Diagnostics][dotnet-trace] Add collect-linux verb #47894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
+237 −32
Open

[Diagnostics][dotnet-trace] Add collect-linux verb #47894

Changes from 6 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a666306
[Diagnostics] Cleanup
mdh1418 Aug 7, 2025
a2e4f74
[Diagnostisc][dotnet-trace] Add collect-linux verb
mdh1418 Aug 7, 2025
e1a3ec7
[Diagnostics][dotnet-trace] Add event filter option
mdh1418 Aug 7, 2025
6c9fdb6
Revert "[Diagnostics][dotnet-trace] Add event filter option"
mdh1418 Aug 14, 2025
16053b4
Address Feedback
mdh1418 Aug 15, 2025
26adb7b
[Diagnostics][dotnet-trace] Revamp profiles
mdh1418 Aug 29, 2025
0d09d90
Address profiles feedback
mdh1418 Sep 9, 2025
8118b86
Refine descriptions and limitations
mdh1418 Sep 22, 2025
72027e3
[DotnetTrace][CollectLinux] Cut target process options
mdh1418 Oct 7, 2025
54d662b
Add collect-linux example
mdh1418 Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 239 additions & 24 deletions docs/core/diagnostics/dotnet-trace.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ The `dotnet-trace` tool:
* Is a cross-platform .NET Core tool.
* Enables the collection of .NET Core traces of a running process without a native profiler.
* Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime.
* Delivers the same experience on Windows, Linux, or macOS.
* Delivers the same experience on Windows, Linux, and macOS.
* Offers an additional command on Linux to collect Linux perf events.

## Options

Expand All @@ -55,15 +56,12 @@ The `dotnet-trace` tool:

Displays the version of the dotnet-trace utility.

- **`--duration`**

How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds.

## Commands

| Command |
|-----------------------------------------------------------|
| [dotnet-trace collect](#dotnet-trace-collect) |
| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) |
| [dotnet-trace convert](#dotnet-trace-convert) |
| [dotnet-trace ps](#dotnet-trace-ps) |
| [dotnet-trace list-profiles](#dotnet-trace-list-profiles) |
Expand All @@ -76,13 +74,23 @@ Collects a diagnostic trace from a running process or launches a child process a
### Synopsis

```dotnetcli
dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--clrevents <clrevents>]
dotnet-trace collect
[--buffersize <size>]
[--clreventlevel <clreventlevel>]
[--clrevents <clrevents>]
[--dsrouter <ios|ios-sim|android|android-emu>]
[--format <Chromium|NetTrace|Speedscope>] [-h|--help] [--duration dd:hh:mm:ss]
[-n, --name <name>] [--diagnostic-port] [-o|--output <trace-file-path>] [-p|--process-id <pid>]
[--profile <profile-name>] [--providers <list-of-comma-separated-providers>]
[--format <Chromium|NetTrace|Speedscope>]
[-h|--help]
[--duration dd:hh:mm:ss]
[-n, --name <name>]
[--diagnostic-port]
[-o|--output <trace-file-path>]
[-p|--process-id <pid>]
[--profiles <list-of-comma-separated-profile-names>]
[--providers <list-of-comma-separated-providers>]
[-- <command>] (for target applications running .NET 5 or later)
[--show-child-io] [--resume-runtime]
[--show-child-io]
[--resume-runtime]
[--stopping-event-provider-name <stoppingEventProviderName>]
[--stopping-event-event-name <stoppingEventEventName>]
[--stopping-event-payload-filter <stoppingEventPayloadFilter>]
Expand Down Expand Up @@ -119,7 +127,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| ------------ | ------------------- |
| `gc` | `0x1` |
| `gchandle` | `0x2` |
| `fusion` | `0x4` |
| `assemblyloader` | `0x4` |
| `loader` | `0x8` |
| `jit` | `0x10` |
| `ngen` | `0x20` |
Expand All @@ -138,7 +146,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| `gcheapdump` | `0x100000` |
| `gcsampledobjectallocationhigh` | `0x200000` |
| `gcheapsurvivalandmovement` | `0x400000` |
| `gcheapcollect` | `0x800000` |
| `managedheapcollect` | `0x800000` |
| `gcheapandtypenames` | `0x1000000` |
| `gcsampledobjectallocationlow` | `0x2000000` |
| `perftrack` | `0x20000000` |
Expand All @@ -152,13 +160,16 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| `compilationdiagnostic` | `0x2000000000` |
| `methoddiagnostic` | `0x4000000000` |
| `typediagnostic` | `0x8000000000` |
| `jitinstrumentationdata` | `0x10000000000` |
| `profiler` | `0x20000000000` |
| `waithandle` | `0x40000000000` |
| `allocationsampling` | `0x80000000000` |

You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).

- **`--dsrouter {ios|ios-sim|android|android-emu}**

Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.
Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.

- **`--format {Chromium|NetTrace|Speedscope}`**

Expand Down Expand Up @@ -200,19 +211,24 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
> [!NOTE]
> On Linux and macOS, using this option requires the target application and `dotnet-trace` to share the same `TMPDIR` environment variable. Otherwise, the command will time out.

- **`--profile <profile-name>`**
- **`--profiles <list-of-comma-separated-profile-names>`**

A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. The union of profiles and providers will be in effect.

A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available:
Default behavior (when `--profiles`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`.

| Profile | Description |
|---------|-------------|
|`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|
Available profiles:

| Profile | Description |
|---------|-------------|
|`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.|
|`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|

- **`--providers <list-of-comma-separated-providers>`**

A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile <profile-name>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile.
A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles <list-of-comma-separated-profile-names>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles.

This list of providers is in the form:

Expand Down Expand Up @@ -259,6 +275,205 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--

> - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped.

## dotnet-trace collect-linux

Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes.

This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events.

### Default collection behavior

When `--profiles`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition:

- `dotnet-common` — lightweight .NET runtime diagnostics.
- `dotnet-thread-time` — runtime thread stack sampling.
- `kernel-cpu` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`.
- `kernel-cswitch` — kernel context switches via `Universal.Events/cswitch` for on/off-CPU analysis.

By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. `dotnet-trace` will need to share the same `TMPDIR` value as .NET Processes of interest, otherwise events will not be collected from those processes.

If collecting events from all .NET Processes is undesired, `-n, --name <name>` or `-p|--process-id <PID>` can be used to specify a particular process. **NOTE**: Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.

### Prerequisites

- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+)
- Root permissions
- .NET 10+

### Synopsis

```dotnetcli
dotnet-trace collect-linux
[-h|--help]

# Provider/Event Specification
[--providers <list-of-comma-separated-providers>]
[--clreventlevel <clreventlevel>]
[--clrevents <clrevents>]
[--perf-event-tracepoints <list-of-perf-event-tracepoints>]
[--profiles <list-of-comma-separated-profile-names>]

# Trace Collection
[--format <Chromium|NetTrace|Speedscope>]
[-o|--output <trace-file-path>]
[--duration dd:hh:mm:ss]

# .NET Process Target (Optional)
[-n, --name <name>]
[-p|--process-id <pid>]
[-- <command>]
[--show-child-io]
```

### Options

#### Provider/Event Specification Options

- **`--providers <list-of-comma-separated-providers>`**

A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles <list-of-comma-separated-profile-names>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles.

This list of providers is in the form:

- `Provider[,Provider]`
- `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`.
- `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`.

To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md).

- **`--clreventlevel <clreventlevel>`**

Verbosity of CLR events to be emitted.
The following table shows the available event levels.

| String value | Numeric value |
| --------------- | :-----------: |
| `logalways` | `0` |
| `critical` | `1` |
| `error` | `2` |
| `warning` | `3` |
| `informational` | `4` |
| `verbose` | `5` |

- **`--clrevents <clrevents>`**

A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords:

| Keyword String Alias | Keyword Hex Value |
| ------------ | ------------------- |
| `gc` | `0x1` |
| `gchandle` | `0x2` |
| `assemblyloader` | `0x4` |
| `loader` | `0x8` |
| `jit` | `0x10` |
| `ngen` | `0x20` |
| `startenumeration` | `0x40` |
| `endenumeration` | `0x80` |
| `security` | `0x400` |
| `appdomainresourcemanagement` | `0x800` |
| `jittracing` | `0x1000` |
| `interop` | `0x2000` |
| `contention` | `0x4000` |
| `exception` | `0x8000` |
| `threading` | `0x10000` |
| `jittedmethodiltonativemap` | `0x20000` |
| `overrideandsuppressngenevents` | `0x40000` |
| `type` | `0x80000` |
| `gcheapdump` | `0x100000` |
| `gcsampledobjectallocationhigh` | `0x200000` |
| `gcheapsurvivalandmovement` | `0x400000` |
| `managedheapcollect` | `0x800000` |
| `gcheapandtypenames` | `0x1000000` |
| `gcsampledobjectallocationlow` | `0x2000000` |
| `perftrack` | `0x20000000` |
| `stack` | `0x40000000` |
| `threadtransfer` | `0x80000000` |
| `debugger` | `0x100000000` |
| `monitoring` | `0x200000000` |
| `codesymbols` | `0x400000000` |
| `eventsource` | `0x800000000` |
| `compilation` | `0x1000000000` |
| `compilationdiagnostic` | `0x2000000000` |
| `methoddiagnostic` | `0x4000000000` |
| `typediagnostic` | `0x8000000000` |
| `jitinstrumentationdata` | `0x10000000000` |
| `profiler` | `0x20000000000` |
| `waithandle` | `0x40000000000` |
| `allocationsampling` | `0x80000000000` |

You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).

- **`--perf-event-tracepoints <list-of-perf-event-tracepoints>`**

A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events.

Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup`

- **`--profiles <list-of-comma-separated-profile-names>`**

A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and duplicates are ignored.

Available profiles:

| Profile | Description |
|---------|-------------|
|`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.|
|`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.|
|`kernel-cpu`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.|
|`kernel-cswitch`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.|
|`kernel-precise-thread-time`|Combines `kernel-cpu` + `kernel-cswitch` for a precise thread-time view using kernel events only.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|

> [!NOTE]
> The former default `cpu-sampling` profile is now named as `dotnet-thread-time`.

#### Trace Collection Options

- **`--format {Chromium|NetTrace|Speedscope}`**

Sets the output format for the trace file conversion. The default is `NetTrace`.

- **`-o|--output <trace-file-path>`**

The output path for the collected trace data. If not specified it defaults to `<appname>_<yyyyMMdd>_<HHmmss>.nettrace`, for example, `myapp_20210315_111514.nettrace``.

- **`--duration <time-to-run>`**

The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds.

#### .NET Process Target Options

See [Default collection behavior](#default-collection-behavior)

- **`-n, --name <name>`**

The name of the process to collect the trace from.

> [!NOTE]
> Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.

- **`-p|--process-id <PID>`**

The process ID to collect the trace from.

> [!NOTE]
> Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.

- **`-- <command>`**

After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors.

- **`--show-child-io`**

Only applicable when `-- <command>` is used. Shows the input and output streams of a launched child process in the current console.

> [!NOTE]

> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace.

> - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events.

## dotnet-trace convert

Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools.
Expand Down Expand Up @@ -406,11 +621,11 @@ dotnet-trace collect -- hello.exe arg1 arg2
The preceding command generates output similar to the following:

```output
No profile or providers specified, defaulting to trace profile 'cpu-sampling'
No profile or providers specified. Using default composition: dotnet-common + dotnet-thread-time

Provider Name Keywords Level Enabled By
Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile
Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profile
Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profiles
Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profiles

Process : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe
Output File : E:\temp\gcperfsim\trace.nettrace
Expand Down