diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index b2f6512906ca9..616f3d7e01754 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,7 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* Delivers the same experience on Windows, Linux, or macOS. +* On Linux, provides additional integration with kernel user_events for native tracing tool compatibility. ## Options @@ -55,15 +55,12 @@ The `dotnet-trace` tool: Displays the version of the dotnet-trace utility. -- **`--duration`** - - How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds. - ## Commands | Command | |-----------------------------------------------------------| | [dotnet-trace collect](#dotnet-trace-collect) | +| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) | | [dotnet-trace convert](#dotnet-trace-convert) | | [dotnet-trace ps](#dotnet-trace-ps) | | [dotnet-trace list-profiles](#dotnet-trace-list-profiles) | @@ -76,16 +73,27 @@ Collects a diagnostic trace from a running process or launches a child process a ### Synopsis ```dotnetcli -dotnet-trace collect [--buffersize ] [--clreventlevel ] [--clrevents ] +dotnet-trace collect + [--buffersize ] + [--clreventlevel ] + [--clrevents ] [--dsrouter ] - [--format ] [-h|--help] [--duration dd:hh:mm:ss] - [-n, --name ] [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profile ] [--providers ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] [-- ] (for target applications running .NET 5 or later) - [--show-child-io] [--resume-runtime] + [--show-child-io] + [--resume-runtime] [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] + [--event-filters ] ``` ### Options @@ -158,7 +166,7 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- - **`--dsrouter {ios|ios-sim|android|android-emu}** - Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. + Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. - **`--format {Chromium|NetTrace|Speedscope}`** @@ -204,11 +212,11 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: - | Profile | Description | - |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| - |`gc-verbose`|Tracks GC collections and samples object allocations.| - |`gc-collect`|Tracks GC collections only at very low overhead.| + | Profile | Description | + |---------|-------------| + |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| - **`--providers `** @@ -249,6 +257,34 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- A string, parsed as [payload_field_name]:[payload_field_value] pairs separated by commas, that will stop the trace upon hitting an event containing all specified payload pairs. Requires `--stopping-event-provider-name` and `--stopping-event-event-name` to be set. for example, `--stopping-event-provider-name Microsoft-Windows-DotNETRuntime --stopping-event-event-name Method/JittingStarted --stopping-event-payload-filter MethodNameSpace:Program,MethodName:OnButtonClick` to stop the trace upon the first `Method/JittingStarted` event for the method `OnButtonClick` in the `Program` namespace emitted by the `Microsoft-Windows-DotNETRuntime` event provider. +- **`--event-filters `** + + Defines an additional optional filter for each provider's events. When no `--event-filters` is specified for a provider, all events allowed by the provider's keywords and level configuration are collected. Event filters provide additional granular control beyond the keyword/level filtering. + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `Enable` : Boolean value indicating whether EventIds will be enabled or disabled, defaults to false + - `EventIds`: Plus-delimited event IDs to enable or disable, defaults to empty. + + **Examples:** + ``` + # Scenario: Disable specific events from Microsoft-Windows-DotNETRuntime + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3+4+5+6+7+8+9" + + # Scenario: Enable specific events from a provider + --event-filters "Microsoft-Windows-DotNETRuntime:true:80+129+130+250" + # Only events 80, 129, 130, and 250 will be collected from this provider (others are filtered out) + + # Scenario: Multiple providers with mixed filtering - some providers have no filters + --providers "Microsoft-Windows-DotNETRuntime:0xFFFFFFFF:5,System.Threading.Tasks.TplEventSource:0xFFFFFFFF:5,MyCustomProvider:0xFFFFFFFF:5" + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3,System.Threading.Tasks.TplEventSource:true:7+8+9" + # Microsoft-Windows-DotNETRuntime: All events EXCEPT 1,2,3 are collected + # System.Threading.Tasks.TplEventSource: ONLY events 7,8,9 are collected + # MyCustomProvider: ALL events are collected (no filter specified - follows provider keywords/level) + ``` + > [!NOTE] > - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. @@ -259,6 +295,138 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- > - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped. +## dotnet-trace collect-linux + +Collects diagnostic traces from .NET applications using Linux user_events as a transport layer. This command provides the same functionality as [`dotnet-trace collect`](#dotnet-trace-collect) but routes .NET runtime events through the Linux kernel's user_events subsystem before writing them to `.nettrace` files. + +This transport approach enables automatic unification of user-space .NET events with kernel-space system events, since both are captured in the same kernel tracing infrastructure. Linux tools like `perf` and `ftrace` can monitor events in real-time while maintaining full compatibility with existing .NET profiling workflows. + +### Prerequisites + +- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) +- Appropriate permissions to access `/sys/kernel/tracing/user_events_data` +- .NET 10+ + +### Synopsis + +```dotnetcli +dotnet-trace collect-linux + [--buffersize ] + [--clreventlevel ] + [--clrevents ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] + [-- ] (for target applications running .NET 10 or later) + [--show-child-io] + [--resume-runtime] + [--stopping-event-provider-name ] + [--stopping-event-event-name ] + [--stopping-event-payload-filter ] + [--event-filters ] + [--tracepoint-configs ] + [--kernel-events ] +``` + +### Options + +`dotnet-trace collect-linux` supports all the same options as [`dotnet-trace collect`](#dotnet-trace-collect), excluding `--dsrouter`, and additionally offers: + +- **`--tracepoint-configs ` (required)** + + Defines the explicit mapping between EventPipe providers and kernel tracepoints. Each provider in `--providers` must have a corresponding entry in `--tracepoint-configs` + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `DefaultTracepointName`: Default tracepoint name for this provider (can be empty to require explicit assignment) + - `TracepointSets`: Semi-colon delimited `TracepointName=` + - `EventIds`: Plus-delimited event IDs to route to that tracepoint + + > [!NOTE] + > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + + > [!TIP] + > Use `--event-filters` to disable specific events before they are routed to tracepoints. Event filtering happens before tracepoint routing - only events that pass the filter will be sent to their assigned tracepoints. + + **Examples:** + ``` + # Scenario: All events from provider go to a default tracepoint + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + # All enabled events from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_dotnet_runtime + + # Scenario: Split events by categories + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3;jit_events=10+11+12" + # EventIDs 1, 2, and 3 will be written to Microsoft_Windows_DotNETRuntime_gc_events + # EventIDs 10, 11, and 12 will be written to Microsoft_Windows_DotNETRuntime_jit_events + + # Multiple providers (comma-separated) + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3,MyCustomProvider:custom_events" + # EventIds 1, 2, and 3 from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_gc_events + # All enabled events from MyCustomProvider will be written to MyCustomProvider_custom_events + ``` + +- **`--kernel-events ` (optional)** + + A comma-separated list of kernel event categories to include in the trace. These events are automatically grouped into kernel-named tracepoints. Available categories include: + + | Category | Description | Linux Tracepoints | + |----------|-------------|-------------------| + | `syscalls` | System call entry/exit events | `syscalls:sys_enter_*`, `syscalls:sys_exit_*` | + | `sched` | Process scheduling events | `sched:sched_switch`, `sched:sched_wakeup` | + | `net` | Network-related events | `net:netif_rx`, `net:net_dev_xmit` | + | `fs` | Filesystem I/O events | `ext4:*`, `vfs:*` | + | `mm` | Memory management events | `kmem:*`, `vmscan:*` | + + These events correspond to Linux kernel tracepoints documented in the [Linux kernel tracing documentation](https://www.kernel.org/doc/html/latest/trace/index.html). For more details on available tracepoints, see [ftrace](https://www.kernel.org/doc/html/latest/trace/ftrace.html) and [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html). + + Example: `--kernel-events syscalls,sched,net` + +### Linux Integration + +**Tracepoint Configuration Requirements:** + +- **Mandatory Mapping**: Every provider must be explicitly mapped to at least a default tracepoint and/or exclusive tracepoint sets via `--tracepoint-configs` +- **Tracepoint Isolation**: Each tracepoint can only receive events from one provider +- **Event Routing**: Different event IDs within a provider can be routed to different tracepoints for granular control +- **Automatic Prefixing**: All tracepoint names are prefixed with the provider name to avoid collisions + +**Kernel Integration Points:** + +The kernel tracepoints can be accessed through standard Linux tracing interfaces: + +- **ftrace**: `/sys/kernel/tracing/events/user_events/` +- **perf**: Use `perf list user_events*` to see available events +- **System monitoring tools**: Any tool that can consume Linux tracepoints + +### Examples + +```dotnetcli +# All runtime events to one tracepoint +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8000:5 \ + --kernel-events syscalls,sched \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + +# Split runtime events by category +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8001:5 \ + --kernel-events syscalls,sched,net,fs \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::exception_events=80;gc_events=1+2" + +# Multiple providers +dotnet-trace collect-linux --process-id 1234 \ + --providers "Microsoft-Windows-DotNETRuntime:0x8001:5,MyCustomProvider:0xFFFFFFFF:5" \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime,MyCustomProvider:custom_events" +``` + ## dotnet-trace convert Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools. diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index 14e7dcbc6975a..fd6b2a32d9dc6 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -214,7 +214,7 @@ into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/ Trace completed. ``` - dotnet-trace uses the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in + dotnet-trace uses a comma-delimited variant of the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in the `--providers` argument. For more options on how to take traces using dotnet-trace, see the [dotnet-trace docs](./dotnet-trace.md#collect-a-trace-with-dotnet-trace).