Cross-stack profiling for Go <-> CGO <- Rust

[Elvis339]

Is your feature request related to a problem? Please describe.

Firewood (Rust) is consumed by AvalancheGo (Go) through CGO. Current profiling tools only see one side:

• Go pprof: shows C.fwd_propose_on_db as a black box
• Rust profilers: no visibility into which Go code triggered the call

We cannot trace a hot path across the boundary. Performance work requires guessing or correlating two separate profiles manually.

Describe the solution you'd like

Unified profiling that shows Go -> CGO -> Rust in one call stack.

Describe alternatives you've considered

• Rust-side metrics: timing only, no call stacks
• Separate profiles: manual correlation, error-prone

Additional context

Prototype working on macOS with Instruments. Requirements:

1. Rust FFI with frame pointers (-C force-frame-pointers=yes)
2. Go binary with debug symbols (-gcflags="all=-N -l")
3. Local FFI via go.mod replace (module cache lacks debug info)

Linux (perf) workflow needs validation.

Progress (review in order)

1. feat(vm_reexecute): add pprof profiling support avalanchego#4790 - runs reexecution with optional pprof flag
2. Es/cross stack profiling #1583 - builds Firewood with debug symbols enabling profiling visibility
3. feat(reexecution): add optional profiling support to benchmarking script avalanchego#4791 - Connects Es/cross stack profiling #1583 to reexecution preserving Rust debug info

[AminR443]

Are you working on this? I already have profiling scripts with perf that I used few months ago, and it's pretty much the steps that you've described. Let me know the state of the work please so I can share existing work, and we don't duplicate work.

I planned on cleaning up and creating a PR on all performance profilings I did, so for future cases we don't redo them. I'll still be doing that as it covers different profilings, such as memory and disk.

[Elvis339]

Are you working on this? I already have profiling scripts with perf that I used few months ago, and it's pretty much the steps that you've described. Let me know the state of the work please so I can share existing work, and we don't duplicate work.

I planned on cleaning up and creating a PR on all performance profilings I did, so for future cases we don't redo them. I'll still be doing that as it covers different profilings, such as memory and disk.

Hey Amin,

Great that you already have perf scripts that will save time on the Linux side.

pprof is enabled in reexecution but shows CGO calls as a black box.

Showing top 10 nodes out of 147
      flat  flat%   sum%        cum   cum%
   335.56s 56.49% 56.49%    335.56s 56.49%  <unknown>
    94.38s 15.89% 72.38%     94.48s 15.91%  runtime.cgocall <-- black box
    49.22s  8.29% 80.67%     49.23s  8.29%  syscall.syscall
    33.02s  5.56% 86.23%     33.02s  5.56%  runtime.pthread_cond_wait
    16.11s  2.71% 88.94%     16.11s  2.71%  runtime.pthread_cond_signal
    11.28s  1.90% 90.84%     11.28s  1.90%  runtime.usleep
     8.34s  1.40% 92.24%      8.34s  1.40%  [c]
     4.91s  0.83% 93.07%      4.91s  0.83%  syscall.syscall6
     4.78s   0.8% 93.88%      6.91s  1.16%  runtime.greyobject
     4.44s  0.75% 94.62%     16.87s  2.84%  runtime.scanobject

The workflow is validated on macOS (Instruments/xctrace) since that's what we have locally in the org for quick iteration. We can now generate unmangled flamegraphs showing the full Go -> CGO -> Rust call stack.

Next steps:

• CI integration to produce pprof (cpu, mem) and flamegraphs as artifacts
• Add profiling tools to nix

Your existing perf work would be really helpful for the Linux validation. Happy to sync and make sure we build on what you have rather than duplicate effort.

What kind of artifacts are you producing?