gh-135953: Add Gecko reporter to sampling profiler #139364

ivonastojanovic · 2025-09-26T15:40:45Z

Issue: Expose internal stack introspection APIs as a statistical runtime analysis tool #135953

pablogsal · 2025-09-26T15:43:46Z

Copilot

Pull Request Overview

This PR adds Gecko reporter functionality to the sampling profiler, enabling export of profiling data in Gecko Profile format for web-based visualization tools. The implementation includes a new GeckoCollector class and supporting data structures.

Adds new gecko_format.py module with dataclasses and builder for Gecko profile format
Implements GeckoCollector class in stack_collector.py that exports profiling data as JSON
Updates sample.py to support --gecko command line option and integrate the new collector

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
Lib/profiling/sampling/gecko_format.py	New module defining Gecko profile format dataclasses and builder
Lib/profiling/sampling/stack_collector.py	Adds GeckoCollector class and category constants
Lib/profiling/sampling/sample.py	Integrates Gecko collector option and updates CLI help text

Comments suppressed due to low confidence (1)

Lib/profiling/sampling/stack_collector.py:1

GeckoCollector is instantiated without the skip_idle parameter that other collectors receive. Consider whether skip_idle functionality should be supported for consistency with other collectors.

import base64

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Lib/profiling/sampling/gecko_format.py

Lib/profiling/sampling/stack_collector.py

bedevere-app · 2025-09-26T15:45:01Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pablogsal · 2025-09-27T15:36:26Z

There are several problems here:

The file doesn't really work. If you try to lead it into the firefox profiler it just fails with "Couldn’t read the file or parse the profile in it".
Generating the profiler with ./python -m profiling.sampling -i 1 -d100000 --realtime-stats --all-threads --gecko -o flamegraph.gecko -m test test_asyncio takes 2GB of memory. That's unfortunately not acceptable
Generating the profiler when the process ends is super slow (takes 40 seconds on my machine).

pablogsal · 2025-09-27T16:16:33Z

This also will need some tests

pablogsal · 2025-09-27T16:27:48Z

@ivonastojanovic I have pushed a draft version of a simpler collector that uses raw dictionaries. This can be processed by the firefox profiler, doesn't blow up the memory in my experiments and also is relatively faster. It needs some polishing but should cover most problems.

Lib/profiling/sampling/gecko_collector.py

Signed-off-by: Pablo Galindo Salgado <[email protected]>

pablogsal · 2025-10-01T20:19:02Z

Great work @ivonastojanovic !

bedevere-bot · 2025-10-01T21:15:32Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x (tier-1) has failed when building commit 75b1afe.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/506/builds/11528) and take a look at the build logs.
Check if the failure is related to this commit (75b1afe) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/506/builds/11528

Failed tests:

test_profiling
test_pyrepl

Failed subtests:

test_sample_target_script - test.test_profiling.test_sampling_profiler.TestSampleProfilerIntegration.test_sample_target_script
test_repl_eio - test.test_pyrepl.test_unix_console.TestUnixConsoleEIOHandling.test_repl_eio

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__
    _warn("subprocess %s is still running" % self.pid,
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: subprocess 3224727 is still running
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_profiling/test_sampling_profiler.py", line 1924, in test_sample_target_script
    self.assertIn("slow_fibonacci", output)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'slow_fibonacci' not found in 'Captured 9986 samples in 1.00 seconds\nSample rate: 9985.99 samples/sec\nError rate: 0.00%\nWarning: missed 14 samples from the expected total of 10000 (0.14%)\n\x1b[1;34mProfile Stats:\x1b[0m\n\x1b[1;34m       nsamples\x1b[0m  \x1b[1;34m sample%\x1b[0m  \x1b[1;34mtottime (ms)\x1b[0m  \x1b[1;34m  cumul%\x1b[0m  \x1b[1;34mcumtime (ms)\x1b[0m  \x1b[1;34mfilename:lineno(function)\x1b[0m\n         0/9986       0.0         0.000     100.0       998.600  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m63\x1b[0m(\x1b[36m<module>\x1b[0m)\n         0/9918       0.0         0.000      99.3       991.800  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m52\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n      5005/5005      50.1       500.500      50.1       500.500  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m15\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n      3232/3232      32.4       323.200      32.4       323.200  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m16\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n      1651/1651      16.5       165.100      16.5       165.100  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m14\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n           1/68       0.0         0.100       0.7         6.800  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m40\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n           0/68       0.0         0.000       0.7         6.800  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m58\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n           0/67       0.0         0.000       0.7         6.700  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m38\x1b[0m(\x1b[36mnested_calls.<locals>.level1.<locals>.level2\x1b[0m)\n           0/67       0.0         0.000       0.7         6.700  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m39\x1b[0m(\x1b[36mnested_calls.<locals>.level1\x1b[0m)\n          42/42       0.4         4.200       0.4         4.200  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m24\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n          30/30       0.3         3.000       0.3         3.000  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m17\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n          25/25       0.3         2.500       0.3         2.500  \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m23\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n\n\x1b[1;34mLegend:\x1b[0m\n  \x1b[33mnsamples\x1b[0m: Direct/Cumulative samples (direct executing / on call stack)\n  \x1b[33msample%\x1b[0m: Percentage of total samples this function was directly executing\n  \x1b[33mtottime\x1b[0m: Estimated total time spent directly in this function\n  \x1b[33mcumul%\x1b[0m: Percentage of total samples when this function was on the call stack\n  \x1b[33mcumtime\x1b[0m: Estimated cumulative time (including time in called functions)\n  \x1b[33mfilename:lineno(function)\x1b[0m: Function location and name\n\n\x1b[1;34mSummary of Interesting Functions:\x1b[0m\n\n\x1b[1;34mFunctions with Highest Direct/Cumulative Ratio (Hot Spots):\x1b[0m\n  1.000 direct/cumulative ratio, 99.3% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n  1.000 direct/cumulative ratio, 0.7% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n  0.015 direct/cumulative ratio, 0.0% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n\n\x1b[1;34mFunctions with Highest Call Frequency (Indirect Calls):\x1b[0m\n  9986 indirect calls, 100.0% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36m<module>\x1b[0m)\n  9986 indirect calls, 100.0% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n  67 indirect calls, 0.7% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n\n\x1b[1;34mFunctions with Highest Call Magnification (Cumulative/Direct):\x1b[0m\n  68.0x call magnification, 67 indirect calls from 1 direct: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n'


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb' closefd=True>
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=10 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__
    _warn("subprocess %s is still running" % self.pid,
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ResourceWarning, source=self)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: subprocess 3247738 is still running
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test
    regrtest_runner(result, test_func, runtests)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner
    test_result = test_func()
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_func
    return run_unittest(test_mod, runtests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittest
    return _run_suite(tests)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suite
    raise support.TestFailedWithDetails(err, errors, failures, stats=stats)


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True>


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/test_unix_console.py", line 362, in test_repl_eio
    _, err = proc.communicate(timeout=5)  # sleep for pty to settle
             ~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1219, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 2126, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1266, in _check_timeout
    raise TimeoutExpired(
    ...<2 lines>...
            stderr=b''.join(stderr_seq) if stderr_seq else None)
subprocess.TimeoutExpired: Command '['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-E', '-S', '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/eio_test_script.py']' timed out after 5 seconds

pablogsal · 2025-10-01T23:34:33Z

This looks unrelated to this PR as this PR only adds test_gecko_collector_basic

bedevere-app bot added the awaiting review label Sep 26, 2025

bedevere-app bot mentioned this pull request Sep 26, 2025

Expose internal stack introspection APIs as a statistical runtime analysis tool #135953

Open

pablogsal requested review from Copilot and pablogsal September 26, 2025 15:43

Copilot AI reviewed Sep 26, 2025

View reviewed changes

Lib/profiling/sampling/gecko_format.py Outdated Show resolved Hide resolved

Lib/profiling/sampling/stack_collector.py Outdated Show resolved Hide resolved

pablogsal requested changes Sep 26, 2025

View reviewed changes

Lib/profiling/sampling/stack_collector.py Outdated Show resolved Hide resolved

bedevere-app bot added awaiting changes and removed awaiting review labels Sep 26, 2025

pablogsal force-pushed the gecko_reporter branch from cf36ebb to cc770b6 Compare September 27, 2025 16:26

pablogsal force-pushed the gecko_reporter branch from cc770b6 to fe9adcd Compare September 27, 2025 16:29

ivonastojanovic force-pushed the gecko_reporter branch 2 times, most recently from 1b3b603 to 352ea21 Compare September 29, 2025 09:52

Wulian233 suggested changes Sep 29, 2025

View reviewed changes

Lib/profiling/sampling/gecko_collector.py Outdated Show resolved Hide resolved

pythongh-135953: Add Gecko format to sampling profiler

35c4e5c

Signed-off-by: Pablo Galindo Salgado <[email protected]>

pablogsal force-pushed the gecko_reporter branch from 44ec90d to 35c4e5c Compare October 1, 2025 18:07

pablogsal removed the awaiting changes label Oct 1, 2025

Add NEWS entry

4947d41

pablogsal approved these changes Oct 1, 2025

View reviewed changes

bedevere-app bot added the awaiting merge label Oct 1, 2025

pablogsal merged commit 75b1afe into python:main Oct 1, 2025
47 checks passed

bedevere-app bot removed the awaiting merge label Oct 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-135953: Add Gecko reporter to sampling profiler #139364

gh-135953: Add Gecko reporter to sampling profiler #139364

ivonastojanovic commented Sep 26, 2025 •

edited by bedevere-app bot

Loading

Uh oh!

pablogsal commented Sep 26, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app bot commented Sep 26, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

Uh oh!

Uh oh!

pablogsal commented Oct 1, 2025

Uh oh!

bedevere-bot commented Oct 1, 2025

Uh oh!

pablogsal commented Oct 1, 2025

Uh oh!

Uh oh!

Uh oh!

gh-135953: Add Gecko reporter to sampling profiler #139364

gh-135953: Add Gecko reporter to sampling profiler #139364

Conversation

ivonastojanovic commented Sep 26, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Sep 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app bot commented Sep 26, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

pablogsal commented Sep 27, 2025

Uh oh!

Uh oh!

Uh oh!

pablogsal commented Oct 1, 2025

Uh oh!

bedevere-bot commented Oct 1, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

pablogsal commented Oct 1, 2025

Uh oh!

Uh oh!

ivonastojanovic commented Sep 26, 2025 •

edited by bedevere-app bot

Loading