gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` #139403

hartwork · 2025-09-28T22:21:08Z

Issue: Heap-use-after-free issue in pyexpat related to .ExternalEntityParserCreate #139400

Within libexpat, a parser created via XML_ExternalEntityParserCreate is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected by CPython. This fixes related reference counting, to stop that from happening.

picnixz

Now, this looks like Tcl/Tk and their callbacks. If you want to check for reference leaks, you should also run the tests with -R (python -m test -R the_test)

Modules/pyexpat.c

@picnixz

Idea by @picnixz

@picnixz

Idea by @picnixz

hartwork · 2025-09-28T23:12:13Z

Now, this looks like Tcl/Tk and their callbacks.

@picnixz I'm not sure what that means 😃

If you want to check for reference leaks, you should also run the tests with -R (python -m test -R the_test)

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏

picnixz · 2025-09-28T23:26:35Z

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏

Sorry, to be precise the -R : option checks if there are reference leaks. In your case run:

$ ./python -m test test_pyexpat -m test_parent_parser_outlives_its_subparsers -R :

picnixz · 2025-09-28T23:27:40Z

I'm not sure what that means

Tcl/Tk is a graphic library that allows to have delayed callbacks and the callback data is the Python object. Here, for Expat, the handler data is the XML parser itself as well. I wanted to make a parallel between those two constructions.

picnixz

I'm a bit tired (it's 01:33 AM here) so the review might not be perfect. I'll have another look tomorrow.

Modules/pyexpat.c

Lib/test/test_pyexpat.py

Modules/pyexpat.c

hartwork · 2025-09-28T23:44:46Z

I cannot figure out why or what to do with python -m test -R the_test. Could you spell it out for me? 🙏

Sorry, to be precise the -R : option checks if there are reference leaks. In your case run:
$ ./python -m test test_pyexpat -m test_parent_parser_outlives_its_subparsers -R :

@picnixz after a recompile with --with-pydebug that worked — thank you! 🙏

@picnixz

Idea by @picnixz

@picnixz

Idea by @picnixz

As requested.

hartwork · 2025-09-28T23:54:25Z

I'm a bit tired (it's 01:33 AM here) so the review might not be perfect. I'll have another look tomorrow.

@picnixz no worries, same UTC+2 here. Thank you! 👍 👍

hartwork · 2025-09-28T23:57:42Z

Related: libexpat/libexpat#1066

Misc/NEWS.d/next/Core_and_Builtins/2025-09-29-00-01-28.gh-issue-139400.X2T-jO.rst

Modules/pyexpat.c

@picnixz

As suggested by @picnixz

@picnixz

Suggested by @picnixz

Modules/pyexpat.c

@picnixz

Kudos to @picnixz

@picnixz

As suggested by @picnixz

miss-islington-app · 2025-10-05T15:38:25Z

Sorry, @hartwork and @picnixz, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6edb2ddb5f3695cf4938979d645f31d7fba43ec8 3.11

miss-islington-app · 2025-10-05T15:38:29Z

Sorry, @hartwork and @picnixz, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6edb2ddb5f3695cf4938979d645f31d7fba43ec8 3.13

miss-islington-app · 2025-10-05T15:38:32Z

Sorry, @hartwork and @picnixz, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 6edb2ddb5f3695cf4938979d645f31d7fba43ec8 3.12

picnixz · 2025-10-05T15:38:34Z

Lucky me, 3.14 has a clean backport :)

picnixz · 2025-10-05T15:46:01Z

@hartwork Could you make the backports for 3.10-3.13 please? I'm not on my dev session now (sorry for bothering you with this).

hartwork · 2025-10-05T15:46:36Z

@picnixz thanks for your help getting this over the finish line 🙏 🎉 I can look into the backports for this and report back if I hit any roadblocks.

picnixz · 2025-10-05T15:50:18Z

Thank you for writing libexpat :')

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:01:28Z

GH-139608 is a backport of this pull request to the 3.13 branch.

…ers in `pyexpat` (GH-139403) (#139606) Co-authored-by: Sebastian Pipping <[email protected]>

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:08:16Z

GH-139609 is a backport of this pull request to the 3.12 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:21:49Z

GH-139612 is a backport of this pull request to the 3.11 branch.

bedevere-app · 2025-10-05T16:22:13Z

GH-139608 is a backport of this pull request to the 3.13 branch.

bedevere-app · 2025-10-05T16:22:19Z

GH-139609 is a backport of this pull request to the 3.12 branch.

bedevere-app · 2025-10-05T16:22:25Z

GH-139612 is a backport of this pull request to the 3.11 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:27:07Z

GH-139613 is a backport of this pull request to the 3.10 branch.

…rs in `pyexpat` (python#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

bedevere-app · 2025-10-05T16:35:02Z

GH-139614 is a backport of this pull request to the 3.9 branch.

…ers in `pyexpat` (GH-139403) (GH-139608) Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd)

…ers in `pyexpat` (GH-139403) (#139612) * gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` (#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd) * Move news item from section "Core and Builtins" to section "Security"

…ers in `pyexpat` (GH-139403) (#139613) * gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` (#139403) * Modules/pyexpat.c: Disallow collection of in-use parent parsers. Within libexpat, a parser created via `XML_ExternalEntityParserCreate` is relying on its parent parser throughout its entire lifetime. Prior to this fix, is was possible for the parent parser to be garbage-collected too early. (cherry picked from commit 6edb2dd) * Move news item to from section "Core and Builtins" to section "Security"

bedevere-bot · 2025-10-07T01:50:06Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable Refleaks 3.11 (tier-1) has failed when building commit 1459d1f.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1027/builds/823) and take a look at the build logs.
Check if the failure is related to this commit (1459d1f) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1027/builds/823

Failed tests:

test_socket

Failed subtests:

test_aead_aes_gcm - test.test_socket.LinuxKernelCryptoAPI.test_aead_aes_gcm

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.11.cstratak-fedora-stable-x86_64.refleak/build/Lib/test/test_socket.py", line 6557, in test_aead_aes_gcm
    op.sendall(plain)
OSError: [Errno 22] Invalid argument

hartwork added 3 commits September 29, 2025 00:17

Add a regression test for issue 139400

3b0afda

Add news item for issue 139400

8f70991

bedevere-app bot added the awaiting review label Sep 28, 2025

bedevere-app bot mentioned this pull request Sep 28, 2025

Heap-use-after-free issue in pyexpat related to .ExternalEntityParserCreate #139400

Open

picnixz self-requested a review September 28, 2025 22:45

Improve test's docstring

a63bd34

picnixz reviewed Sep 28, 2025

View reviewed changes

Modules/pyexpat.c Outdated Show resolved Hide resolved

Modules/pyexpat.c Outdated Show resolved Hide resolved

hartwork added 2 commits September 29, 2025 01:04

Simplify xmlparseobject_ to xmlparseobject

22417ec

Idea by @picnixz

Leverage Py_CLEAR

20caacd

Idea by @picnixz

hartwork requested a review from picnixz September 28, 2025 23:13

picnixz reviewed Sep 28, 2025

View reviewed changes

Modules/pyexpat.c Outdated Show resolved Hide resolved

Lib/test/test_pyexpat.py Outdated Show resolved Hide resolved

Lib/test/test_pyexpat.py Show resolved Hide resolved

Modules/pyexpat.c Outdated Show resolved Hide resolved

hartwork added 3 commits September 29, 2025 01:45

Simplify call to Py_CLEAR

f8c58a3

Idea by @picnixz

Streamline docsting of ParentParserLifetimeTest

f1e3e36

Idea by @picnixz

pyexpat.c: Adjust whitespace around pointer

1b2937d

As requested.

hartwork requested a review from picnixz September 29, 2025 14:06

picnixz reviewed Sep 29, 2025

View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-09-29-00-01-28.gh-issue-139400.X2T-jO.rst Outdated Show resolved Hide resolved

Modules/pyexpat.c Outdated Show resolved Hide resolved

Modules/pyexpat.c Show resolved Hide resolved

Re-write news item

8c5a4e8

As suggested by @picnixz

hartwork requested a review from picnixz September 29, 2025 21:04

hartwork mentioned this pull request Sep 30, 2025

[3.14] gh-90949: add Expat API to prevent XML deadly allocations (CVE-2025-59375) (GH-139234) #139359

Open

Turn parent into a PyObject pointer

a6f5520

Suggested by @picnixz

picnixz reviewed Sep 30, 2025

View reviewed changes

Modules/pyexpat.c Outdated Show resolved Hide resolved

hartwork added 2 commits September 30, 2025 15:49

Add missing parent visit to xmlparse_traverse

4470cea

Kudos to @picnixz

Re-write comment about mission of "PyObject *parent"

3759ac7

As suggested by @picnixz

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Oct 5, 2025

hugovk pushed a commit that referenced this pull request Oct 5, 2025

[3.14] gh-139400: Make sure that parent parsers outlive their subpars…

66ca0d9

…ers in `pyexpat` (GH-139403) (#139606) Co-authored-by: Sebastian Pipping <[email protected]>

bedevere-app bot removed the needs backport to 3.12 only security fixes label Oct 5, 2025

bedevere-app bot removed the needs backport to 3.11 only security fixes label Oct 5, 2025

bedevere-app bot removed the needs backport to 3.10 only security fixes label Oct 5, 2025

hartwork mentioned this pull request Oct 6, 2025

[3.9] gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat (GH-139403) #139614

Open

Uh oh!

gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat #139403

gh-139400: Make sure that parent parsers outlive their subparsers in pyexpat #139403

Uh oh!

Conversation

hartwork commented Sep 28, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

picnixz commented Sep 28, 2025

Uh oh!

picnixz commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

hartwork commented Sep 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

miss-islington-app bot commented Oct 5, 2025

Uh oh!

miss-islington-app bot commented Oct 5, 2025

Uh oh!

miss-islington-app bot commented Oct 5, 2025

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

hartwork commented Oct 5, 2025

Uh oh!

picnixz commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-app bot commented Oct 5, 2025

Uh oh!

bedevere-bot commented Oct 7, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Uh oh!

gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` #139403

gh-139400: Make sure that parent parsers outlive their subparsers in `pyexpat` #139403

hartwork commented Sep 28, 2025 •

edited by bedevere-app bot

Loading

picnixz commented Sep 28, 2025 •

edited

Loading