Skip to content

Commit 969f1c5

Browse files
committed
Merge branch 'main' into uuid-v6-89083
2 parents a5682f8 + 08f98f4 commit 969f1c5

File tree

19 files changed

+401
-90
lines changed

19 files changed

+401
-90
lines changed

Doc/c-api/marshal.rst

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,12 @@ binary mode.
1313

1414
Numeric values are stored with the least significant byte first.
1515

16-
The module supports two versions of the data format: version 0 is the
17-
historical version, version 1 shares interned strings in the file, and upon
18-
unmarshalling. Version 2 uses a binary format for floating-point numbers.
19-
``Py_MARSHAL_VERSION`` indicates the current file format (currently 2).
16+
The module supports several versions of the data format; see
17+
the :py:mod:`Python module documentation <marshal>` for details.
2018

19+
.. c:macro:: Py_MARSHAL_VERSION
20+
21+
The current format version. See :py:data:`marshal.version`.
2122

2223
.. c:function:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version)
2324

Doc/library/marshal.rst

Lines changed: 46 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -38,23 +38,39 @@ supports a substantially wider range of objects than marshal.
3838
maliciously constructed data. Never unmarshal data received from an
3939
untrusted or unauthenticated source.
4040

41+
There are functions that read/write files as well as functions operating on
42+
bytes-like objects.
43+
4144
.. index:: object; code, code object
4245

4346
Not all Python object types are supported; in general, only objects whose value
4447
is independent from a particular invocation of Python can be written and read by
45-
this module. The following types are supported: booleans, integers, floating-point
46-
numbers, complex numbers, strings, bytes, bytearrays, tuples, lists, sets,
47-
frozensets, dictionaries, and code objects (if *allow_code* is true),
48-
where it should be understood that
49-
tuples, lists, sets, frozensets and dictionaries are only supported as long as
50-
the values contained therein are themselves supported. The
51-
singletons :const:`None`, :const:`Ellipsis` and :exc:`StopIteration` can also be
52-
marshalled and unmarshalled.
53-
For format *version* lower than 3, recursive lists, sets and dictionaries cannot
54-
be written (see below).
48+
this module. The following types are supported:
49+
50+
* Numeric types: :class:`int`, :class:`bool`, :class:`float`, :class:`complex`.
51+
* Strings (:class:`str`) and :class:`bytes`.
52+
:term:`Bytes-like objects <bytes-like object>` like :class:`bytearray` are
53+
marshalled as :class:`!bytes`.
54+
* Containers: :class:`tuple`, :class:`list`, :class:`set`, :class:`frozenset`,
55+
and (since :data:`version` 5), :class:`slice`.
56+
It should be understood that these are supported only if the values contained
57+
therein are themselves supported.
58+
Recursive containers are supported since :data:`version` 3.
59+
* The singletons :const:`None`, :const:`Ellipsis` and :exc:`StopIteration`.
60+
* :class:`code` objects, if *allow_code* is true. See note above about
61+
version dependence.
62+
63+
.. versionchanged:: 3.4
64+
65+
* Added format version 3, which supports marshalling recursive lists, sets
66+
and dictionaries.
67+
* Added format version 4, which supports efficient representations
68+
of short strings.
69+
70+
.. versionchanged:: next
71+
72+
Added format version 5, which allows marshalling slices.
5573

56-
There are functions that read/write files as well as functions operating on
57-
bytes-like objects.
5874

5975
The module defines these functions:
6076

@@ -140,11 +156,24 @@ In addition, the following constants are defined:
140156

141157
.. data:: version
142158

143-
Indicates the format that the module uses. Version 0 is the historical
144-
format, version 1 shares interned strings and version 2 uses a binary format
145-
for floating-point numbers.
146-
Version 3 adds support for object instancing and recursion.
147-
The current version is 4.
159+
Indicates the format that the module uses.
160+
Version 0 is the historical first version; subsequent versions
161+
add new features.
162+
Generally, a new version becomes the default when it is introduced.
163+
164+
======= =============== ====================================================
165+
Version Available since New features
166+
======= =============== ====================================================
167+
1 Python 2.4 Sharing interned strings
168+
------- --------------- ----------------------------------------------------
169+
2 Python 2.5 Binary representation of floats
170+
------- --------------- ----------------------------------------------------
171+
3 Python 3.4 Support for object instancing and recursion
172+
------- --------------- ----------------------------------------------------
173+
4 Python 3.4 Efficient representation of short strings
174+
------- --------------- ----------------------------------------------------
175+
5 Python 3.14 Support for :class:`slice` objects
176+
======= =============== ====================================================
148177

149178

150179
.. rubric:: Footnotes
@@ -154,4 +183,3 @@ In addition, the following constants are defined:
154183
around in a self-contained form. Strictly speaking, "to marshal" means to
155184
convert some data from internal to external form (in an RPC buffer for instance)
156185
and "unmarshalling" for the reverse process.
157-

Doc/library/uuid.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -342,7 +342,7 @@ The following options are accepted:
342342
is used.
343343

344344
.. versionadded:: next
345-
Allow generating UUID version 6.
345+
Allow generating UUID version 6 and 8.
346346

347347
.. option:: -n <namespace>
348348
--namespace <namespace>

Include/marshal.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromString(const char *,
1313
Py_ssize_t);
1414
PyAPI_FUNC(PyObject *) PyMarshal_WriteObjectToString(PyObject *, int);
1515

16-
#define Py_MARSHAL_VERSION 4
16+
#define Py_MARSHAL_VERSION 5
1717

1818
PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *);
1919
PyAPI_FUNC(int) PyMarshal_ReadShortFromFile(FILE *);

Lib/test/test_capi/test_object.py

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,7 @@ def test_ClearWeakRefsNoCallbacks_no_weakref_support(self):
134134
_testcapi.pyobject_clear_weakrefs_no_callbacks(obj)
135135

136136

137+
@threading_helper.requires_working_threading()
137138
class EnableDeferredRefcountingTest(unittest.TestCase):
138139
"""Test PyUnstable_Object_EnableDeferredRefcount"""
139140
@support.requires_resource("cpu")
@@ -158,21 +159,13 @@ def silly_func(obj):
158159

159160
silly_list = [1, 2, 3]
160161
threads = [
161-
Thread(target=silly_func, args=(silly_list,)) for _ in range(5)
162+
Thread(target=silly_func, args=(silly_list,)) for _ in range(4)
162163
]
163164

164-
with threading_helper.catch_threading_exception() as cm:
165-
for t in threads:
166-
t.start()
167-
165+
with threading_helper.start_threads(threads):
168166
for i in range(10):
169167
silly_list.append(i)
170168

171-
for t in threads:
172-
t.join()
173-
174-
self.assertIsNone(cm.exc_value)
175-
176169
if support.Py_GIL_DISABLED:
177170
self.assertTrue(_testinternalcapi.has_deferred_refcount(silly_list))
178171

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
import os
2+
import sys
3+
import unittest
4+
import platform
5+
6+
FOO_C = r"""
7+
#include <unistd.h>
8+
9+
/* This is a 'GNU indirect function' (IFUNC) that will be called by
10+
dlsym() to resolve the symbol "foo" to an address. Typically, such
11+
a function would return the address of an actual function, but it
12+
can also just return NULL. For some background on IFUNCs, see
13+
https://willnewton.name/uncategorized/using-gnu-indirect-functions.
14+
15+
Adapted from Michael Kerrisk's answer: https://stackoverflow.com/a/53590014.
16+
*/
17+
18+
asm (".type foo STT_GNU_IFUNC");
19+
20+
void *foo(void)
21+
{
22+
write($DESCRIPTOR, "OK", 2);
23+
return NULL;
24+
}
25+
"""
26+
27+
28+
@unittest.skipUnless(sys.platform.startswith('linux'),
29+
'Test only valid for Linux')
30+
class TestNullDlsym(unittest.TestCase):
31+
"""GH-126554: Ensure that we catch NULL dlsym return values
32+
33+
In rare cases, such as when using GNU IFUNCs, dlsym(),
34+
the C function that ctypes' CDLL uses to get the address
35+
of symbols, can return NULL.
36+
37+
The objective way of telling if an error during symbol
38+
lookup happened is to call glibc's dlerror() and check
39+
for a non-NULL return value.
40+
41+
However, there can be cases where dlsym() returns NULL
42+
and dlerror() is also NULL, meaning that glibc did not
43+
encounter any error.
44+
45+
In the case of ctypes, we subjectively treat that as
46+
an error, and throw a relevant exception.
47+
48+
This test case ensures that we correctly enforce
49+
this 'dlsym returned NULL -> throw Error' rule.
50+
"""
51+
52+
def test_null_dlsym(self):
53+
import subprocess
54+
import tempfile
55+
56+
# To avoid ImportErrors on Windows, where _ctypes does not have
57+
# dlopen and dlsym,
58+
# import here, i.e., inside the test function.
59+
# The skipUnless('linux') decorator ensures that we're on linux
60+
# if we're executing these statements.
61+
from ctypes import CDLL, c_int
62+
from _ctypes import dlopen, dlsym
63+
64+
retcode = subprocess.call(["gcc", "--version"],
65+
stdout=subprocess.DEVNULL,
66+
stderr=subprocess.DEVNULL)
67+
if retcode != 0:
68+
self.skipTest("gcc is missing")
69+
70+
pipe_r, pipe_w = os.pipe()
71+
self.addCleanup(os.close, pipe_r)
72+
self.addCleanup(os.close, pipe_w)
73+
74+
with tempfile.TemporaryDirectory() as d:
75+
# Create a C file with a GNU Indirect Function (FOO_C)
76+
# and compile it into a shared library.
77+
srcname = os.path.join(d, 'foo.c')
78+
dstname = os.path.join(d, 'libfoo.so')
79+
with open(srcname, 'w') as f:
80+
f.write(FOO_C.replace('$DESCRIPTOR', str(pipe_w)))
81+
args = ['gcc', '-fPIC', '-shared', '-o', dstname, srcname]
82+
p = subprocess.run(args, capture_output=True)
83+
84+
if p.returncode != 0:
85+
# IFUNC is not supported on all architectures.
86+
if platform.machine() == 'x86_64':
87+
# It should be supported here. Something else went wrong.
88+
p.check_returncode()
89+
else:
90+
# IFUNC might not be supported on this machine.
91+
self.skipTest(f"could not compile indirect function: {p}")
92+
93+
# Case #1: Test 'PyCFuncPtr_FromDll' from Modules/_ctypes/_ctypes.c
94+
L = CDLL(dstname)
95+
with self.assertRaisesRegex(AttributeError, "function 'foo' not found"):
96+
# Try accessing the 'foo' symbol.
97+
# It should resolve via dlsym() to NULL,
98+
# and since we subjectively treat NULL
99+
# addresses as errors, we should get
100+
# an error.
101+
L.foo
102+
103+
# Assert that the IFUNC was called
104+
self.assertEqual(os.read(pipe_r, 2), b'OK')
105+
106+
# Case #2: Test 'CDataType_in_dll_impl' from Modules/_ctypes/_ctypes.c
107+
with self.assertRaisesRegex(ValueError, "symbol 'foo' not found"):
108+
c_int.in_dll(L, "foo")
109+
110+
# Assert that the IFUNC was called
111+
self.assertEqual(os.read(pipe_r, 2), b'OK')
112+
113+
# Case #3: Test 'py_dl_sym' from Modules/_ctypes/callproc.c
114+
L = dlopen(dstname)
115+
with self.assertRaisesRegex(OSError, "symbol 'foo' not found"):
116+
dlsym(L, "foo")
117+
118+
# Assert that the IFUNC was called
119+
self.assertEqual(os.read(pipe_r, 2), b'OK')
120+
121+
122+
if __name__ == "__main__":
123+
unittest.main()

Lib/test/test_gc.py

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1082,6 +1082,44 @@ def __del__(self):
10821082
gc.collect()
10831083
self.assertTrue(collected)
10841084

1085+
def test_traverse_frozen_objects(self):
1086+
# See GH-126312: Objects that were not frozen could traverse over
1087+
# a frozen object on the free-threaded build, which would cause
1088+
# a negative reference count.
1089+
x = [1, 2, 3]
1090+
gc.freeze()
1091+
y = [x]
1092+
y.append(y)
1093+
del y
1094+
gc.collect()
1095+
gc.unfreeze()
1096+
1097+
def test_deferred_refcount_frozen(self):
1098+
# Also from GH-126312: objects that use deferred reference counting
1099+
# weren't ignored if they were frozen. Unfortunately, it's pretty
1100+
# difficult to come up with a case that triggers this.
1101+
#
1102+
# Calling gc.collect() while the garbage collector is frozen doesn't
1103+
# trigger this normally, but it *does* if it's inside unittest for whatever
1104+
# reason. We can't call unittest from inside a test, so it has to be
1105+
# in a subprocess.
1106+
source = textwrap.dedent("""
1107+
import gc
1108+
import unittest
1109+
1110+
1111+
class Test(unittest.TestCase):
1112+
def test_something(self):
1113+
gc.freeze()
1114+
gc.collect()
1115+
gc.unfreeze()
1116+
1117+
1118+
if __name__ == "__main__":
1119+
unittest.main()
1120+
""")
1121+
assert_python_ok("-c", source)
1122+
10851123

10861124
class IncrementalGCTests(unittest.TestCase):
10871125

Lib/test/test_marshal.py

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,13 @@ def helper(self, sample, *extra):
2828
finally:
2929
os_helper.unlink(os_helper.TESTFN)
3030

31+
def omit_last_byte(data):
32+
"""return data[:-1]"""
33+
# This file's code is used in CompatibilityTestCase,
34+
# but slices need marshal version 5.
35+
# Avoid the slice literal.
36+
return data[slice(0, -1)]
37+
3138
class IntTestCase(unittest.TestCase, HelperMixin):
3239
def test_ints(self):
3340
# Test a range of Python ints larger than the machine word size.
@@ -241,7 +248,8 @@ def test_bug_5888452(self):
241248
def test_patch_873224(self):
242249
self.assertRaises(Exception, marshal.loads, b'0')
243250
self.assertRaises(Exception, marshal.loads, b'f')
244-
self.assertRaises(Exception, marshal.loads, marshal.dumps(2**65)[:-1])
251+
self.assertRaises(Exception, marshal.loads,
252+
omit_last_byte(marshal.dumps(2**65)))
245253

246254
def test_version_argument(self):
247255
# Python 2.4.0 crashes for any call to marshal.dumps(x, y)
@@ -594,6 +602,19 @@ def testNoIntern(self):
594602
s2 = sys.intern(s)
595603
self.assertNotEqual(id(s2), id(s))
596604

605+
class SliceTestCase(unittest.TestCase, HelperMixin):
606+
def test_slice(self):
607+
for obj in (
608+
slice(None), slice(1), slice(1, 2), slice(1, 2, 3),
609+
slice({'set'}, ('tuple', {'with': 'dict'}, ), self.helper.__code__)
610+
):
611+
with self.subTest(obj=str(obj)):
612+
self.helper(obj)
613+
614+
for version in range(4):
615+
with self.assertRaises(ValueError):
616+
marshal.dumps(obj, version)
617+
597618
@support.cpython_only
598619
@unittest.skipUnless(_testcapi, 'requires _testcapi')
599620
class CAPI_TestCase(unittest.TestCase, HelperMixin):
@@ -654,7 +675,7 @@ def test_read_last_object_from_file(self):
654675
self.assertEqual(r, obj)
655676

656677
with open(os_helper.TESTFN, 'wb') as f:
657-
f.write(data[:1])
678+
f.write(omit_last_byte(data))
658679
with self.assertRaises(EOFError):
659680
_testcapi.pymarshal_read_last_object_from_file(os_helper.TESTFN)
660681
os_helper.unlink(os_helper.TESTFN)
@@ -671,7 +692,7 @@ def test_read_object_from_file(self):
671692
self.assertEqual(p, len(data))
672693

673694
with open(os_helper.TESTFN, 'wb') as f:
674-
f.write(data[:1])
695+
f.write(omit_last_byte(data))
675696
with self.assertRaises(EOFError):
676697
_testcapi.pymarshal_read_object_from_file(os_helper.TESTFN)
677698
os_helper.unlink(os_helper.TESTFN)
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Fix error handling in :class:`ctypes.CDLL` objects
2+
which could result in a crash in rare situations.
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Fix crash during garbage collection on an object frozen by :func:`gc.freeze` on the
2+
free-threaded build.

0 commit comments

Comments
 (0)