Skip to content

PYTHON-4450 Support free-threaded Python with no-GIL #1906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Oct 10, 2024
Merged

PYTHON-4450 Support free-threaded Python with no-GIL #1906

merged 9 commits into from
Oct 10, 2024

Conversation

ShaneHarvey
Copy link
Member

@ShaneHarvey ShaneHarvey commented Oct 8, 2024 •
edited
Loading

PYTHON-4450 Support free-threaded Python with no-GIL

References:

So far I've tested python3.13t locally on macos and haven't seen any failures.

New cp313t builds:

ls -l all-dist-11245136974/ | grep 'cp313' 
-rw-r--r--@ 1 shane  staff   943263 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-macosx_10_13_x86_64.whl
-rw-r--r--@ 1 shane  staff   943055 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-macosx_11_0_arm64.whl
-rw-r--r--@ 1 shane  staff  1917248 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
-rw-r--r--@ 1 shane  staff  1993617 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
-rw-r--r--@ 1 shane  staff  1957979 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl
-rw-r--r--@ 1 shane  staff  1918913 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r--@ 1 shane  staff  1868610 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
-rw-r--r--@ 1 shane  staff   904180 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-win32.whl
-rw-r--r--@ 1 shane  staff   926873 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313-win_amd64.whl
-rw-r--r--@ 1 shane  staff   999155 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-macosx_10_13_x86_64.whl
-rw-r--r--@ 1 shane  staff   999264 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-macosx_11_0_arm64.whl
-rw-r--r--@ 1 shane  staff  2242289 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
-rw-r--r--@ 1 shane  staff  2327332 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
-rw-r--r--@ 1 shane  staff  2286736 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl
-rw-r--r--@ 1 shane  staff  2238466 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r--@ 1 shane  staff  2178033 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
-rw-r--r--@ 1 shane  staff   952423 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-win32.whl
-rw-r--r--@ 1 shane  staff   981048 Oct  8  2024 pymongo-4.11.0.dev0-cp313-cp313t-win_amd64.whl

TODO:

  • Add a test for python3.13t.
  • Confirm that the test suite is actually running with the GIL disabled.

    The new sys._is_gil_enabled() function can be used to check whether the GIL is actually disabled in the running process.

@ShaneHarvey ShaneHarvey marked this pull request as draft October 8, 2024 23:10
@ShaneHarvey ShaneHarvey marked this pull request as ready for review October 9, 2024 19:57
@blink1073
Copy link
Member

I think we should add a test in GHA at the very least: https://py-free-threading.github.io/ci/

@ShaneHarvey
Copy link
Member Author

ShaneHarvey commented Oct 9, 2024 •
edited
Loading

Okay I added 3.13t testing. I had to remove hatch because it depends on zstandard which fails to build on 3.13t. For example:

$ pip install hatch
Collecting hatch
  Using cached hatch-1.12.0-py3-none-any.whl.metadata (5.5 kB)
Collecting click>=8.0.6 (from hatch)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting hatchling>=1.24.2 (from hatch)
  Using cached hatchling-1.25.0-py3-none-any.whl.metadata (3.8 kB)
Collecting httpx>=0.22.0 (from hatch)
  Using cached httpx-0.27.2-py3-none-any.whl.metadata (7.1 kB)
Collecting hyperlink>=21.0.0 (from hatch)
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting keyring>=23.5.0 (from hatch)
  Using cached keyring-25.4.1-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: packaging>=23.2 in /Users/shane/work/pycharm/pymongo-3.13t/lib/python3.13t/site-packages (from hatch) (24.1)
Collecting pexpect~=4.8 (from hatch)
  Using cached pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB)
Collecting platformdirs>=2.5.0 (from hatch)
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Collecting rich>=11.2.0 (from hatch)
  Using cached rich-13.9.2-py3-none-any.whl.metadata (18 kB)
Collecting shellingham>=1.4.0 (from hatch)
  Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting tomli-w>=1.0 (from hatch)
  Using cached tomli_w-1.1.0-py3-none-any.whl.metadata (5.7 kB)
Collecting tomlkit>=0.11.1 (from hatch)
  Using cached tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Collecting userpath~=1.7 (from hatch)
  Using cached userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uv>=0.1.35 (from hatch)
  Using cached uv-0.4.20-py3-none-macosx_11_0_arm64.whl.metadata (11 kB)
Collecting virtualenv>=20.26.1 (from hatch)
  Using cached virtualenv-20.26.6-py3-none-any.whl.metadata (4.5 kB)
Collecting zstandard<1 (from hatch)
  Using cached zstandard-0.23.0.tar.gz (681 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pathspec>=0.10.1 (from hatchling>=1.24.2->hatch)
  Using cached pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Requirement already satisfied: pluggy>=1.0.0 in /Users/shane/work/pycharm/pymongo-3.13t/lib/python3.13t/site-packages (from hatchling>=1.24.2->hatch) (1.5.0)
Collecting trove-classifiers (from hatchling>=1.24.2->hatch)
  Using cached trove_classifiers-2024.9.12-py3-none-any.whl.metadata (2.2 kB)
Collecting anyio (from httpx>=0.22.0->hatch)
  Using cached anyio-4.6.0-py3-none-any.whl.metadata (4.6 kB)
Collecting certifi (from httpx>=0.22.0->hatch)
  Using cached certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx>=0.22.0->hatch)
  Using cached httpcore-1.0.6-py3-none-any.whl.metadata (21 kB)
Collecting idna (from httpx>=0.22.0->hatch)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting sniffio (from httpx>=0.22.0->hatch)
  Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.22.0->hatch)
  Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting jaraco.classes (from keyring>=23.5.0->hatch)
  Using cached jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring>=23.5.0->hatch)
  Using cached jaraco.functools-4.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring>=23.5.0->hatch)
  Using cached jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Collecting ptyprocess>=0.5 (from pexpect~=4.8->hatch)
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting markdown-it-py>=2.2.0 (from rich>=11.2.0->hatch)
  Using cached markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich>=11.2.0->hatch)
  Using cached pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv>=20.26.1->hatch)
  Using cached distlib-0.3.9-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting filelock<4,>=3.12.2 (from virtualenv>=20.26.1->hatch)
  Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=11.2.0->hatch)
  Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Collecting more-itertools (from jaraco.classes->keyring>=23.5.0->hatch)
  Using cached more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Using cached hatch-1.12.0-py3-none-any.whl (125 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached hatchling-1.25.0-py3-none-any.whl (84 kB)
Using cached httpx-0.27.2-py3-none-any.whl (76 kB)
Using cached httpcore-1.0.6-py3-none-any.whl (78 kB)
Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Using cached keyring-25.4.1-py3-none-any.whl (38 kB)
Using cached pexpect-4.9.0-py2.py3-none-any.whl (63 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached rich-13.9.2-py3-none-any.whl (242 kB)
Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Using cached tomli_w-1.1.0-py3-none-any.whl (6.4 kB)
Using cached tomlkit-0.13.2-py3-none-any.whl (37 kB)
Using cached userpath-1.9.2-py3-none-any.whl (9.1 kB)
Using cached uv-0.4.20-py3-none-macosx_11_0_arm64.whl (11.8 MB)
Using cached virtualenv-20.26.6-py3-none-any.whl (6.0 MB)
Using cached distlib-0.3.9-py2.py3-none-any.whl (468 kB)
Using cached filelock-3.16.1-py3-none-any.whl (16 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Using cached markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
Using cached pathspec-0.12.1-py3-none-any.whl (31 kB)
Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Using cached pygments-2.18.0-py3-none-any.whl (1.2 MB)
Using cached anyio-4.6.0-py3-none-any.whl (89 kB)
Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Using cached jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Using cached jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Using cached trove_classifiers-2024.9.12-py3-none-any.whl (13 kB)
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Using cached more_itertools-10.5.0-py3-none-any.whl (60 kB)
Building wheels for collected packages: zstandard
  Building wheel for zstandard (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for zstandard (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module '_cffi_backend', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
      not modified: 'build/zstandard/_cffi.c'
      generating build/zstandard/_cffi.c
      (already up-to-date)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.macosx-10.13-universal2-cpython-313
      creating build/lib.macosx-10.13-universal2-cpython-313/zstandard
      copying zstandard/__init__.py -> build/lib.macosx-10.13-universal2-cpython-313/zstandard
      copying zstandard/backend_cffi.py -> build/lib.macosx-10.13-universal2-cpython-313/zstandard
      copying zstandard/__init__.pyi -> build/lib.macosx-10.13-universal2-cpython-313/zstandard
      copying zstandard/py.typed -> build/lib.macosx-10.13-universal2-cpython-313/zstandard
      running build_ext
      building 'zstandard.backend_c' extension
      creating build/temp.macosx-10.13-universal2-cpython-313
      creating build/temp.macosx-10.13-universal2-cpython-313/c-ext
      clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -arch arm64 -arch x86_64 -Ic-ext -Izstd -I/Users/shane/work/pycharm/pymongo-3.13t/include -I/Library/Frameworks/PythonT.framework/Versions/3.13/include/python3.13t -c c-ext/backend_c.c -o build/temp.macosx-10.13-universal2-cpython-313/c-ext/backend_c.o -DZSTD_SINGLE_FILE -DZSTDLIB_VISIBLE= -DZDICTLIB_VISIBLE= -DZSTDERRORLIB_VISIBLE= -fvisibility=hidden
      c-ext/backend_c.c:316:17: error: no member named 'ob_refcnt' in 'struct _object'
          if ((*obj)->ob_refcnt == 1) {
              ~~~~~~  ^
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for zstandard
Failed to build zstandard
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (zstandard)

blink1073
blink1073 approved these changes Oct 10, 2024
View reviewed changes
Copy link
Member

@blink1073 blink1073 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ShaneHarvey ShaneHarvey merged commit 8f26f43 into mongodb:master Oct 10, 2024
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blink1073 blink1073 blink1073 approved these changes

@Jibola Jibola Awaiting requested review from Jibola

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ShaneHarvey @blink1073