`pip install hatch` fails on free-threaded build

[ngoldbaum]

This ends up coming down to the dependency on zstandard:

± pip install hatch
Collecting hatch
  Downloading hatch-1.13.0-py3-none-any.whl.metadata (5.6 kB)
Collecting click>=8.0.6 (from hatch)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting hatchling>=1.24.2 (from hatch)
  Downloading hatchling-1.26.1-py3-none-any.whl.metadata (3.8 kB)
Collecting httpx>=0.22.0 (from hatch)
  Using cached httpx-0.27.2-py3-none-any.whl.metadata (7.1 kB)
Collecting hyperlink>=21.0.0 (from hatch)
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting keyring>=23.5.0 (from hatch)
  Downloading keyring-25.5.0-py3-none-any.whl.metadata (20 kB)
Requirement already satisfied: packaging>=23.2 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (24.1)
Collecting pexpect~=4.8 (from hatch)
  Using cached pexpect-4.9.0-py2.py3-none-any.whl.metadata (2.5 kB)
Requirement already satisfied: platformdirs>=2.5.0 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (4.3.6)
Collecting rich>=11.2.0 (from hatch)
  Downloading rich-13.9.4-py3-none-any.whl.metadata (18 kB)
Collecting shellingham>=1.4.0 (from hatch)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting tomli-w>=1.0 (from hatch)
  Downloading tomli_w-1.1.0-py3-none-any.whl.metadata (5.7 kB)
Collecting tomlkit>=0.11.1 (from hatch)
  Downloading tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Collecting userpath~=1.7 (from hatch)
  Downloading userpath-1.9.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uv>=0.1.35 (from hatch)
  Downloading uv-0.5.1-py3-none-macosx_11_0_arm64.whl.metadata (11 kB)
Requirement already satisfied: virtualenv>=20.26.1 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from hatch) (20.27.1)
Collecting zstandard<1 (from hatch)
  Downloading zstandard-0.23.0.tar.gz (681 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 681.7/681.7 kB 28.5 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting packaging>=23.2 (from hatch)
  Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pathspec>=0.10.1 (from hatchling>=1.24.2->hatch)
  Downloading pathspec-0.12.1-py3-none-any.whl.metadata (21 kB)
Collecting pluggy>=1.0.0 (from hatchling>=1.24.2->hatch)
  Using cached pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
Collecting trove-classifiers (from hatchling>=1.24.2->hatch)
  Downloading trove_classifiers-2024.10.21.16-py3-none-any.whl.metadata (2.2 kB)
Collecting anyio (from httpx>=0.22.0->hatch)
  Using cached anyio-4.6.2.post1-py3-none-any.whl.metadata (4.7 kB)
Collecting certifi (from httpx>=0.22.0->hatch)
  Using cached certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx>=0.22.0->hatch)
  Using cached httpcore-1.0.6-py3-none-any.whl.metadata (21 kB)
Collecting idna (from httpx>=0.22.0->hatch)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting sniffio (from httpx>=0.22.0->hatch)
  Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting h11<0.15,>=0.13 (from httpcore==1.*->httpx>=0.22.0->hatch)
  Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting jaraco.classes (from keyring>=23.5.0->hatch)
  Downloading jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring>=23.5.0->hatch)
  Downloading jaraco.functools-4.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring>=23.5.0->hatch)
  Downloading jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Collecting ptyprocess>=0.5 (from pexpect~=4.8->hatch)
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting markdown-it-py>=2.2.0 (from rich>=11.2.0->hatch)
  Downloading markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich>=11.2.0->hatch)
  Using cached pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
Requirement already satisfied: distlib<1,>=0.3.7 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from virtualenv>=20.26.1->hatch) (0.3.9)
Requirement already satisfied: filelock<4,>=3.12.2 in /Users/goldbaum/.pyenv/versions/3.13.0t/lib/python3.13t/site-packages (from virtualenv>=20.26.1->hatch) (3.16.1)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=11.2.0->hatch)
  Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Collecting more-itertools (from jaraco.classes->keyring>=23.5.0->hatch)
  Downloading more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Downloading hatch-1.13.0-py3-none-any.whl (125 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
Downloading hatchling-1.26.1-py3-none-any.whl (75 kB)
Using cached httpx-0.27.2-py3-none-any.whl (76 kB)
Using cached httpcore-1.0.6-py3-none-any.whl (78 kB)
Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Downloading keyring-25.5.0-py3-none-any.whl (39 kB)
Using cached packaging-24.2-py3-none-any.whl (65 kB)
Using cached pexpect-4.9.0-py2.py3-none-any.whl (63 kB)
Downloading rich-13.9.4-py3-none-any.whl (242 kB)
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading tomli_w-1.1.0-py3-none-any.whl (6.4 kB)
Downloading tomlkit-0.13.2-py3-none-any.whl (37 kB)
Downloading userpath-1.9.2-py3-none-any.whl (9.1 kB)
Downloading uv-0.5.1-py3-none-macosx_11_0_arm64.whl (12.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.5/12.5 MB 29.5 MB/s eta 0:00:00
Using cached idna-3.10-py3-none-any.whl (70 kB)
Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
Downloading pathspec-0.12.1-py3-none-any.whl (31 kB)
Using cached pluggy-1.5.0-py3-none-any.whl (20 kB)
Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Using cached pygments-2.18.0-py3-none-any.whl (1.2 MB)
Using cached anyio-4.6.2.post1-py3-none-any.whl (90 kB)
Using cached sniffio-1.3.1-py3-none-any.whl (10 kB)
Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
Downloading jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Downloading trove_classifiers-2024.10.21.16-py3-none-any.whl (13 kB)
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB)
Building wheels for collected packages: zstandard
  Building wheel for zstandard (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for zstandard (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      <string>:41: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <string>:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module '_cffi_backend', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
      not modified: 'build/zstandard/_cffi.c'
      generating build/zstandard/_cffi.c
      (already up-to-date)
      running bdist_wheel
      running build
      running build_py
      creating build/lib.macosx-14.7-arm64-cpython-313
      creating build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/__init__.py -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/backend_cffi.py -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/__init__.pyi -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      copying zstandard/py.typed -> build/lib.macosx-14.7-arm64-cpython-313/zstandard
      running build_ext
      building 'zstandard.backend_c' extension
      creating build/temp.macosx-14.7-arm64-cpython-313
      creating build/temp.macosx-14.7-arm64-cpython-313/c-ext
      clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -I/opt/homebrew/opt/zlib -I/opt/homebrew/opt/zlib -Ic-ext -Izstd -I/Users/goldbaum/.pyenv/versions/3.13.0t/include/python3.13t -c c-ext/backend_c.c -o build/temp.macosx-14.7-arm64-cpython-313/c-ext/backend_c.o -DZSTD_SINGLE_FILE -DZSTDLIB_VISIBLE= -DZDICTLIB_VISIBLE= -DZSTDERRORLIB_VISIBLE= -fvisibility=hidden
      c-ext/backend_c.c:316:17: error: no member named 'ob_refcnt' in 'struct _object'
        316 |     if ((*obj)->ob_refcnt == 1) {
            |         ~~~~~~  ^
      1 error generated.
      error: command '/opt/homebrew/opt/ccache/libexec/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for zstandard
Failed to build zstandard

see indygreg/python-zstandard#231 for the zstandard issue.

It looks like zstandard is only used if you encounter a .tar.zst file. I'm not sure if there's an easy way to do this, but if it were an optional dependency then most users who don't need zstandard support could get past this issue.

[ofek]

This is a tough one, I'm going to see if the maintainer is willing to accept a contribution and release it.

[ngoldbaum]

I pinged again on the python-zstandard issue tracker.

The lack of a way to specify the free-threaded build in a pyproject.toml makes it impossible to do the most obvious fix and remove python-zstandard from hatch's dependencies on the free-threaded build. The only thing that's possible right now is to e.g. make it optional on 3.13 or newer or just remove it from the required dependencies completely and perhaps add it to an extras section.

That could work, but it's certainly annoying that users would see a runtime error about the python-zstandard library not being installed instead of just having it work on the GIL-enabled build of 3.13 as it does right now.

I would only do this as a last resort, but would you be open to adding a dependency on e.g. python-zstandard-ft-support which would track the upstream project (which doesn't change much these days) and apply the PR that adds free-threading support. Obviously not a great option but it might be the best possibility?

[ngoldbaum]

It's not pretty, but for what it's worth this is sufficient to install hatch on the free-threaded build in a fresh environment:

# python-zstandard doesn't yet support free-threaded
pip install git+https://github.com/indygreg/python-zstandard.git@refs/pull/243/head
# install cryptography build dependencies
pip install git+https://github.com/colesbury/cffi.git@ft
pip install maturin setuptools
# cryptography doesn't yet support free-threaded
pip install git+https://github.com/pyca/cryptography.git@refs/pull/12555/head --no-build-isolation

This is to work around zstandard, cryptography, and cffi all not readily supporting the free-threaded build.

[ngoldbaum]

@ofek maybe the new backport package for the zstandard support in 3.14 is a way forward here: Rogdham/backports.zstd#27

[Rogdham]

maybe the new backport package for the zstandard support in 3.14 is a way forward here: Rogdham/backports.zstd#27

I was actually planning to create a PR on hatch repo in a few weeks when I consider backports.zstd to be feature-complete (see roadmap)! Is there any urgency to move forward on this?

Happy to take any feedback/suggestions, either through any github issue or directly by email (that you can fetch from the metadata of my commits).

[ngoldbaum]

Is there any urgency to move forward on this?

Not in particular. This is just the No. 1 annoyance that users might run into related to python-zstandard not having free-threaded wheels so I wanted to make sure the connection got made.

[Rogdham]

I have created #2034, but I'm not sure how to test it properly in the context of free-threaded builds.

More precisely, it fails installing cffi dependency:

Stacktrace

(root) root@d71a5e5c4d66:~# uv pip install git+https://github.com/pypa/hatch/.git@refs/pull/2034/head
   Updating https://github.com/pypa/hatch/.git (refs/pull/2034/head)                                                                                      error: Git operation failed
  Caused by: failed to clone into: .cache/uv/git-v0/db/bb5930d96575cce6
  Caused by: failed to fetch ref `refs/pull/2034/head`
  Caused by: process didn't exit successfully: `/usr/bin/git fetch --force --update-head-ok 'https://github.com/pypa/hatch/.git' '+refs/pull/2034/head:refs/pull/2034/head'` (exit status: 128)
--- stderr
remote: Not Found
fatal: repository 'https://github.com/pypa/hatch/.git/' not found
(root) root@d71a5e5c4d66:~# uv pip install git+https://github.com/pypa/hatch.git@refs/pull/2034/head
    Updated https://github.com/pypa/hatch.git (3b1677c425e8697de71b9bc16a628c3615c78791)
Resolved 42 packages in 5.31s
      Built hatch @ git+https://github.com/pypa/hatch.git@3b1677c425e8697de71b9bc16a628c3615c78791
  x Failed to build `cffi==1.17.1`
  |-> The build backend returned an error
  `-> Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)

      [stdout]
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/pkgconfig.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/__init__.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/commontypes.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/cparser.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/lock.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/_imp_emulation.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/verifier.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/api.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/error.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/_shimmed_dist_utils.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/model.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/recompiler.py -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/_embedding.h -> build/lib.linux-x86_64-cpython-313t/cffi
      copying src/cffi/_cffi_errors.h -> build/lib.linux-x86_64-cpython-313t/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-cpython-313t/src/c
      cc -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fPIC -fPIC
      -DFFI_BUILDING=1 -I/usr/include/ffi -I/usr/include/libffi -I/root/.cache/uv/builds-v0/.tmpGPEH52/include
      -I/root/.local/share/uv/python/cpython-3.13.5+freethreaded-linux-x86_64-gnu/include/python3.13t -c src/c/_cffi_backend.c -o
      build/temp.linux-x86_64-cpython-313t/src/c/_cffi_backend.o

      [stderr]

          No working compiler found, or bogus compiler options passed to
          the compiler from Python's standard "distutils" module.  See
          the error messages above.  Likely, the problem is not related
          to CFFI but generic to the setup.py of any Python package that
          tries to compile C code.  (Hints: on OS/X 10.8, for errors about
          -mno-fused-madd see http://stackoverflow.com/questions/22313407/
          Otherwise, see https://wiki.python.org/moin/CompLangPython or
          the IRC channel #python on irc.libera.chat.)

          Trying to continue anyway.  If you are trying to install CFFI from
          a build done in a different context, you can ignore this warning.
      
      /root/.cache/uv/builds-v0/.tmpGPEH52/lib/python3.13t/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are
      deprecated.
      !!

              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:

              License :: OSI Approved :: MIT License

              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************

      !!
        self._finalize_license_expression()
      error: command 'cc' failed: No such file or directory

      hint: This usually indicates a problem with the package or the build environment.
  help: `cffi` (v1.17.1) was included because `hatch` (v1.12.1.dev52) depends on `keyring` (v25.6.0) which depends on `secretstorage` (v3.3.3) which
        depends on `cryptography` (v45.0.5) which depends on `cffi`

It looks like an issue in cffi not supporting free-threaded builds though.

[ofek]

Coming soon: python-cffi/cffi#178

[ngoldbaum]

Also see pyca/cryptography#12555. That will be merged once CFFI has a final 2.0 release and fix the build error you're hitting here.

[Carreau]

For what it is worth, on a fresh conda python-3.14rc3 env, pip install hatch; hatch works.