Enable Free-threaded Python (FTP)

[jannikmi]

https://py-free-threading.github.io/

[aureliobarbosa]

I did some investigation on this.

Looking into the dependencies of this project, keeping in mind free-threaded python version 3.14, the situation is the following:

dependencies = [
    "numpy>=2",            # Support is improving on each version. Numpy 2.4 supports python 3.11-14
    "h3>=4",               # Status unknown, nothing is mentioned on h3-py repository
    "cffi<3,>=1.15.1",     # Supports Python 3.14 free-threaded on version 2.0.
    "flatbuffers>=25.2.10",# Doesn't mention.
]

Considering additional dependencies, installing numba comes with problems. Only the latest numba (Version 0.65.0, 31 March 2026) release accepts free-threaded python 3.14. So, only recently people got it to work on this package. Also, when I installed numba version 0.65 on my environment using uv, the tool automatically selected numpy version 2.4.4 (which is the latest one).

While running the complete test suit pytest complains about h3 not declaring that it can run on free-threaded python:

❯ uv run make testall
platform linux -- Python 3.14.4, pytest-9.0.3, pluggy-1.6.0
**<frozen importlib._bootstrap>:491
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'h3._cy.cells', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.**
=========================== 2341 passed, 2 skipped, 1 warning in 110.63s (0:01:50) ============================

But it also indicates that it is possible to force execution (at your own risk!) using PYTHON_GIL=0...

By doing this all regular tests passes. It doen't mean much, since it should be necessary to implement specific parallel tests.

timezonefinder on  free_thread_python [!] is 📦 v8.2.2 via 🐍 v3.14.4 (timezonefinder) 
❯ PYTHON_GIL=0 uv run make testall
============================================= test session starts =============================================
platform linux -- Python 3.14.4, pytest-9.0.3, pluggy-1.6.0
configfile: pyproject.toml
collected 2343 items                                                                                          

tests/cli_test.py ......                                                                                [  0%]
tests/global_functions_test.py ........................................................................ [  3%]
....................................................................................................... [  7%]
....................................................................................................... [ 12%]
....................................................................................................... [ 16%]
....................................................................................................... [ 20%]
....................................................................................................... [ 25%]
.....................................                                                                   [ 26%]
tests/main_test.py .................................................................................... [ 30%]
....................................................................................................... [ 34%]
....................................................................................................... [ 39%]
....................................................................................................... [ 43%]
....................................................................................................... [ 48%]
....................................................................................................... [ 52%]
....................................................................................................... [ 56%]
....................................................................................................... [ 61%]
....................................................................................................... [ 65%]
....................................................................................................... [ 70%]
....................................................................................................... [ 74%]
....................................................................................................... [ 78%]
....................................................................................................... [ 83%]
                                                                                                        [ 83%]
tests/shortcut_test.py ................                                                                 [ 83%]
tests/test_compatibility.py .                                                                           [ 83%]
tests/test_example_scripts.py .s.s                                                                      [ 84%]
tests/test_flatbuf_polygon_utils.py ......                                                              [ 84%]
tests/test_integration.py ...                                                                           [ 84%]
tests/test_optimized_hybrid_shortcuts.py .................                                              [ 85%]
tests/test_package_contents.py ........................................................................ [ 88%]
....................................................................................................... [ 92%]
....................................................................................................... [ 97%]
................................                                                                        [ 98%]
tests/test_resource_management.py ..                                                                    [ 98%]
tests/utils_test.py ..................................                                                  [100%]

================================= 2341 passed, 2 skipped in 109.69s (0:01:49) =================================

I think this points into some directions. Next step would involve creating parallelized code tests using this library. But possibly pytest-run-parallel package can help with this.

[jannikmi]

Thanks for you investigation! Let's raise an issue with h3 about this.

[aureliobarbosa]

Hi @jannikmi

I continue the investigation: pytest-run-parallel plugin was installed and executed. There were some failings due to files .claude and .devcontainer being inserted on wheel... maybe they could be included on .gitignore and be avoided on setuptools configuration file (never done that, but I can do a PR if necessary - claude actually gave an script to do that in one run but I just let it go). But there were four errors on 'test_optimized_hybrid_shortcuts.py ' which certainly are related to the work being done in parallel.

Right now I am interested in learning multithread programming. If you think it should be relevant I have time to investigate the four errors on test_optimized_hybrid_shortcuts.py and the associated code.

Showing the header and footer of the output:

timezonefinder on use_pytest_run_parallel_plugin [!?] is 📦 v8.2.2 via 🐍 v3.14.0 (timezonefinder) took 15m45s 
❯ PYTHON_GIL=0 uv run pytest --parallel-threads=10
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.14.0, pytest-9.0.2, pluggy-1.6.0
rootdir: #######
configfile: pyproject.toml
plugins: run-parallel-0.8.2
collected 2347 items                                                                                                                                                                                                                         
Collected 2347 items to run in parallel
...
tests/test_integration.py eee                                                                                                                                                                                                          [ 84%]
tests/test_optimized_hybrid_shortcuts.py ee····ee······ee·                                                                                                                                                                             [ 85%]
tests/test_package_contents.py e············································································e·················································································e······································· [ 93%]
··································e·····e·········································································e                                                                                                                    [ 98%]
...
********************************************************************************************************* pytest-run-parallel report *********************************************************************************************************
All tests were run in parallel! 🎉
========================================================================================================== short test summary info ===========================================================================================================
PARALLEL FAILED tests/test_integration.py::test_install_from_artifacts[wheel] - subprocess.CalledProcessError: Command '['/tmp/pytest-of-aureliobarbosa/pytest-0/integration-venv0/venv/bin/pip', 'install', '/home/aureliobarbosa/mycode/opensource/timezonefinder/dist/timezonefinder-8.2.2-cp314-cp314t-linux_x86_64.w...
PARALLEL FAILED tests/test_integration.py::test_install_from_artifacts[sdist] - subprocess.CalledProcessError: Command '['/tmp/pytest-of-aureliobarbosa/pytest-0/integration-venv0/venv/bin/pip', 'install', '/home/aureliobarbosa/mycode/opensource/timezonefinder/dist/timezonefinder-8.2.2.tar.gz']' returned non-zero...
PARALLEL FAILED tests/test_integration.py::test_install_from_artifacts[source] - subprocess.CalledProcessError: Command '['/tmp/pytest-of-aureliobarbosa/pytest-0/integration-venv0/venv/bin/pip', 'install', '/home/aureliobarbosa/mycode/opensource/timezonefinder']' returned non-zero exit status 1.
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_write_read_roundtrip[zone_id_dtype0] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_write_read_roundtrip[zone_id_dtype1] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_empty_data[zone_id_dtype0] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_empty_data[zone_id_dtype1] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_single_element_arrays_should_not_occur[zone_id_dtype0] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_optimized_hybrid_shortcuts.py::TestOptimizedHybridShortcuts::test_single_element_arrays_should_not_occur[zone_id_dtype1] - struct.error: unpack_from requires a buffer of at least 4 bytes for unpacking 4 bytes at offset 0 (actual buffer size is 0)
PARALLEL FAILED tests/test_package_contents.py::test_no_unwanted_files_in_distribution[sdist-.hypothesis/] - FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/home/aureliobarbosa/mycode/opensource/timezonefinder/dist')
PARALLEL FAILED tests/test_package_contents.py::test_no_unwanted_files_in_distribution[wheel-.hypothesis/] - FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/home/aureliobarbosa/mycode/opensource/timezonefinder/dist')
PARALLEL FAILED tests/test_package_contents.py::test_essential_files_in_distribution[sdist-expected_file5] - AssertionError: Essential file '.claude/settings.local.json' not found in sdist.
PARALLEL FAILED tests/test_package_contents.py::test_essential_files_in_distribution[sdist-expected_file79] - AssertionError: Essential file '.devcontainer/devcontainer.json' not found in sdist.
PARALLEL FAILED tests/test_package_contents.py::test_essential_files_in_distribution[wheel-expected_file5] - AssertionError: Essential file '.claude/settings.local.json' not found in wheel.
PARALLEL FAILED tests/test_package_contents.py::test_essential_files_in_distribution[wheel-expected_file79] - AssertionError: Essential file '.devcontainer/devcontainer.json' not found in wheel.
================================================================================================ 2332 passed, 15 errors in 1821.40s (0:30:21) ================================================================================================

[aureliobarbosa]

Thanks for you investigation! Let's raise an issue with h3 about this.

Just did it.

[jannikmi]

@aureliobarbosa thanks for continuing your work on this.

The main change which is required from my point of view is opening the binary files in a thread safe manner. Parallel seek and read operations must not interfere with each other.

After a short glimpse this also seems to be the cause of the test errors you posted here (end of file reached)

[aureliobarbosa]

I took a look on some examples and thought that the first step would be to add pytest-run-parallel as a dependency on group test and to mark tests which happened to fail due to parallel issues as failures (mark them as parallel code, so that it doesn't burden the CI, and mark them as failing). Would you mind if I proceed in this direction?

[jannikmi]

Sure. Go Ahead