Support Python 3.12

[sschuberth]

Description

ScanCode fails to get installed on Fedora Linux 39 with Python 3.12.

How To Reproduce

Run sudo -H pip install scancode-toolkit==32.0.8. Result:

  Building wheel for intbitset (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for intbitset (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [184 lines of output]
      <string>:37: SyntaxWarning: invalid escape sequence '\s'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-312
      copying intbitset/intbitset_helper.py -> build/lib.linux-x86_64-cpython-312
      copying intbitset/intbitset_version.py -> build/lib.linux-x86_64-cpython-312
      running egg_info
      writing intbitset/intbitset.egg-info/PKG-INFO
      writing dependency_links to intbitset/intbitset.egg-info/dependency_links.txt
      writing requirements to intbitset/intbitset.egg-info/requires.txt
      writing top-level names to intbitset/intbitset.egg-info/top_level.txt
      reading manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.sh'
      warning: no files found matching '*.yml'
      warning: no files found matching '*.css' under directory 'docs/_themes'
      warning: no files found matching '*.css_t' under directory 'docs/_themes'
      warning: no files found matching '*.conf' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_themes'
      warning: no files found matching 'COPYING' under directory 'docs/_themes'
      warning: no files found matching 'README' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_templates'
      adding license file 'LICENSE'
      adding license file 'AUTHORS.rst'
      adding license file 'README.rst'
      adding license file 'CHANGELOG.rst'
      writing manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
      running build_ext
      building 'intbitset' extension
      creating build/temp.linux-x86_64-cpython-312
      creating build/temp.linux-x86_64-cpython-312/intbitset
      gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fPIC -I/usr/include/python3.12 -c intbitset/intbitset.c -o build/temp.linux-x86_64-cpython-312/intbitset/intbitset.o
      intbitset/intbitset.c: In function ‘__pyx_pf_9intbitset_9intbitset___cinit__’:
      intbitset/intbitset.c:2290:7: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       2290 |       __Pyx_GetModuleGlobalName(__pyx_t_6, __pyx_n_s_array); if (unlikely(!__pyx_t_6)) __PYX_ERR(0, 189, __pyx_L3_error)
            |       ^~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /usr/include/python3.12/dictobject.h:90,
                       from /usr/include/python3.12/Python.h:61,
                       from intbitset/intbitset.c:6:
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:2325:13: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       2325 |             __Pyx_GetModuleGlobalName(__pyx_t_5, __pyx_n_s_array); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 191, __pyx_L16_error)
            |             ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:2375:13: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       2375 |             __Pyx_GetModuleGlobalName(__pyx_t_9, __pyx_n_s_zlib); if (unlikely(!__pyx_t_9)) __PYX_ERR(0, 193, __pyx_L16_error)
            |             ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:2405:13: warning: ‘PyObject_AsReadBuffer’ is deprecated [-Wdeprecated-declarations]
       2405 |             __pyx_t_4 = ((PyObject_AsReadBuffer(__pyx_v_tmp, (&__pyx_v_buf), (&__pyx_v_size)) < 0) != 0);
            |             ^~~~~~~~~
      In file included from /usr/include/python3.12/Python.h:100:
      /usr/include/python3.12/abstract.h:369:17: note: declared here
        369 | PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj,
            |                 ^~~~~~~~~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__pyx_pf_9intbitset_9intbitset_44__reduce__’:
      intbitset/intbitset.c:8182:3: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       8182 |   __Pyx_GetModuleGlobalName(__pyx_t_1, __pyx_n_s__12); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 516, __pyx_L1_error)
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__pyx_f_9intbitset_9intbitset_fastdump’:
      intbitset/intbitset.c:9379:3: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       9379 |   __Pyx_GetModuleGlobalName(__pyx_t_2, __pyx_n_s_zlib); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 569, __pyx_L1_error)
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__pyx_f_9intbitset_9intbitset_fastload’:
      intbitset/intbitset.c:9583:7: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       9583 |       __Pyx_GetModuleGlobalName(__pyx_t_1, __pyx_n_s_array); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 580, __pyx_L3_error)
            |       ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:9633:7: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       9633 |       __Pyx_GetModuleGlobalName(__pyx_t_2, __pyx_n_s_zlib); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 583, __pyx_L3_error)
            |       ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:9663:7: warning: ‘PyObject_AsReadBuffer’ is deprecated [-Wdeprecated-declarations]
       9663 |       __pyx_t_9 = ((PyObject_AsReadBuffer(__pyx_v_tmp, (&__pyx_v_buf), (&__pyx_v_size)) < 0) != 0);
            |       ^~~~~~~~~
      /usr/include/python3.12/abstract.h:369:17: note: declared here
        369 | PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj,
            |                 ^~~~~~~~~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__pyx_pf_9intbitset_18intbitset_iterator___cinit__’:
      intbitset/intbitset.c:13551:3: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      13551 |   __Pyx_GetModuleGlobalName(__pyx_t_1, __pyx_n_s_CFG_INTBITSET_ENABLE_SANITY_CHEC); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 775, __pyx_L1_error)
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__pyx_pymod_exec_intbitset’:
      intbitset/intbitset.c:15258:3: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      15258 |   __Pyx_GetModuleGlobalName(__pyx_t_2, __pyx_n_s_CFG_INTBITSET_ENABLE_SANITY_CHEC); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 167, __pyx_L1_error)
            |   ^~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx_Raise’:
      intbitset/intbitset.c:15725:34: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
      15725 |         PyObject* tmp_tb = tstate->curexc_traceback;
            |                                  ^~
      intbitset/intbitset.c:15728:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
      15728 |             tstate->curexc_traceback = tb;
            |                   ^~
      intbitset/intbitset.c: In function ‘__Pyx_get_tp_dict_version’:
      intbitset/intbitset.c:15743:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      15743 |     return likely(dict) ? __PYX_GET_DICT_VERSION(dict) : 0;
            |     ^~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx_get_object_dict_version’:
      intbitset/intbitset.c:15755:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      15755 |     return (dictptr && *dictptr) ? __PYX_GET_DICT_VERSION(*dictptr) : 0;
            |     ^~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx_object_dict_version_matches’:
      intbitset/intbitset.c:15759:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      15759 |     if (unlikely(!dict) || unlikely(tp_dict_version != __PYX_GET_DICT_VERSION(dict)))
            |     ^~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx__GetModuleGlobalName’:
      intbitset/intbitset.c:15776:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      15776 |     __PYX_UPDATE_DICT_CACHE(__pyx_d, result, *dict_cached_value, *dict_version)
            |     ^~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx_CLineForTraceback’:
      intbitset/intbitset.c:17178:9: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      17178 |         __PYX_PY_DICT_LOOKUP_IF_MODIFIED(
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c:17178:9: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
      17178 |         __PYX_PY_DICT_LOOKUP_IF_MODIFIED(
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      /usr/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      intbitset/intbitset.c: In function ‘__Pyx_PyInt_As_int’:
      intbitset/intbitset.c:17479:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      17479 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c:17534:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      17534 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c: In function ‘__Pyx_PyInt_As_unsigned_int’:
      intbitset/intbitset.c:17713:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      17713 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c:17768:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      17768 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c: In function ‘__Pyx_PyInt_As_long’:
      intbitset/intbitset.c:17947:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      17947 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c:18002:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      18002 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      intbitset/intbitset.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
      intbitset/intbitset.c:18453:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
      18453 |     const digit* digits = ((PyLongObject*)b)->ob_digit;
            |                                             ^~
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for intbitset

System configuration

• What OS are you running on? Fedora Linux 39 (KDE spin), amd64
• What version of scancode-toolkit was used to generate the scan file? n/a
• What installation method was used to install/run scancode? pip

[AyanSinhaMahapatra]

@sschuberth thanks for the report, python 3.12 is not supported yet by SCTK (and some of it's dependencies) nor tested in the CI, but should be supported soon enough.

[sschuberth]

Thanks for the quick reply @AyanSinhaMahapatra! I'm always surprised by how much breaks with Python's small version updates... 😞

[AyanSinhaMahapatra]

This is because of the performance oriented native C dependencies like intbitset, as these need to be supported and released for each new python version (idk about breaking with python small version updates, this only requires updates once a year for python minor version updates, which are not so minor), the situation should improve a bit with #3210

[xsuchy]

@sschuberth Note that Fedora already uses Python 3.12 and intbitset is packaged there. So you can proceed and workaround it with:

dnf install python3-intbitset
pip install scancode-toolkit

In fact, all scancode-toolkit deps are already packaged and we are in the progress of packaging the scancode-toolkit itself.

[pombredanne]

@xsuchy re:

Note that Fedora already uses Python 3.12 and intbitset is packaged there. So you can proceed and workaround it with:

This is interesting and great to know. I actually maintain intbitset and did not knew this was packaged in Fedora! ❤️ 🙇 awesome!

[xsuchy]

FYI, the package review for Fedora is tracked in https://bugzilla.redhat.com/show_bug.cgi?id=2235055

[armijnhemel]

This is also a problem when building from Git (which is what I do). Fedora 38 will be EOL soon (mid-May 2024) and since Fedora 39 the minimum Python version is 3.12.

[armijnhemel]

#3658 fixed the build problems for me.