Add support for the newly released Python 3.10 and 3.11

[pombredanne]

This should work out of the box but there are many small things to do:

• ensure that our code works on Linux (syntax
• ensure that the code of all deps works on Linux
• recompile and publish wheels for all native on linux, windows and macos
• setup CI
• publish app archives for 3.8 - 3.11 on linux, windows, mac

Of note are some requirements meaning that some older Linux versions may not be able to run 3.10

[jakajancar]

The docs say python 3.10 is supported in several places, but running codescan results in:

/opt/scancode/configure: line 232: python3.8: command not found

It would be good to update the docs with relevant instructions.

[pombredanne]

@jakajancar yes! can you tell what's you setup? OS/Arch/Python version? Which download did you use? And how did you installed ScanCode?

[jakajancar]

Here's a minimal repro:

# codescan.Dockerfile

FROM --platform=amd64 ubuntu:22.04

# deps from https://scancode-toolkit.readthedocs.io/en/stable/getting-started/install.html#install-prerequisites
# section "On Debian and Debian-based distros run"
# there's also a section "On Ubuntu 16, 18, 20 and 22 run", but python-dev doesn't exist on Ubuntu 22:04
RUN apt-get update && apt-get install -y python3-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0

RUN apt-get install -y curl
RUN curl -sSL -o /opt/scancode.tar.gz https://github.com/nexB/scancode-toolkit/releases/download/v31.2.1/scancode-toolkit-v31.2.1_py3.8-linux.tar.gz
RUN tar -xv -f /opt/scancode.tar.gz --one-top-level=/opt/scancode --strip-components 1
RUN /opt/scancode/scancode

results in:

# docker build -f scancode.Dockerfile .
[+] Building 153.2s (9/9) FINISHED                                                                                                                                                                           
 => [internal] load build definition from scancode.Dockerfile                                                                                                                                           0.0s
 => => transferring dockerfile: 727B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                                                         0.2s
 => CACHED [1/6] FROM docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a91d098f7f3f9285415a9                                                                            0.0s
 => [2/6] RUN apt-get update && apt-get install -y python3-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0                                                                            85.4s
 => [3/6] RUN apt-get install -y curl                                                                                                                                                                  40.3s 
 => [4/6] RUN curl -sSL -o /opt/scancode.tar.gz https://github.com/nexB/scancode-toolkit/releases/download/v31.2.1/scancode-toolkit-v31.2.1_py3.8-linux.tar.gz                                         21.8s 
 => [5/6] RUN tar -xv -f /opt/scancode.tar.gz --one-top-level=/opt/scancode --strip-components 1                                                                                                        3.9s 
 => ERROR [6/6] RUN /opt/scancode/scancode                                                                                                                                                              1.5s 
------                                                                                                                                                                                                       
 > [6/6] RUN /opt/scancode/scancode:                                                                                                                                                                         
#9 1.259 * Configuring ScanCode for first use...                                                                                                                                                             
#9 1.522 /opt/scancode/configure: line 232: python3.8: command not found                                                                                                                                     
------                                                                                                                                                                                                       
executor failed running [/bin/sh -c /opt/scancode/scancode]: exit code: 127

Maybe I should be downloading another release, but I only see scancode-toolkit-v31.2.1_py3.8-linux.tar.gz in releases.

[pombredanne]

We used to have bundles for all supported Python version and supported all OS/arch combos... these were 15 combos. If you use Python 3.10, you could use a pip install... or more simply install Python 3.8 in your container? may be with the deadsnakes ppa?

We could also re-create a few bundles but not for every combos IMHO that was too much.

[jakajancar]

I see there's a PYTHON_EXECUTABLE file. Is just doing echo python3 >/opt/scancode/PYTHON_EXECUTABLE a good solution?

It seems to work, but I'm skeptical because I assume you don't just use python3 instead of python3.8 or autodetect in configure for a reason.

[pombredanne]

I see there's a PYTHON_EXECUTABLE file. Is just doing echo python3 >/opt/scancode/PYTHON_EXECUTABLE a good solution?

Yes.

It seems to work, but I'm skeptical because I assume you don't just use python3 instead of python3.8 or autodetect in configure for a reason.

ScanCode supports Python from 3.7 to 3.10 alright (and likely 3.11 too) but we only made available a bundle for 3.8.

[AyanSinhaMahapatra]

inveniosoftware-contrib/intbitset#81 is fixed but not released so I'm getting the same error on python 3.11 trying to install scancode. We need that for python 3.11 support.