Install failure in Docker image using en_core_web_trf

[brentonmallen1]

I'm using the following docker file to create a basic image to reproduce the issue I'm coming across.

FROM python:3.10

RUN apt-get update
RUN apt-get install -y \
    g++ \
    gcc \
    python3-dev \
    build-essential \
    curl

# Get Rust
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y

RUN pip install -U pip
RUN pip install spacy==3.2.4 spacy-transformers
RUN python -m spacy download en_core_web_trf

This is the error that I get:

=> ERROR [6/7] RUN pip install spacy==3.2.4 spacy-transformers 359.1s

#9 356.9   Created wheel for spacy: filename=spacy-3.2.4-cp310-cp310-linux_aarch64.whl size=26314778 sha256=b0d7a484cfcafa8a060dc873db23c9df9b3cdf6271b81f77d41ddb0a67f7f0b1
#9 356.9   Stored in directory: /root/.cache/pip/wheels/bb/9d/b7/ab2041e3b8a2c7a2efd722efc0085f79d92190524ddaef3376
#9 356.9   Building wheel for spacy-alignments (pyproject.toml): started
#9 357.6   Building wheel for spacy-alignments (pyproject.toml): finished with status 'error'
#9 357.6   error: subprocess-exited-with-error
#9 357.6
#9 357.6   × Building wheel for spacy-alignments (pyproject.toml) did not run successfully.
#9 357.6   │ exit code: 1
#9 357.6   ╰─> [31 lines of output]
#9 357.6       running bdist_wheel
#9 357.6       running build
#9 357.6       running build_py
#9 357.6       creating build
#9 357.6       creating build/lib.linux-aarch64-cpython-310
#9 357.6       creating build/lib.linux-aarch64-cpython-310/spacy_alignments
#9 357.6       copying spacy_alignments/__init__.py -> build/lib.linux-aarch64-cpython-310/spacy_alignments
#9 357.6       creating build/lib.linux-aarch64-cpython-310/spacy_alignments/tests
#9 357.6       copying spacy_alignments/tests/__init__.py -> build/lib.linux-aarch64-cpython-310/spacy_alignments/tests
#9 357.6       copying spacy_alignments/tests/test_main.py -> build/lib.linux-aarch64-cpython-310/spacy_alignments/tests
#9 357.6       running egg_info
#9 357.6       writing spacy_alignments.egg-info/PKG-INFO
#9 357.6       writing dependency_links to spacy_alignments.egg-info/dependency_links.txt
#9 357.6       writing top-level names to spacy_alignments.egg-info/top_level.txt
#9 357.6       reading manifest file 'spacy_alignments.egg-info/SOURCES.txt'
#9 357.6       reading manifest template 'MANIFEST.in'
#9 357.6       adding license file 'LICENSE'
#9 357.6       writing manifest file 'spacy_alignments.egg-info/SOURCES.txt'
#9 357.6       running build_ext
#9 357.6       running build_rust
#9 357.6       error: can't find Rust compiler
#9 357.6
#9 357.6       If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
#9 357.6
#9 357.6       To update pip, run:
#9 357.6
#9 357.6           pip install --upgrade pip
#9 357.6
#9 357.6       and then retry package installation.
#9 357.6
#9 357.6       If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
#9 357.6       [end of output]
#9 357.6
#9 357.6   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 357.6   ERROR: Failed building wheel for spacy-alignments
#9 357.6 Successfully built spacy
#9 357.6 Failed to build spacy-alignments
#9 357.6 ERROR: Could not build wheels for spacy-alignments, which is required to install pyproject.toml-based projects
------
executor failed running [/bin/sh -c pip install spacy==3.2.4 spacy-transformers]: exit code: 1

I'm not sure what I'm doing wrong to get this to work. Is there an expectation that both spacy and spacy-alignments should be always be built?

[brentonmallen1]

It seems to work if I set the platform using --platform=linux/amd64. But I'm not sure what the implications of using that platform are.

[brentonmallen1]

This worked because I'm on an M1 mac and things would have to be compiled using the default platform.

[adrianeboyd]

This is on an M1, I'm guessing? I think the concrete problem is related to this part of the error message:

#9 357.6       error: can't find Rust compiler

I think you've installed rust but not added the compiler to your path. When you run this particular script, there is usually a message after installing that tells you to run a command like source $HOME/.cargo/env to add the newly installed tools to your path.

It's not critical that you have an extremely new version of rust here, so it might be easier to install rust instead with apt-get install rust-all or similar, which should use standard paths and avoid this issue, I think (but to be clear I haven't tested it personally).

[brentonmallen1]

Yes, you're correct. I'm on an M1, I should have mentioned that.

In an effort to try to avoid the need to compile, I specified the platform in the Dockerfile as mentioned above. This did two things for me, 1) made it so spacy didn't have to build a wheel and 2) solved the main issue I've been having with getting transformers to work.

I didn't know about setting the path so that bit is very helpful for future knowledge, thank you!