Issue with Docker + Poetry path env variables.

[innat]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Gloal

I'm trying build a Docker file with Poetry. I'm trying build multi-stage docker with poetry in order to reduce the size of image. Here is the overall docker statements.

Docker: 2.2.0.5 (on win 10 - desktop)
Poetry: 1.2.0
Python: 3.8.10

# Dockerfile
FROM nvidia/cuda:11.0.3-devel-ubuntu20.04 AS python_base_cuda
ENV PYTHONUNBUFFERED=1 \
  PYTHONFAULTHANDLER=1 \
  PYTHONHASHSEED=random \
  PYTHONDONTWRITEBYTECODE=1 \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100 \
  POETRY_NO_INTERACTION=1 \
  POETRY_HOME="/opt/poetry" \ 
  PYSETUP_PATH="/opt/pysetup" \
  VENV_PATH="/opt/pysetup/.venv" \
  POETRY_VERSION=1.2.0 \
  POETRY_VIRTUALENVS_IN_PROJECT=true \
  POETRY_NO_INTERACTION=1 

# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"
FROM python_base_cuda AS builder-base

# Update system and install wget
RUN apt-get update && \
    DEBIAN_FRONTEND="noninteractive" apt-get -yq install --no-install-recommends -y \
        wget=1.20.3-1ubuntu2 \
        libpython3.8=3.8.10-0ubuntu1~20.04.5 \
        git=1:2.25.1-1ubuntu3.5 \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Install dependencies
RUN apt-get update && apt-get install -y \
    software-properties-common \
    curl \
    python3.8 \
    python3-pip

# System deps:
RUN pip install "poetry==$POETRY_VERSION"

WORKDIR $PYSETUP_PATH
COPY poetry.lock pyproject.toml ./

# Project initialization
RUN poetry install --only main --no-root --no-ansi

FROM python_base_cuda as development
ENV CUSTOM_DEV_ENV=development
WORKDIR $PYSETUP_PATH

# copy in our built poetry + venv
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

# quicker install as runtime deps are already installed
RUN poetry install

# Moving working files to docker file system.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

Project directory:

[others]
poetry.lock
pyproject.toml
Dockerfile

Issue

The main issue occurs at the time of the following stage

COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

at running time it gives

COPY failed: stat /var/lib/docker/overlay2/76b0353080c8120a73973315f2b3b65b4769c7dc1e128aef2b6ca190dcb99612/merged/opt/poetry: no such file or directory

Any catch? It looks like path opt/poetry is empty. Thus my suspect goes to the follow set up.

POETRY_HOME="/opt/poetry" \ 
PYSETUP_PATH="/opt/pysetup" \
VENV_PATH="/opt/pysetup/.venv" \

References

I've been following this great discussion.. cc. @michaeloliverx

[dimbleby]

Suggest you're better off taking this to the discussion.

You don't seem to be reporting a bug or feature request in poetry, rather you have questions about how to write dockerfiles

[Secrus]

You are installing Poetry via pip, which will not respect $POETRY_HOME. Only our installer script respects $POETRY_HOME.

[neersighted]

Agreed -- POETRY_HOME is only respected by the installer or if you make use of the path by creating a virtual environment there and using the copy of pip there for your pip install. I'd suggest Discord or a Discussion as the best venue if this doesn't make sense and you need further support.

[innat]

@neersighted Please transform this ticket to discussion if you've the privilege.
@Secrus Thanks for the pointer. Let me check.

[innat]

...
...
# copy in our built poetry + venv
COPY --from=builder-base $POETRY_HOME $POETRY_HOME
COPY --from=builder-base $PYSETUP_PATH $PYSETUP_PATH

# quicker install as runtime deps are already installed
RUN poetry install # <---

@neersighted

With the above dockerfile, when it times to run poetry install (after successful copy command), it's getting the following issue. Not sure if it's related to this! Any suggestion?

/opt/poetry/venv/bin/python: error while loading shared libraries: libexpat.so.1: cannot open shared object file

[neersighted]

The example build you linked uses the old get-poetry.py script which respects/installs into POETRY_HOME in a virtual environment (hence not having to explicitly create one) as well as forcing --copies/similar.

[innat]

Could you please suggest what would be the updated version of the above reference build by @michaeloliverx

[neersighted]

As documented in the docs/the script itself, the replacement for get-poetry.py is https://install.python-poetry.org

[innat]

create the virtual environment as a copy instead of symlink environment, or install the exact same Python in the final stage as was used to create the Poetry environment in builder-base.

Creating the exact python in the final stage works. But it doesn't seem intuitive to do the same operation twice. Here is the gist. How to create the virtual env as copy and make it enable in the final stage? With both of the above option, would there be any difference in image size?

[neersighted]

The official installer will create a 'fat' virtual environment with a full copy of the Python runtime. If you prefer to create an environment manually (as suggested in the docs for CI), you can use python -m venv --copies or python -m virtualenv --always-copy to do the same. That being said, the 'fat' environment will not include runtime dependencies of Python, so you still may need to install some packages depending on your base image.

If you want to avoid duplicate work in your Dockerfile, you should perform your apt-get install python in a stage that comes before your build and final stages (say, AS base-with-python) and then build your subsequent stages on top of it (FROM base-with-python).