`poetry install --no-root` with only poetry.lock

[eliasmistler]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

When doing poetry install, both pyproject.toml and poetry.lock are required. This makes sense as the package itself is installed along with its dependencies.

However, this breaks caching in Docker. Let's say I have these lines in docker:

COPY pyproject.toml poetry.lock README.md /
RUN  poetry install --no-dev

(Note how I also need the README.md as that is referenced in the pyproject.toml)

The main problem here is: On every build, I bump the version using poetry version .... This changes the pyproject.toml file, therefore breaks the docker caching mechanism. In comparison, with an additional requirements.txt, I can do something like:

COPY requirements.txt /
RUN pip install -r requirements.txt
COPY pyproject.toml poetry.lock README.md /
RUN  poetry install --no-dev

In this case, all the dependencies (which quite often do not change between builds) can be cached. This speeds up my docker build by 80-90%, but it's obviously ugly to have to rely on requirements.txt

FYI: I have a workaround for requirements.txt, where I simply generate it from poetry.lock:

def _poetry_lock_to_requirements_txt(start_dir):
    with open(f'{start_dir}/poetry.lock') as f:
        lock = toml.load(f)

    requirements = {package['name']: package['version']
                    for package in lock['package']}

    with open(f'{start_dir}/requirements.txt', 'w') as f:
        f.writelines(f'{name}=={version}\n' for name, version in sorted(requirements.items()))

Alternatively, I could do some trickery around when to (re-)set the version, but this is getting uglier and uglier [plus I need requirements.txt anyway due to an other issue with poetry that I find really hard to pin down - more on that soon]

I would suggest adding the flag poetry install --dependencies-only which only requires poetry.lock, not pyproject.toml

[dmontagu]

In the preview version there is an export command to have poetry generate the requirements.txt; I’ve been using that in my own projects (and otherwise following almost exactly the same steps you described).

That said, I would also prefer to make use of this feature if it existed.

[a-recknagel]

Honest question, why would you use poetry to set up a container? pip (with export from preview builds, as dmontagu pointed out) is enough. See here for a stack overflow post where someone had a different problem with using poetry in docker, where just using pip seems a good alternative.

[iwoloschin]

I'm using poetry in a docker container because of subtle bugs like #1129. I probably could work around it by not having optional git dependencies or by creating an internal package repo and building/publishing packages, or manually editing the broken METADATA file, but it's a lot easier to just install the couple of internal dependencies straight out of our git hosting by using poetry instead of pip. Silly, but effective, and since my container is a multibuilder pattern poetry doesn't even wind up in the actual deployed image.

Separately, there's a bigger question of ergonomics, typing an extra poetry export ... command isn't really a big deal, but it's one more thing to forget before building a container. I could write my own tooling to manage that, but this could also be a great use case for a simple poetry plugin that always ran poetry export before docker build or something like that. No need to go crazy here, but if this is a common use case then it probably makes sense to have an obvious, standard way to handle it.

[a-recknagel]

This isn't related to the main post that closely, but a well-implemented solution to #537 with a poetry bundle command that allows a deployment-focused build of a poetry project would take care of your specific issue for sure (and also allow me to use poetry instead of pip in my docker containers, which I'd prefer but currently can't).

But yeah, it probably won't help with the docker cache, since a bundle wouldn't make difference between dependencies (which are mostly static between builds) and the package that is under development and changes in every build...

[dmontagu]

@a-recknagel I actually currently use poetry in all my containers used to deploy my poetry projects (despite also using requirements.txt) -- like @iwoloschin I've run into subtle issues when not using poetry directly; in particular, I previously had issues with the exported requirements.txt generating an invalid environment (I think there was an issue where the platform flags were generated wrong and I was missing one required dependency). Also, in some of my projects, I use poetry to perform C-extension compilation; I develop on macos so I also use poetry install to build the extensions in the (non-macos) deployment container.

I just use the requirements.txt to facilitate build caching, then add the pyproject.toml and poetry.lock and run poetry install (which runs much faster with the dependencies already pip installed, and the pip install step can generally stay cached for a while).

From more of an abstract perspective, since I use poetry to set up my development environment, I just use poetry to set up the deployment environment to ensure it is as similar as possible (with minimal effort). If I could spare the time/energy to optimize container size or maximally lock down the image, I would certainly try to remove poetry from the final image, but this approach has been most efficient for me in development so far.

[dmontagu]

@iwoloschin On the topic of export ergonomics, I have the following commands in a lock.sh script:

poetry lock
poetry export -f requirements.txt --without-hashes > requirements_tmp.txt
mv requirements_tmp.txt requirements/requirements-poetry.txt

(I use requirements-poetry.txt so that I can include it into a requirements.txt which also lists some pre-built wheels for local path-based installation, which I generally update more frequently than I update the non-local dependencies.)

(Also, I think the without-hashes is safe since I follow up with a poetry install during the image-building process anyway, which should check the hashes.)

In the dockerfile, it looks like:

RUN pip install "poetry==${POETRY_VERSION}"

COPY ./app/requirements/requirements-poetry.txt /app/requirements/
RUN pip install -r /app/requirements/requirements-poetry.txt

COPY ./app/requirements /app/requirements
RUN pip install \
    --find-links=/app/requirements/wheels \
    -r /app/requirements/requirements.txt

COPY ./app /app
RUN poetry install --no-dev

Since the export happens automatically whenever I lock the dependencies, I never really have issues with forgetting a step prior to building the container.

[a-recknagel]

@dmontagu I tried using both pip and poetry in my CI before, and I had a bad time due to poetry implicitly creating virtual envs on a fresh poetry install. I think it should work if the pre-poetry dependency installation with pip creates a virtualenv before, but at that point I just dropped poetry because it felt like too much work to accommodate it (and because I also like my containers as small as possible). Have you found a way around that?

I can see why having an "as similar as possible to dev" approach to build and deployment environments is a good idea, so I'd like to improve there.

[dmontagu]

@a-recknagel
You can prevent poetry from creating a virtual environment by setting the environment variable POETRY_VIRTUALENVS_CREATE=false (I think this might be preview only; not sure when this was added). Note that this can cause problems if you call poetry install --no-dev, as the no-dev will cause it to remove packages from the environment if they are dev-only in your project. In particular, this can break poetry by removing one of its own dependencies if this is performed in the same environment in which poetry was installed in your image (at least, if you use pip to install it in the image; maybe the recommended install script wouldn't have this problem).

When I have run into this problem in the past, I've worked around it by just manually pip-reinstalling the uninstalled packages after the poetry install --no-dev call.

---

Another approach I have used is to add a placeholder pyproject.toml prior to the pip install -r requirements.txt with these contents:

[tool.poetry]
name = "app"
version = "0.1.0"
description = "Placeholder"
authors = ["Placeholder"]

Then run poetry install (I think...) to create the virtual environment, then follow all the steps from the dockerfile listed above (using poetry run or modifying the path where necessary). Ultimately it results in the pyproject.toml being replaced, but the same virtual environment is still used (and plays nice with the docker cache).

---

After reminding myself of all of these tricks necessary to get this to work nicely, I'm definitely in favor of a --dependencies-only flag to poetry install that would simplify this whole process!

[a-recknagel]

POETRY_VIRTUALENVS_CREATE=false

Didn't know that, nifty. I'm using preview anyways for poetry export so that won't be a problem.

The --no-dev install will work nicely with the script-installed poetry, it vendors all its dependencies for cases like this one. I think this tips the scale for me, --dependencies-only will make everything nicer but I can keep a hack or two around until something like it has arrived.

[dmontagu]

@a-recknagel Any suggestions for how to make use of the installer script in a container? Would you just call the curl -sSL command in the dockerfile? Store it in the repo and add it to the container in the dockerfile? (For some reason it feels more dangerous to me than relying on PyPI, I'm not sure that's justified though.)

Obviously there are many ways it could be done, but I'd be interested to hear if anyone had thought through the "best" way to accomplish this.

[iwoloschin]

@dmontagu I just do this:

# Install Poetry & ensure it is in $PATH
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | POETRY_PREVIEW=1 python
ENV PATH "/root/.poetry/bin:/opt/venv/bin:${PATH}"

As you can see, I'm also using a virtual environment, because then later on I just copy the entire virtual environment out of the builder into the final container, leaving Poetry (and any other build dependencies!) behind. This is an artifact of me using an Alpine base, which makes wheels hard to use, so I need to install gcc & friends, but obviously do not want to keep those all around in the final image!

Generally the installer script portion is cached unless I do a --no-cache build. I find this to be an acceptable risk for me, but I could see a healthy debate about that subject, but it'd quickly turn into "vendor all of Poetry!" which seems unreasonable from my perspective.

[a-recknagel]

@dmontagu I was going to comment on the downside of the script installer, namely having curl in the image. Since it's just ~220KB I don't care that much about it, but since I already have a script folder in all my projects I might as well throw get-poetry.py in there and add it that way instead. Both sound a little ugly but ok to me. Since it's advertised as the recommended way to install poetry, I'd expect good or better stability than PyPI.

---

Now that I read it, @iwoloschin's approach sounds best. After installing poetry, do you run the dependency install with poetry run pip install -r requirements.txt? Mind also telling us how you copy the env, is it more complicated than moving the site_packages over into the system python?

[iwoloschin]

Sure, here's a minimal version of my dockerfile:

FROM python:3-alpine as python_builder
RUN apk update && \
  apk add \
  build-base \
  curl \
  git \
  libffi-dev \
  openssh-client \
  postgresql-dev

# Install Poetry & ensure it is in $PATH
RUN curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | POETRY_PREVIEW=1 python
ENV PATH "/root/.poetry/bin:/opt/venv/bin:${PATH}"

# Install any dependencies, this step will be cached so long as pyproject.toml is unchanged
# This could (should?) probably be changed to do a "pip install -r requirements.txt" in order
# to remain more cacheable, but it really depends on how long your dependencies take to install
COPY poetry.lock pyproject.toml /opt/project
RUN python -m venv /opt/venv && \
  source /opt/venv/bin/activate && \
  pip install -U pip && \
  cd /opt/project && \
  poetry install --no-dev --no-interaction

# Install the project itself (this is almost never cached)
COPY . /opt/project
RUN source /opt/venv/bin/activate && \
  cd /opt/project && \
  poetry install --no-dev --no-interaction

# Below this line is now creating the deployed container
# Anything installed above but not explicitly copied below is *not* available in the final container!
FROM python:3-alpine

# Any general deployed container setup that you want cached should go here

# Copy Project Virtual Environment from python_builder
COPY --from=python_builder /opt /opt

# Add the VirtualEnv to the beginning of $PATH
ENV PATH="/opt/venv/bin:$PATH"

CMD ...

This allows my container to be moderately cacheable, so long as pyproject.toml and poetry.lock remain unchanged. It could be improved by generating requirements.txt and pip install -r requirements.txt instead of the first poetry install ..., and in fact perhaps I'll go play around with @dmontagu's lock.sh idea and see if that helps me out at all. In theory, the second poetry install ... could handle any updated dependencies until you've got time to trigger a --no-cache build.

Of course, it goes without saying that the primary benefit here is that you can be "lazy" and use poetry to install your project in your container, but also get the benefit of not having poetry, or any other build-time dependencies in your final container. Eventually this should all just be replaced by a pip install command, but until Poetry matures a bit more this is an acceptable solution, particularly because it means I get to use Poetry to manage my dependencies in my development environment and not manually generate a requirements.txt file, and that is awesome!