Extremely slow load_model times in docker container

[drisspg]

I am currently using spacy 2.3.2 my nlp pipeline consists of 2 text cats an NER and a custom phrase matching component which populates doc entities. I have created a custom loading function that adds then custom component to the Language.factories and the calls spacy.load(drct). The saved model is stored in a directory and the whole python package is wheeled. On my machine if I run %timeit load_model() in a Jupyter notebook I see it takes around 4.5 seconds to load the full pipeline (long but manageable). I have created a flask application to serve this model and we are seeing extremely long load times in this docker container.

Docker file:

FROM python:3.7

WORKDIR /project

COPY . /project

RUN pip install -r requirements.txt

RUN python scripts/nltk_download.py

Our flask applications is failing to get scheduled on k8s because of the extreme loading times. I have ran a terminal in this docker container on my local machine where my custom wheel is installed and the result of running is

%timeit load_model() = 1min 10s

Does anyone know why there would be such a discrepancy in load times and how would go about reducing these times?

[svlandeg]

As a quick hunch - could it be that it's the PhraseMatcher causing this delay in loading times? Could you experiment with leaving it out and check the times? (just to find the culprit)

[drisspg]

In the process of our debugging we were narrowing down on the PhraseMatcher being the culprit. However before doing some more testing on that I pip freeze our last working version and found that a dependency of a dependency was not frozen and had changed. Specifically on the working version importlib-metadata==3.7.3 and on the broken version importlib-metadata==3.9.1. We for now just pinned this version and things seem to be working. I was reading this package description and not total sure of what aspect of it changed to cause this delay.

[adrianeboyd]

We ran into some unrelated test failures related to newer versions of importlib-metadata in catalogue, and in tracking it down I was surprised that there were so many recent changes and releases in a library that I thought was primarily a backport. We had tests that succeeded in python 3.8 with the built-in version but failed in python 3.6, which was pretty unexpected.

We ended up pinning it in catalogue to importlib-metadata<3.3.0 because we haven't managed to track down the exact cause of the test failure (which may well be due to the test itself and not any required functionality in importlib-metadata, since we haven't heard of any problems with it related to spacy at all), but this change hasn't made it into a catalogue release yet. If there are also performance regressions that is even more of a reason to pin the version, although in general I'd prefer not to for such a widely used library.

[adrianeboyd]

After more reports of performance regressions, we decided to publish a new release of catalogue with these pins. The next patch releases of our other libraries that require catalogue will also include this update.

For now, if you update catalogue in an existing environment or install spacy in a new environment, you should end up with importlib-metadata==3.2.0.

[svlandeg]

I guess I'll mark this thread as the "answer" just to avoid having this question marked as unanswered. That doesn't mean discussion needs to stop though - and we'd be happy to hear anyone's experience with the updated pin/version, and whether that resolves the performance issues previously reported!