spaCy issue when run in AWS Lambda: [ERROR] Runtime.ImportModuleError: Unable to import module '***': No module named 'en_core_web_sm'

[dnk8n]

How to reproduce the behaviour

I am using library, AllenNLP which calls spaCy. The logs show the problem:

Installing collected packages: en-core-web-sm
Successfully installed en-core-web-sm-3.1.0
�[38;5;2m✔ Download and installation successful�[0m
You can now load the package via spacy.load('en_core_web_sm')
[ERROR] Runtime.ImportModuleError: Unable to import module 'handlers': No module named 'en_core_web_sm'

To get spaCy to work on AWS lambda I mounted an EFS filestorage (it is an NFS), by following the following two instructions

• https://blog.theodo.com/2020/01/internet-access-to-lambda-in-vpc/
• https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/ (except I used user/group id as 1001 as instructed in https://docs.aws.amazon.com/lambda/latest/dg/configuration-filesystem.html

Your Environment

It's difficult to include these details from the AWS lambda environment. But I use the image method and I use 10GB RAM and 15s invocation. Here is the Dockerfile used:

FROM public.ecr.aws/lambda/python:3.9

ENV CORREFOR=https://storage.googleapis.com/allennlp-public-models/coref-spanbert-large-2021.03.10.tar.gz

RUN \
    yum install -y clang make && \
    python3 -m pip install --upgrade pip && \
    pip3 install wheel

RUN pip3 install torch==1.10.1+cpu torchvision==0.11.2+cpu torchaudio==0.10.1+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html

COPY requirements.txt .
RUN pip3 install -r requirements.txt

COPY .   .

where requirements are:

boto3
allennlp
allennlp-models

In the AWS lambda I set HOME and TMPDIR variables to point to EFS mount, e.g.

HOME=/mnt/efs/home
NLTK_DATA=/mnt/efs/home/nltk_data (needed because nltk expects the directory created during module installation to be there at runtime)
TMPDIR=/mnt/efs/tmp

then the following Python code should reproduce the problem:

from typing import Dict
import os
from pathlib import Path

from allennlp.predictors.predictor import Predictor


for var in ['NLTK_DATA', 'HOME', 'TMPDIR']:
    val = os.getenv(var)
    if val:
        Path(val).mkdir(parents=True, exist_ok=True)
        print(val, 'exists!')

correfor = Predictor.from_path(os.environ['CORREFOR'])


def handler_example(event: Dict, context: Dict):
    pass

Eventually sPacy is called, the appropriate model is downloaded (as can be seen in the logs) and then it fails to locate it. Anyone have further idea on how to debug?

Thanks.

[dnk8n]

Adding this line to the Dockerfile DOES work:

RUN python -m spacy download en_core_web_sm

So this issue can be summarised as follows:

• If AWS Lambda is being used, and a library that calls on a spaCy model is used, you are forced to preempt which model is required and include it at build time
• Letting spaCy model download at runtime does not work

I realise that this may well be an issue with AllenNLP's mechanism of ensuring a spaCy model is downloaded when required. Any feedback?

[polm]

It sounds like the issue is that the model is being downloaded and then loaded in the same process. Because models are found using entry points, which are only loaded at the start of a Python process, that generally doesn't work. That's expected behavior. Take a look at the model installation docs.

What AllenNLP could do is load the model using the directory it's downloaded to instead of the name, which woudn't rely on entry points.

[dnk8n]

E.g. Docker run command would look like this:

docker run -it --rm -e HOME=/mnt/home -e TMPDIR=/mnt/tmp --entrypoint python example_image:latest

and in the python shell, I would run the python code above.

Note: It needs around 8GB RAM

[polm]

Ah, Docker and Python entry points are unrelated concepts. See here:

https://packaging.python.org/en/latest/specifications/entry-points/

[dnk8n]

Nevertheless AllenNLP works as expected outside the Lambda environment. I was not able to reprduce the behaviour inside Docker.

[polm]

OK, not really sure what's going on then.

I guess one thing to check is, in the cases where it doesn't work, is the spaCy model actually downloaded and usable? You can check that with spacy.load("some/path"). If something weird is happening with the lambda FS or something that could explain the issue.

[dnk8n]

I think that is the point at which it fails. If time permits I can explore it. But already I am looking to alternative solutions departing from Lambda so I don't think time will permit. I think I have provided enough details to allow someone else curious to examine if they feel motivated to.