Loading transformer on AWS Lambda throws OMP errno 38

[djdevdev]

We have trained a Spacy 3.3.1 transformer textcat which we're deploying as an AWS Python 3.9 Docker image to AWS Lambda. The model loads and infers correctly on the Linux development host (both using a test Python script and also using AWS SAM local), but fails in the Lambda runtime with OpenMP runtime error #38 (see Lambda error output below). A search suggests this error occurs because Lambda doesn't support Python multiprocessing, specifically it doesn't mount /dev/shm, leading to the error.

Loading a blank Spacy model inside the Lambda runtime works without a problem, indicating the error occurs only for this specific trained model or perhaps for transformer models more generally.

1. The simple solution seems to simply disable multiprocessing when loading the current model. Looking through the Spacy docs we're unable to find a setting to disable multiprocessing, hoping this might sidestep the error. How can we disable OpenMP multiprocessing in a Spacy transformer model, and is this something that affects inference only or would we need to retrain the model?

2. Alternatively we may need to try Python 3.7 (suggested in links below). Can we stay on Spacy 3.3.1 if we downgrade Python?

3. As a last resort we may need to modify Spacy or OpenMP to override multiprocessing.Pool/Queue with multiprocessing.Process/Pipe which apparently do work on Lamda (suggested in links below).

Lambda error output

OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #38: Function not implemented
OMP: Error #179: Function Can't open SHM2 failed:
OMP: System error #38: Function not implemented
START RequestId: XYZ Version: $LATEST
RequestId: XYZ Error: Runtime exited with error: signal: aborted
Runtime.ExitError
END RequestId: XYZ
REPORT RequestId: XYZ	Duration: 547.37 ms	Billed Duration: 548 ms	Memory Size: 3008 MB	Max Memory Used: 142 MB

Relevant links
https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/
https://spacy.io/usage/processing-pipelines#multiprocessing
https://forum.opennmt.net/t/unable-to-create-ctranslate2-translator-in-aws-lambda/4922
https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda

[polm]

spaCy shouldn't use multiprocessing directly unless you instruct it to, but some of the libraries we use might have multiprocessing under the hood. In particular, for spacy-transformers, HuggingFace Transformers is doing a lot of things and may be the source of the issue.

To help you narrow down the cause, you could try using a non-transformer spaCy model and see if it has the same error.

[djdevdev]

Hi Paul

Thank you for your quick and helpful reply, as always.

Python 3.7 throws the same error, as does the base transformer model spacy.load("en_core_web_trf").

The non-transformer model spacy.load("en_core_web_sm") works correctly, indicating the problem lies somewhere inside the transformer model or its dependencies. I'm happy to raise an issue with HuggingFace Transformer if you're fairly certain the problem lies there; otherwise any ideas where I start looking for the culprit?

[polm]

Well, if the error doesn't happen without the transformer model, that does narrow it down to some kind of transformers-related code. I'm not familiar enough with AWS or this particular error to say exactly what's going on, but it would probably help to ask at the HuggingFace forums - someone must have run into this before.