Improving startup time for AWS Lambda with Spacy Transformer models

[dandiep]

I am doing some work where I am using Spacy transformers for a bunch of different languages. Because they take up a lot of memory, I am trying to deploy them on Lambda so that I don't need to have a giant server with a ton of memory constantly.

The approach I adopted is to deploy all the models on Amazon EFS so I don't have a giant docket image or a new image for every language. Then when the Lambda starts it loads them from there.

The problem I am running into is that it takes a long time to load these models from a cold start. More than 30 seconds a lot of times.

This can be somewhat worked around by setting up provisioned concurrency. But not perfect since anticipating demand is hard.

Does anyone have any tips for ways to improve startup time? I am happy make patches too if there are ideas on where to look that could help.

Thanks
Dan

[ljvmiranda921]

Hi @dandiep ,

One optimization may be to call spacy.load outside the actual Lambda handler function. I saw this AWS documentation that may be of help (be sure to download their example demo and check the classify_images.py script). The only difference is that they store the model in S3 instead of EFS. Ideally, we don't want to call spacy.load every time the lambda function is invoked. Lastly, I'm not very familiar with AWS Lambda but perhaps there are also caching capabilities via Docker?

[dandiep]

Reporting back with the findings of the last 2 days:

• Loading a model from EFS appears to be faster than S3 or a container layer (with or without a VPC)
• There is a 10 second init timeout limit. Loading a spacy transformer model in this time period is dodgy on Lambda
• If you don't make it in the 10 seconds, it restarts - extending your load time even more.
• Yes, ideally you load your model outside the handler function (i.e. during init), but because it's too slow that doesn't work.

Ultimately, I couldn't find a way to do a cold start in less than 20-30 seconds.

Perhaps if AWS releases something like SnapStart for Python in the future, it will work. Otherwise, I wouldn't waste your time with this.

If anyone else has experiences with other services that provide a serverless way to serve Spacy with a reasonable start time, would love to hear it.