LambdaSurprise

What happens when you use a background thread in an AWS lambda?

This repository was created as part of a blog post about using Open Source Software without giving consideration to whether it is fit for the execution model. In my case, hours of confusion ensued as a consequence of wanting to iterate quickly, not reading the SDK source code and not giving thought to whether the SDK was right for use in an AWS Lambda.

TLDR

Background threads shouldn't be used in AWS lambdas, since "the background thread will be frozen once a Lambda event is processed and may not ever be unfrozen if more Lambda events are not received for some time."

The above quote was taken from the aws logging dotnet project

Fig 1.0 An extract from cloud watch logs showing how log statements from one Lambda execution appear with the log statements from a subsequent execution. Caused by background threads being frozen, as described above.

In my case, the real world problem that occurred in a production system was caused by this one line of code. Thankfully, the implications were not significant, but they could be subject to the system you're building.

Give it a try

If you have any desire to pull down the code, launch and then execute an AWS Lambda to see what happens, follow these steps.

To compile and package:

1. Clone/Fork the repository
2. Install the .net core 2.1 sdk (if not already installed)
3. Install the Serverless framework (if not already installed)
4. Install node and npm (if not already installed)
5. Run npm run build from {project_root}/src/LambdaSurprise.Services directory
6. Run npm run publish from {project_root}/src/LambdaSurprise.Services directory
7. Run npm run package from {project_root}/src/LambdaSurprise.Services directory

To deploy to AWS:

1. Install the AWS sdk for javascript
2. Run npm run deploy from {project_root}/src/LambdaSurprise.Services directory.

There are a multitude of reasons why deploying a Lambda to AWS can fail. If you do experience problems, which turn out to be a policy/permissions issue, this may help

Triggering the Lambda

There are two Lambda triggers. Firstly there's an sns trigger and secondly a cron trigger. Be sure that you disable the scheduled trigger in the AWS console once you've managed to replicate the scenario.

I'd recommend that you trigger the Lambda 10-20 times by running the publish-to-sns.js node script repeatedly, and then leave it for a while (I think I left it > 30 minutes before re-triggering). After a period > 30 minutes, repeat the re-triggering of the Lambda. Hopefully you'll see evidence from the CloudWatch logs of the above.

BEFORE running the script though, you'll need to edit the script, replacing the "REGIONHERE" placeholder and the "ACCOUNTNUMBERHERE" placeholder as per your account and region details.