Differences between these images and AWS Lambda itself

[davidjb]

In trying to deploy a project to the provided.al2 runtime on AWS via SAM, I've encountered a number of key differences between the published container base image and the AWS Lambda environment itself. So far, I've found differences in libraries and shared data - e.g. libmagic and its databases installed in the base images, but not in AWS, and the same for glib-2.0. I'm sure there are other differences too, it's just that I've not encountered them yet; it's a trial-and-error process to see how something built successfully in a published image fails in AWS Lambda.

I'm aware of the Maintenance policy in that (emphasis mine):

These images are similar to the AWS Lambda execution environment on the cloud to allow customers to easily packaging functions to the container image. However, we may choose to optimize the container images by changing the components or dependencies included.

Is there a better way than trial-and-error for ascertaining differences between published Lambda base images and AWS Lambda itself? Likewise, is there a rationale as to why certain libraries are present in one environment but not another?

So far, I've taken running an interactive shell to analyse AWS Lambda but needing to do this for every dependency seems to negate the purpose of having published base images, since you can't be be sure that anything present in the base image will be on Lambda. I appreciate there may be an aim "optimize" these published container images, but knowing what this entails is key to their usability.

[Khaled-El-Mansoury-EG]

+1 Setting the environment variable LD_DEBUG to all on the Amazon Linux Provided AL2, one can see that a GNU libc (GLIBC) version of 2.34 is being used, whilst the lambda/provided:al2 docker image uses the version: 2.26.

I am looking to package an AOT .NET 7 Lambda function which works fine on the AWS Lambda environment (provided.al2), but not on the same equivalent docker image as it needs version 2.29 at least to run.

A workaround such as the ability to build our own base images would be much appreciated.

[justinmk3]

A workaround such as the ability to build our own base images would be much appreciated.

That is possible with the "Image" packagetype (as opposed to the old "Zip" packagetype, which is sourced from this repo). Both SAM CLI and AWS Toolkit support local debugging with "Image" packagetype.

[Khaled-El-Mansoury-EG]

A workaround such as the ability to build our own base images would be much appreciated.

That is possible with the "Image" packagetype (as opposed to the old "Zip" packagetype, which is sourced from this repo). Both SAM CLI and AWS Toolkit support local debugging with "Image" packagetype.

Thanks @justinmk3 that's the approach I am using. My Dockerfile looks like this:

FROM public.ecr.aws/lambda/provided:latest

# Copy the published Lambda function files to the task directory
COPY "bin/Release/lambda-publish" ${LAMBDA_TASK_ROOT}

# Set the Lambda function entrypoint as the bootstrap
ENTRYPOINT [ "/var/task/bootstrap" ]

Whilst this builds just fine, the issue is that lambda/provided:latest (or lambda/provided:al2) uses an old GNU libc version, so I ultimately get the following error when trying to run the function: /var/task/bootstrap: /lib64/libm.so.6: version GLIBC_2.29 not found (required by /var/task/bootstrap).
The AWS Lambda environment that you could get using the ZIP file approach uses version 2.34.

I know the AWS documentation mentions that it's possible to use a non-AWS base image, but there's very little documentation on that. I will try and go down that line though. Thanks!

[Khaled-El-Mansoury-EG]

For anyone else's benefit, this is what I've managed to put together as an up-to-date alternative to the amazon/aws-lambda-provided:al2 image which works fine for building native AOT-compiled .NET 7 lambda functions:

# https://github.com/aws/aws-lambda-base-images/tree/provided.al2
# Use an Amazon Linux 2023 base as amazon/aws-lambda-provided:al2 uses an older Amazon Linux with an outdated GNU libc
FROM amazonlinux:2023

# Download & Install the AWS Lambda Runtime Interface Emulator
RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie \
	&& chmod +x /usr/local/bin/aws-lambda-rie

# Create the lambda-entrypoint.sh normally found in amazon/aws-lambda-provided:al2
COPY <<-"EOF" /lambda-entrypoint.sh
#!/bin/sh
# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.

if [ $# -ne 1 ]; then
	echo "entrypoint requires the handler name to be the first argument" 1>&2
	exit 142
fi
export _HANDLER="$1"

RUNTIME_ENTRYPOINT=/var/runtime/bootstrap
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
	exec /usr/local/bin/aws-lambda-rie $RUNTIME_ENTRYPOINT
else
	exec $RUNTIME_ENTRYPOINT
fi
EOF

# Grant the execute permission on /lambda-entrypoint.sh
RUN chmod +x /lambda-entrypoint.sh

# Install libicu as required by .NET
RUN dnf install libicu -y

# Set the appropriate environment variables, working directory and entrypoint normally found in amazon/aws-lambda-provided:al2
ENV LANG=en_US.UTF-8
ENV TZ=:/etc/localtime
ENV PATH=/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
ENV LD_LIBRARY_PATH=/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib:/usr/local/lib
ENV LAMBDA_TASK_ROOT=/var/task
ENV LAMBDA_RUNTIME_DIR=/var/runtime
WORKDIR /var/task
ENTRYPOINT ["/lambda-entrypoint.sh"]

# Copy the published Lambda function files to the runtime directory
COPY "bin/Release/lambda-publish" ${LAMBDA_RUNTIME_DIR}

[davidjb]

I've had a look through the referenced PR aws/aws-toolkit-vscode#7678 but the only link to this issue is that remote debugging support allows someone to perhaps interrogate differences in SAM images and AWS Lambda slightly more easily (and only when using that plugin in VS Code).

It doesn't appear to explain or document what the differences are and/or how they occur, which is what my original issue is about.

@anzheyazzz Could you please reopen this issue?