Performance issue: Slow retrieval times from Parameter Store in .NET 8 Lambda functions

[mcamacho97]

Describe the bug

Hello,

We have observed that Lambda functions using the .NET 8 runtime experience delays when fetching parameters from AWS Systems Manager Parameter Store. The delay ranges from 4 to 15 seconds, depending on the allocated memory size. This behavior is inconsistent and seems to be more pronounced during cold starts.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

We expect a consistent and faster response when fetching parameters from the Parameter Store, regardless of the memory configuration or cold/warm start scenarios.

Current Behavior

Below are the observed results during cold and warm starts for both multiple Parameter Stores and a single Parameter Store:

Cold Start (Full Path Parameter Store):

• 128MB RAM: 18.637 seconds
• 512MB RAM: 4.30 seconds
• 1GB RAM: 1.69 seconds
• 2GB RAM: 0.81 seconds

Warm Start (Full Path Parameter Store):

• 128MB RAM: 0.439 seconds
• 512MB RAM: 0.151 seconds
• 1GB RAM: 0.127 seconds
• 2GB RAM: 0.113 seconds

Cold Start (Specific Path Parameter Store):

• 128MB RAM: 18.338 seconds
• 512MB RAM: 3.85 seconds
• 1GB RAM: 1.66 seconds
• 2GB RAM: 0.94 seconds

Warm Start (Specific Path Parameter Store):

• 128MB RAM: 0.378 seconds
• 512MB RAM: 0.036 seconds
• 1GB RAM: 0.079 seconds
• 2GB RAM: 0.051 seconds

Reproduction Steps

1. To reproduce the issue with the slow retrieval times from AWS Systems Manager Parameter Store, follow these steps:
2. Set up an AWS Lambda function with the .NET 8 runtime.
3. Use the following code for the Lambda function:

using Amazon.Lambda.Core;
using Microsoft.Extensions.Configuration;
using System.Diagnostics;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace GetSSM.LambdaFunction
{
    public class Function
    {
        public string FunctionHandler(object input, ILambdaContext context)
        {
            var stopWatch = new Stopwatch();
            stopWatch.Start();

            var configuration = GetConfiguration();

            stopWatch.Stop();
            var elapsedTime = stopWatch.ElapsedMilliseconds;

            context.Logger.LogInformation($"Get Parameter Store: {elapsedTime} milliseconds");
            context.Logger.LogInformation($"Get Parameter Store: {elapsedTime / 1000.0} seconds");
            return "Hello World";
        }

        public static IConfiguration GetConfiguration()
        {
            var cb = new ConfigurationBuilder().AddSystemsManager(c =>
                {
                    c.Path = "/my/full/path/";
                });

            return cb.Build();
        }
    }
}

4. Deploy the function and configure an appropriate IAM role with permission to access Parameter Store.
5. Use AWS Systems Manager Parameter Store to create parameters under both:

• A specific path (e.g., /my/full/path/parameter).
• A full path structure with multiple parameters.

6. Test the Lambda function with different memory configurations (e.g., 128MB, 512MB, 1GB, 2GB) and log the results for both:

• Cold starts: Ensure the function is invoked after an idle period or by deploying a new version.
• Warm starts: Invoke the function repeatedly within a short time span.
• Collect the execution times for each scenario and compare the results.

Possible Solution

No response

Additional Information/Context

We’ve also attached a graph visualizing the above results to provide more clarity. This issue seems closely tied to the allocated memory size and the runtime's handling of cold starts.

Key Questions:

1. Is this behavior expected due to some inherent limitation or design of the library?
2. Are there any recommendations to mitigate these cold start delays when fetching parameters?
3. Considering that the function only performs a single GetParameter operation, is it appropriate to scale the memory so significantly to improve performance? Or is there an alternative approach to address this latency without increasing the allocated memory?

Thank you for your assistance! Let us know if further details or tests are required.

AWS .NET SDK and/or Package version used

Amazon.Extensions.Configuration.SystemsManager 6.2.2
Amazon.Lambda.Core 2.2.0
Amazon.Lambda.Serialization.SystemTextJson v

Targeted .NET Platform

.NET 8

Operating System and version

.NET8 Lambda Runtime

[ashishdhingra]

@mcamacho97 Good morning. Lambda cold starts are expected for .NET runtime. One of the ways to reduce cold start time is to use Native AOT compilation.

• Compile .NET Lambda function code to a native runtime format
• Building .NET Lambda functions with Native AOT compilation in AWS SAM for guidance on how to do so.

You may also try to leverage new SnapStart Lambda feature (however, please check pricing considerations).

• AWS Lambda SnapStart for Python and .NET functions is now generally available
• Improving startup performance with Lambda SnapStart

Increasing memory size could help a bit, but cold start would still be an issue due to container initialization upon first request.

CC @normj for any additional inputs.

Thanks,
Ashish

[github-actions]

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.