Feature request: Improve Batch tools for SNS->SQS->Lambda triggers

[kylez-ithaka]

Use case

When a lambda is trigger by an SQS Queue, and that Queue itself is given messages from a subscription with an SNS topic, it results in multiple layers of serialization that need to be unwrapped to get to the real message body. The current implementation of Batch will only deserialize the SQS Message body, which leaves you with an SNS Message, that itself has a String body that needt o be deserialized again.

This results in code like

public HandlerSqs() {
        handler = new BatchMessageHandlerBuilder()
                .withSqsBatchHandler()
                .buildWithMessageHandler(this::processSNSMessage, SNSMessage.class);
    }

private String processSNSMessage(SNSMessage snsMessage) {
        MyInput myInput = extractDataFrom(snsMessage.getMessage()).as(MyInput.class);
        return processMessage(myInput);
    }

public SQSBatchResponse handleRequest(SQSEvent input, Context context) {
        return handler.processBatch(input, context);
    }

As well as the SNSMessage class I had to make myself due to case sensitivity that isn't accounted for in the Lambda Events classes

public class SNSMessage {
    @JsonProperty("Type")
    private String type;
    @JsonProperty("MessageId")
    private String messageId;
    @JsonProperty("TopicArn")
    private String topicArn;
    @JsonProperty("Subject")
    private String subject;
    @JsonProperty("Message")
    private String message;
...
}

Example of the input event from hooking up SNS->SQS->Lambda as json

{
  "Records": [
    {
      "messageId": "dummy-message-id",
      "receiptHandle": "dummy-receipt-handle",
      "eventSourceARN": "arn:aws:sqs:region:account-id:dummy-queue",
      "eventSource": "aws:sqs",
      "awsRegion": "dummy-region",
      "body": "{\"Type\": \"Notification\",\"MessageId\": \"dummy-sns-message-id\",\"TopicArn\": \"arn:aws:sns:region:account-id:dummy-topic\",\"Message\": \"{\\\"field\\\": \\\"dummy\\\", \\\"name\\\": \\\"dummy-name\\\"}\",\"Timestamp\": \"2000-01-01T00:00:00.000Z\",\"SignatureVersion\": \"1\",\"Signature\": \"dummy-signature\",\"SigningCertURL\": \"https://sns.region.amazonaws.com/dummy-cert.pem\",\"UnsubscribeURL\": \"https://sns.region.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:region:account-id:dummy-topic:dummy-subscription\"}",
      "md5OfBody": "dummy-md5",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "0000000000000",
        "SenderId": "dummy-sender-id",
        "ApproximateFirstReceiveTimestamp": "0000000000000"
      },
      "messageAttributes": {}
    }
  ]
}

Solution/User Experience

It would be great if this was a 1 step process like the rest, maybe like:

public HandlerSqs() {
    handler = new BatchMessageHandlerBuilder()
                .withSnsToSqsBatchHandler()
                .buildWithMessageHandler(this::processMessage, MyInput.class);
}

public SQSBatchResponse handleRequest(SQSEvent input, Context context) {
        return handler.processBatch(input, context);
    }

Alternative solutions

Alternatively, if the Events library included an `SNSMessage` or such, that actually could deserialize it directly, instead of maintaining a custom class for it.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Java) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, TypeScript, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[leandrodamascena]

Hi @kylez-ithaka, thanks for opening this issue! For a while now, I've been thinking about how we can help customers to work with nested events in some of the utilities we have in Powertools, and Batch is one of them. I like the idea of ​​helping customers decode nested events because I know it can improve the developer experience, but I don't think we should do anything too specific for SNS -> SQS. I know SNS -> SQS is a popular use case, but AWS also allows for many combinations, such as:

SQS → Lambda
SNS → SQS → Lambda
EventBridge → SQS → Lambda
S3 → SQS → Lambda
API Gateway → SQS → Lambda

Or even more complex like:

API Gateway → SNS → SQS → Lambda
S3 → SNS → SQS → Lambda
EventBridge → SNS → SQS → Lambda
CloudWatch → SNS → SQS → Lambda

For a single-level encoding, it could be handled with hardcoded methods. But as more services, combinations, and complexity come into play, this approach quickly becomes harder to manage. We should think carefully about how to make it flexible enough to support all possible scenarios. We’re open to suggestions or even seeing a small proof of concept to explore how this might work in practice.

To be very transparent, even if we align on a solution in the next iterations, we won’t have the bandwidth to deliver this feature this year, as we’re focused on other launches that will take some time. Still, we truly believe this kind of solution could benefit many customers, and we’re very open to ideas on how to create this.

I'm adding this to the backlog while we keep discussing it.

Thanks

[kylez-ithaka]

@leandrodamascena thanks for the response, the "workaround" isn't much effort, and it seems to really come down to the serialization extractDataFrom tool, which didn't have an obvious path to deal with nested types like those. In some sense it might go all the way to the events library aws-lambda-java-events to be able to specify something like SQSEvent<SNS> or such (which would possibly be a major enhancement to that)

[phipag]

the "workaround" isn't much effort, and it seems to really come down to the serialization extractDataFrom tool, which didn't have an obvious path to deal with nested types like those.

This gives me an idea. We already support JMESPath functions in different utilities such as Validation or Idempotency (https://docs.aws.amazon.com/powertools/java/latest/utilities/serialization/#jmespath-functions).

If we added a feature in EventDeserializer.java to support JMESPath expressions to define the root path for an object we can support something like this (because the Batch utility uses powertools-serialization extractDataFrom under the hood).

public HandlerSqs() {
        handler = new BatchMessageHandlerBuilder()
                .withSqsBatchHandler()
                // Point via JMESPath to the root of the wrapped SNS message. EventDeserializer will then unwrap the SNS message
                // because it is a supported type
                .buildWithMessageHandler(this::processMessage, MyInput.class, "Records[0].body | from_json(@).Message");
    }

private String processMessage(MyInput myInput) {
        // ...
    }

public SQSBatchResponse handleRequest(SQSEvent input, Context context) {
        return handler.processBatch(input, context);
    }

[kylez-ithaka]

From there it would seem like a bunch of predefined paths could be setup (like SNS -> SQS, EventBridge -> SNS -> SQS, etc) and then the line becomes something like

                .buildWithMessageHandler(this::processMessage, MyInput.class, PredefinedPaths.SqsToSns);

[phipag]

We have these known paths for correlation IDs for example: https://docs.aws.amazon.com/powertools/java/latest/core/logging/#built-in-correlation-id-expressions