`InvokeWithResponseStreamResponseEvent` leaks metadata payload into the event stream

[Licenser]

Describe the bug

InvokeWithResponseStreamResponseEvent leaks the initial metadata payload, and the trailing zeros into the event stream.

This seems wrong since as both the documentation and the function names suggest the metadata payload is distinct, well known, and part of the API not part of the events.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When reading the events from InvokeWithResponseStreamResponseEvent the metadata payload is not included but handled by InvokeWithResponseStreamResponseEvent internally.

Current Behavior

The first call of recv() returns the metadata payload json and the 8 trailing zeros

Reproduction Steps

0. set up a lambda with response streaming
1. call, configure and send invoke_with_response_stream() on a lambda client
2. call event_stream().recv() on the result
3. observe that this is the metadata payload not the firs event

Possible Solution

have InvokeWithResponseStreamResponseEvent parse the first message to extract the metadata

Additional Information/Context

No response

Version

aws-sdk-lambda v1.103.0

Environment details (OS name and version, etc.)

MacOs

Logs

No response

[ysaito1001]

Thank you for reporting this issue. This appears to be a regression introduced in aws-sdk-lambda 1.103.0.

The previous version (1.102.0) didn't handle initial-message in the response, which is expected behavior for non-rpc-bound protocols like restJson1.

However, starting with version 1.103.0, the code began handling initial responses, which introduced this regression.

[ysaito1001]

Just to double-check, do you see the same issue in the previous version of aws-sdk-lambda 1.102.0?

Also, does the message containing what you refer to as payload metadata include the header :event-type with the value initial-response?

[Licenser]

I can't speak to 102 as I only started to use streaming the 103 release. The data has no headers. The event received is of the type InvokeWithResponseStreamResponseEvent::PayloadChunk which contains a InvokeResponseStreamUpdate that contains only one field payload which is a Option<Blob> where Blob is a wrapper around a Vec<u8>.

No functionality is provided to disninguish the initial payload metadata message from the payload itself. It isn't always a single event either. I've seen the metadata and the trailing eight 0 bytes being in a single or multiple events.

[ysaito1001]

I've tried reproducing what you observed using aws-sdk-lambda 1.103.0, but haven't been able to:

example code

#[tokio::test]
async fn test_repro() {
    let shared_config = aws_config::from_env()
        .region(aws_sdk_lambda::config::Region::new("us-west-2"))
        .load()
        .await;
    let client = aws_sdk_lambda::Client::new(&shared_config);
    let resp = client
        .invoke_with_response_stream()
        .function_name("yield-chunks")
        .send()
        .await
        .unwrap();

    let mut stream = resp.event_stream;
    while let Ok(event) = stream.recv().await {
        match event.unwrap() {
            aws_sdk_lambda::types::InvokeWithResponseStreamResponseEvent::PayloadChunk(chunk) => {
                if let Some(payload) = chunk.payload {
                    print!("payload {}", String::from_utf8_lossy(payload.as_ref()));
                }
            }
            aws_sdk_lambda::types::InvokeWithResponseStreamResponseEvent::InvokeComplete(_) => {
                println!("\nStream complete");
                break;
            }
            _ => {}
        }
    }
}

lambda: yield-chunks

def lambda_handler(event, context):
    def generate():
        yield "chunk 1\n"
        yield "chunk 2\n"
    
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "text/plain"},
        "body": list(generate()),
    }

Got this response, with no metadata leaked:

payload {"statusCode": 200, "headers": {"Content-Type": "text/plain"}, "body": ["chunk 1\n", "chunk 2\n"]}

I first thought that a difference in code between aws-sdk-lambda 1.102.0 and 1.103.0 might have something to with this, but invoke_with_response_stream in both versions should behave the same at the end of the day.

Do you see this issue from any lambda function with response streaming, not just the one in question?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[Licenser]

This isn't something that happens every time sadly, we only saw a few percent of requests show this issue

[ysaito1001]

Since we haven't been able to reproduce this issue on our end, we'd appreciate your help with one of the following:

1. Provide a minimal reproducer that demonstrates the issue
2. Contribute a fix from your environment where the issue is observable

Either approach would help us move forward with resolving this.

[Licenser]

Heya, this lambda should create the mix of the prelude and the main response:

use http::{HeaderMap, StatusCode};
use lambda_runtime::{Error, LambdaEvent, MetadataPrelude, StreamResponse, run, service_fn};

#[tokio::main]
async fn main() -> Result<(), Error> {
    run(service_fn(|event: LambdaEvent<LambdaRequest>| async {
        let data = Ok::<_, Error>(b"Hello, world!".to_vec());
        Ok::<_, Error>(StreamResponse {
            metadata_prelude: MetadataPrelude {
                status_code: StatusCode::OK,
                headers: HeaderMap::new(),
                cookies: vec![],
            },
            stream: tokio_stream::once(data),
        })
    }))
    .await?;
    Ok(())
}```