Support raw payload preservation to avoid canonical JSON serialization

[alexluong]

Currently, Outpost serializes the data field using canonical JSON when delivering to destinations, which may not be 100% byte-to-byte identical to the original payload (whitespace, key ordering, etc.). For webhook consumers that rely on exact payload reproduction for signature verification or compliance requirements, we need a way to preserve the original raw payload without re-serialization.

[alexbouchardd]

Do we need to serialize the data? We could potentially just use the raw bytes/byte-string.

I'm assuming the issue right now is that the API merged system data, such as metadata and body, into a json payload, which means we need to parse the body. This is tricky as it means we'd need to either move all the system data (id, metadata, event_id, etc.) to the headers and only get the body as the publish operation body.

[alexluong]

Yes, I've been thinking about this flow a lot for performance optimization. One thing about the data payload that may need to be serialize/deserialize is during some template flow such as AWS Kinesis PartitionKeyTemplate. Not 100% sure what the solution is for this just yet.

I'm assuming the issue right now is that the API merged system data, such as metadata and body, into a json payload, which means we need to parse the body

I learned about Golang's json.RawMessage type which seems like the perfect solution for this problem.

[alexbouchardd]

I learned about Golang's json.RawMessage type which seems like the perfect solution for this problem.

That sounds great, we'd effectively just seralize part of the message

One thing about the data payload that may need to be serialize/deserialize is during some template flow such as AWS Kinesis PartitionKeyTemplate. Not 100% sure what the solution is for this just yet.

Some destination might require serialization and that's fine, we can do so at the delivery instead of ingestion for those destinations