Reading JSON events from AWS CloudWatch Logs

[mikelorant]

When trying to read events from AWS CloudWatch Logs, a CloudWatch Logs subscription filter is required. This filter can send to Kinesis or Kinesis Firehose however the events are batched. Kinesis Firehose allows for the events to be written to objects in S3.

When looking at the objects in S3, the batched events are on a single line without a delimiter. This means that 3 objects of A, B and C will be converted to a single line of {A}{B}{C}. This is causing considerable difficulty in transforming this back to individual events.

Is there a method to handle JSON streams where multiple JSON objects are sent on the same line. Obviously if these were newline delimited this would be an easy problem to solve however AWS have made this difficult by not providing a customisable delimiter. Also, these are not a valid JSON array, so array conversion is not useful.

Any help would be greatly appreciated as this has become a blocker to using Vector on a project I am working on.

[jszwedko]

Hi @mikelorant !

Others have run into this before too. The recommended solution is to have whatever is sending the data to CloudWatch Logs include a newline at the end of each entry.

[mikelorant]

This is a problem because we were wanting to get the EKS audit logs. We have absolutely no control over the data that is added to CloudWatch Logs.

It would be great to have a streaming option in addition to the existing JSON delimiter options. Is this something that could be considered?

[jszwedko]

Hey! Yeah, we would definitely consider it. I think it could fit into our framework of "framers" that includes things like newline-delimited framing. Would you want to open a feature request?

[mikelorant]

I can create a feature request for this. Thanks for the providing the guidance of what I might look like.