(s3 source) Select only certain objects to process from S3-SQS events?

[sbalmos]

Asking just to confirm / validate before I go off and start writing another patch. :) In my ongoing work to convert pipelines, I have AWS Cloudtrail logs that are being posted into a shared logging bucket. The S3 source of the current product (which rhymes with tile-neat) allows one to specify a regex to match against the object name in the SQS event, and to ignore (e.g. leave in SQS) non-matching events. I don't see anything particularly like this right now.

A downright horrid workaround would be to ingest all S3-SQS events with the current S3 source and then immediately repost the non-matching events through an SQS sink. But I have a feeling that would be an infinite loop?

[jszwedko]

Hi @sbalmos !

This sounds like a good feature request. Would you mind opening one? https://github.com/vectordotdev/vector/issues/new?assignees=&labels=type%3A+feature&projects=&template=feature.yml

A subpar workaround be to not have Vector directly consume that queue, but set up another queue and create a lambda to copy events over to it, first checking if the object name matches, and have Vector consume that one.