Logs to s3

[shamj]

I am using awss3 sink to send the logs using vector. However I am trying to figure how to read the logs from s3 for a specific time period, is there a way I can do it. I tried ClickHouse , but it says invalid format. Looks like it didn't like the format the way vector writes to s3.
Please help.

[pront]

how to read the logs from s3 for a specific time period, is there a way I can do it.

The aws_s3 source polls SQS for bucket notifications, which means new objects. For older data you would have to implement a custom solution.

Looks like it didn't like the format the way vector writes to s3.

This depends on your configuration. Especially the source decoder and sink encoder.

[shamj]

Thank you. And is there a way to read the data that is written to s3 by vector and send it to grafana. We tested some other tools to read the data, but they didn't like the format and the way vector writes the logs to s3.

[pront]

And is there a way to read the data that is written to s3 by vector and send it to grafana.

Yes, you can use an aws_s3 source and a Loki sink. Note the AWS S3 source requires a SQS queue configured to receive S3 bucket notifications and this applies to new events as they come in while your pipeline is ready to read.

but they didn't like the format and the way vector writes the logs to s3

In general, the remap transform is powerful enough to remediate the data as they go through a Vector pipeline to satisfy the expectations of downstream systems (assuming the out-of-the-box codecs do not work). You will need to understand the event format when it comes out of the source and the format the downstream expects.

[shamj]

I was checking aws s3 source, does it have the option to read the files based on the object and folder structure in s3?
Lets say we have a folder in s3 as vector-logs that has subfolders as date/ hour/min . Can I read the logs from the folder saying read the logs from vector-logs/date1/hr1/min1? there are other folders also under vector-logs. and the logs are already written in the past. Based on your previous comment , it looks like it's not possible. It needs custom solution. Is that right?