Vector using S3 Source: Failed to parse SQS queue #22430
Answered
by
pront
generate-me12
asked this question in
Q&A
-
|
Hi, I'm deploying a vector on an ec2 instance. I have a standard SQS Queue setup to receive S3 bucket notifications. I confirmed the queue can receive messages whenever an item is created in S3. When running vector using an s3 source, I'm receiving the following error message: This is what an incoming message looks like: This is my vector configuration: I'm not too familiar with SQS but is the issue with my message or is there another underlying issue? |
Beta Was this translation helpful? Give feedback.
Answered by
pront
Feb 12, 2025
Replies: 1 comment 6 replies
-
|
Hi @generate-me12, Vector expects the standard S3 Event Notification format. So it should be something like: {
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "2025-02-12T02:12:06.936Z",
"eventName": "ObjectCreated:Put",
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "xyz",
"bucket": {
"name": "my-s3-bucket"
},
"object": {
"key": "offloads/data/2025/02/12/abcoffload.tgz",
"size": 268293
}
}
}
]
}Are these events perhaps coming for another Amazon source e.g. EventBridge? If this is the case, the try to reconfigure your S3 event notifications to send direct S3 event notifications to SQS instead. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
I am not an AWS expert so I cannot advise if this is the optimal way. But, yes, what you describe sounds feasible! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have another question related to this. If I created the lambda function and no longer get the parsing error, I see vector log says retrieve item from S3 bucket. So I setup a sink for this s3 source but looking at my file, I see this: This is my sink configuration: Is this a problem with retrieving the item from the s3 bucket or an encoding problem? |
Beta Was this translation helpful? Give feedback.
-
|
It looks like some messages are encoded differently. In your example, most of them are key/value pairs |
Beta Was this translation helpful? Give feedback.
-
|
I see, that seems to be the case. In the s3 bucket, there are tar files and those tar files contains more tar files inside of them. Is there a way for vector to decompress the tar files inside the tar file? |
Beta Was this translation helpful? Give feedback.
-
|
Can you pre-process there and ensure they have a common format? Otherwise, VRL process functions to decompress payloads. Example: https://vector.dev/docs/reference/vrl/functions/#decode_zstd |
Beta Was this translation helpful? Give feedback.
Answer selected by
generate-me12
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @generate-me12, Vector expects the standard S3 Event Notification format.
So it should be something like:
{ "Records": [ { "eventVersion": "2.1", "eventSource": "aws:s3", "awsRegion": "us-west-2", "eventTime": "2025-02-12T02:12:06.936Z", "eventName": "ObjectCreated:Put", "s3": { "s3SchemaVersion": "1.0", "configurationId": "xyz", "bucket": { "name": "my-s3-bucket" }, "object": { "key": "offloads/data/2025/02/12/abcoffload.tgz", "size": 268293 } } } ] }Are these events perhaps coming for another Amazon source e.g. EventBridge? If this is the case, the try …