Acknowledgement of S3 Source when using Lua transform

[ronkitay]

Hi,

I have the following setup:

AWS_S3 Source (SQS) --> Transform (Lua) --> AWS_S3 Sink

Even though I defined acknowledgements.enabled on the Sink and the source supports it - when there is an error in the Lua code, the message is deleted from the Queue and does not reach the DLQ.

Is this a bug? or am I mis-configuring?

This is a shortened version of my toml:

[sources.my_files]
type = "aws_s3"
region = "<THE REGION>"
strategy = "sqs"
compression = "gzip"
sqs.queue_url = "<THE QUEUE URL>"

[transforms.lua]
inputs = ["my_files"]
type = "lua"
version = "2"
hooks.process = """
  function (event, emit)
    if (event.thisMethodDoesNotExistAndShouldFail())
    then
      event.log.message = "Can't belive this worked!"
      emit(event) -- emit the transformed event
    end
  end
  """

[sinks.write_output]
type = "aws_s3"
inputs = [ "lua" ]
bucket = "<THE OUTPUT BUCKET>"
key_prefix = "date=%F/" 
compression = "gzip"
region = "<THE REGION>"
encoding.codec = "text"
acknowledgements.enabled = true

I'm using this vector release:

❯ vector --version
vector 0.22.0 (x86_64-unknown-linux-gnu 5e937e3 2022-06-01)

[jszwedko]

Hi @ronkitay !

Unfortunately acknowledgements are not supported whenever the lua transform is used as the acknowledgement information isn't passed through the Lua runtime. This is a definite gotcha that we are planning on documenting as part of the official release of end-to-end acknowledgements. It's also something we want to try to address at some point too, if we can, to let acknowledgements flow through the transform. As it stands currently, the event will be considered "acknowledged" as soon as it hits the lua transform.

You can see https://github.com/vectordotdev/vector/pull/11760/files for some draft documentation of the end-to-end acknowledgement feature.

Hopefully this helps!

[ronkitay]

Thank you @jszwedko for the quick answer, luckily I was able to find a way of using remap transforms instead, it is a bit more limiting - but necessary for my needs.