How do acknowledgements work with the File Source and a HTTP Sink?

[mkwan-amzn]

Hi everyone, I have a setup as follows:

• File source w/ remove_after_secs set
• HTTP sink w/ acknowledgements enabled

I made some assumptions here about how E2E acknowledgements here would work. I assumed that the File source's remove_after_secs flag would only delete a file if it had reached EOF and all the data in the File source had been acknowledged by the sink. However, I'm seeing that even when the data has hit the HTTP sink,and the HTTP sink is receiving some retryable error and is still in the process of retrying it, the File source will have already deleted the data files it originally read from. This is concerning because this opens up the possibility of data loss, where retries get exhausted and Vector loses this data, and the original data files that were read have been deleted.

It's likely that I am misunderstanding how the acknowledgements should work in this scenario. My main questions are:

1. How do acknowledgements work for a File source?
2. Is there a way to ensure that files are only deleted via the remove_after_secs flag after the HTTP sink has received a non-error code response for all the data in the file?
   • Ideally, I would want to be able to guard against data loss even in the case of a restart of Vector

[mkwan-amzn]

One answer I've found since posting for Q2 is to use a disk buffer in the HTTP sink. Ideally still though, I would not want the original files to be removed before they are successfully sent to the final destination by the HTTP sink though.

[jszwedko]

You shouldn't need to use a disk buffer. If you have acknowledgements enabled on the file source, it'll wait until the data is delivered by the http sink. I'm not sure how that works with remove_after_secs though. My expectation is that it would wait until all data has been sent and n secs after EOF but I'm not sure if that is how it works in pracitce.

[mkwan-amzn]

Thanks for the response!

If you have acknowledgements enabled on the file source, it'll wait until the data is delivered by the http sink.

Can you clarify a bit here, what is the file source waiting to do here?

My expectation is that it would wait until all data has been sent and n secs after EOF but I'm not sure if that is how it works in practice.

I was able to simulate a scenario where the HTTP sink was receiving 503s from a server and it was attempting to retry events from a single file. Even though the HTTP sink was still retrying requests from that file, remove_after_secs had already deleted it after reaching EOF for n secs. Maybe this is worth a feature request or bug report.

[jszwedko]

Thanks for the response!

If you have acknowledgements enabled on the file source, it'll wait until the data is delivered by the http sink.

Can you clarify a bit here, what is the file source waiting to do here?

The file source will wait to update the checkpoint file until the data has successfully been written by all connected sinks.

My expectation is that it would wait until all data has been sent and n secs after EOF but I'm not sure if that is how it works in practice.

I was able to simulate a scenario where the HTTP sink was receiving 503s from a server and it was attempting to retry events from a single file. Even though the HTTP sink was still retrying requests from that file, remove_after_secs had already deleted it after reaching EOF for n secs. Maybe this is worth a feature request or bug report.

Gotcha, yeah, that sounds like a bug. I'm guessing remove_after_secs wasn't considered when implementing acknowledgements for the file source. A bug report would be welcome 🙂

[mkwan-amzn]

Thanks, makes sense!

I created a bug report here: #20816