fix(aws_s3 source): add retry delay in sqs::Ingestor

[medzin]

Summary

This commit adds logic to wait 500ms between subsequent calls of run_once if it returns an error. This is required to prevent spamming the SQS API if it fails to receive messages, e.g. due to invalid IAM permissions or an SQS outage.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• The CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
    • ./scripts/check_changelog_fragments.sh
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

References

• Closes: No retry backoff for sqs in aws_s3 source #22947

[bits-bot]

All committers have signed the CLA.

[pront]

Thanks @medzin, I understand the problem you are trying to fix here. Left a question.

[pront]

I wonder how this constant was chosen. Would an exponential backoff strategy work better here or it's an overkill?

[medzin]

I was treating this code as a hotfix rather than a long-term solution, so I chose a simple constant delay to quickly address the issue. The current 0.5-second delay doesn't introduce significant latency but greatly reduces the number of retries triggered by the source, which helps stabilize things short-term. That said, I agree a constant delay may not be ideal across all use cases - different values might work better for different users. An exponential backoff could offer a more flexible and resilient solution without needing to make the delay configurable. Happy to explore that in a follow-up PR.

[pront]

I can see how this improves things in your situation but we generally don't want to merge hotfixes. If you have urgent need for this commit, you can always maintain a fork with this commit and use a custom Vector build.

Going back to this PR, what do you think about introducing a new opt-in config parameter here? We can probably reuse ExponentialBackoff from src/sinks/util/retries.rs.

Example:

• 1st retry: 1 second
• 2nd retry: 2 seconds
• 3rd retry: 4 seconds
• 4th retry: 8 seconds
• 5th retry: 16 seconds
• 6th retry: 30 seconds (capped)

[medzin]

Sure. I will check it.

[pront]

Couldn't commit to this branch due to perms, moved to: #23996