Skip to content

Conversation

@juniorrhis1
Copy link

closes #3925

This pull request adds support for uploading files to AWS S3 from a specified local file path, in addition to the existing functionality of uploading message content. The changes include updates to both the documentation and the implementation to introduce a new configuration field, local_file_path, which allows users to specify a local file to upload per message.

Key changes:

Feature: Support for uploading from a local file path

  • Added a new configuration field local_file_path to the S3 output component, allowing users to specify the path of a local file to upload instead of message content. This field supports interpolation functions for dynamic file path resolution. [1] [2] [3] [4] [5] [6]
  • Updated the S3 writer logic to check if local_file_path is set for each message; if so, it reads the specified file and uploads its contents, otherwise it uploads the message content as before. [1] [2]
  • Improved documentation in aws_s3.adoc to describe the new local_file_path field, its type, default value, and usage examples.

Implementation details

  • Refactored the upload logic by introducing a helper method getUploadBody to encapsulate the logic for choosing between local file content and message content.
  • Added necessary imports for file handling in Go (os, io).

@CLAassistant
Copy link

CLAassistant commented Jan 20, 2026

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@mihaitodor
Copy link
Contributor

Hi @juniorrhis1 👋 Could you please let us know why you'd prefer this approach over something like the config below?

input:
  file:
    paths:
      - test.txt
    scanner:
      to_the_end: {}

output:
  aws_s3:
    ...

@juniorrhis1
Copy link
Author

Hi @mihaitodor, to avoid load the entire file in memory.

For small files or json files with one document per line (using lines scanner) this approach is prefered, but is not possible send large files without consumes a lot of memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: support streaming local files to AWS S3

3 participants