feat(aws_s3): add local_file_path configuration for uploading files#3926
feat(aws_s3): add local_file_path configuration for uploading files#3926juniorrhis1 wants to merge 3 commits intoredpanda-data:mainfrom
Conversation
|
|
…connect into output-s3-local-file-stream
|
Hi @juniorrhis1 👋 Could you please let us know why you'd prefer this approach over something like the config below? input:
file:
paths:
- test.txt
scanner:
to_the_end: {}
output:
aws_s3:
... |
|
Hi @mihaitodor, to avoid load the entire file in memory. For small files or json files with one document per line (using lines scanner) this approach is prefered, but is not possible send large files without consumes a lot of memory. |
|
Hey @juniorrhis1, this is cool but I'm a bit worried about the security implications. In some environments this might be considered a vulnerability and would block deployment. I think a good solution might be to add this as a separate an alternative output, maybe called |
closes #3925
This pull request adds support for uploading files to AWS S3 from a specified local file path, in addition to the existing functionality of uploading message content. The changes include updates to both the documentation and the implementation to introduce a new configuration field,
local_file_path, which allows users to specify a local file to upload per message.Key changes:
Feature: Support for uploading from a local file path
local_file_pathto the S3 output component, allowing users to specify the path of a local file to upload instead of message content. This field supports interpolation functions for dynamic file path resolution. [1] [2] [3] [4] [5] [6]local_file_pathis set for each message; if so, it reads the specified file and uploads its contents, otherwise it uploads the message content as before. [1] [2]aws_s3.adocto describe the newlocal_file_pathfield, its type, default value, and usage examples.Implementation details
getUploadBodyto encapsulate the logic for choosing between local file content and message content.os,io).