S3 Plugin: Implement Server-Side Encryption option #8646
Conversation
msolters
requested review from
PettitWesley,
edsiper,
fujimotos,
koleini and
leonardo-albertovich
as code owners
- Introduces a server_side_encryption parameter for the S3 [OUTPUT] plugin. Possible values are AES256 and aws:kms, as per AWS API documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax - If either value is provided, the x-amz-server-side-encryption header will be included with S3 requests with the corresponding value set. Signed-off-by: Mark Solters <[email protected]>
msolters
force-pushed
the
s3-server-side-encryption
branch
from
62b337e to
34a63d5
Compare
| * AWS plugins that support sse will have these options. | ||
| * Referenced function should return -1 on error and 0 on success. | ||
| */ | ||
| static const struct sse_option sse_options[] = { |
There was a problem hiding this comment.
@edsiper is this a recommended style/technique for FLB C code? We have similar bits of code elsewhere in the project that have a set of values but that don't validate them in this way with a object list...
As noted per feedback this code is unrelated to compression Signed-off-by: Mark Solters <[email protected]>
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
We would love to have this - what's missing for merging? |
As far as I know, nothing. I think the MR has just been forgotten? We've been continuing to build a fork of Fluent Bit in order to incorporate this. Would be great for it to be merged upstream. |
|
I’m very interested in this PR as well. Would it be possible to support the other options, particularly SSE-C? (Fluentd plugin reference: https://github.com/fluent/fluent-plugin-s3/blob/master/docs/output.md#sse_customer_key) |
|
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days. |
|
@patrick-stephens could you take a look at this why it's failing go run checks? |
|
Looks like they were never run, usually down to an issue in ci either the workflows or an outage. If it's rebased or otherwise something triggers a push then it should run them. |
6 participants
S3 [OUTPUT]plugin. Possible values areAES256andaws:kms, as per AWS API documentation: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntaxx-amz-server-side-encryptionheader will be included with S3 requests with the corresponding value set.Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Example config:
This shows the S3 output config loading correctly. Obviously the bucket isn't real in this config but the config load throws no error:
When the bucket is real (and IAM permissions exist) this does correctly upload to SSE-enabled buckets.
A negative example, where
server_side_encryption: wumbo, fails to load as expected:Starting without
server_side_encryptionbeing defined at all also doesn't crash:Valgrind heap summary with
server_side_encryption: AES256:If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Documentation PR here
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.