feat: add support for setting S3 storage class on initial object write

[mattsre]

Motivation

The connector currently only writes to the STANDARD S3 storage class. If we want to reduce costs by using archival storage classes like Glacier, we must setup a lifecycle policy to transition objects on some schedule to our desired storage class. Transitioning objects has an associated cost, and we pay extra to keep objects in the STANDARD storage class until they can be transitioned to our desired storage class.

I'd like to reduce costs and simplify operations by having the Pulsar S3 connector simply write the object to the desired storage class when the object is initially written.

Modifications

I've added an s3StorageClass parameter to the Sink config which allows setting the storage class which will be used when making the PutObject API call to S3.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Added unit tests for validating the provided s3StorageClass parameter

Documentation

• doc

I've updated documentation to include the new s3StorageClass parameter

[david-streamlio]

Thanks for this improvement!

I see that you have added unit tests for the configuration validation, which is great, but would it be possible to add some integration tests to this, e.g. using the TestContainers framework, etc?

FWIW, I have attached an AI-generated integration test that you can use as a template for the test.

s3-integration.zip

[mattsre]

@david-streamlio I've added some integration tests here for verifying the storage class behavior. I haven't written much Java so there was a fair bit of AI help getting these together. Let me know if any changes are required or there's any test cases you'd like to see added!