path style access

[simonelbaz]

Is your feature request related to a problem? Please describe.

It would be nice to have path style access support to S3 storage. This feature is deprecated but some environments still use it and do not support virtual host style.

By the way, Opensearch supports it (https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#amazon-s3):

s3.client.default.path_style_access: false # whether to use the deprecated path-style bucket URLs.

Describe the solution you'd like

I would like to submit a PR if it is OK for the data-prepper maintainers. This PR would allow the user to enable path style access to S3 storage.

Describe alternatives you've considered (Optional)
The environment variable does not seem to be taken into account by the software.amazon.awssdk library:

AWS_S3_FORCE_PATH_STYLE: "true"

I have also tried to use configand credentials files. The addressing_style also is not used by the library:

[default]
region = us-east-1
s3 =
  endpoint_url = https://{{ s3_endpoint }}
  signature_version = s3v4
  addressing_style = path

From this link https://docs.aws.amazon.com/sdkref/latest/guide/settings-reference.html, it seems not possible to activate path style from JVM settings or environment variables.

Additional context
data-prepper version 2.13.0, docker image

[dlvenable]

@simonelbaz ,

First, I want to be sure I understand the ask. I understand that you would like Data Prepper to access S3 using paths such as https://s3.<region>.amazonaws.com/<bucket-name>/<key>. Is that correct?

What is the scope of this request? For all access to S3 or for the S3 sink or source? For context, Data Prepper has several configurations that can read from an S3 bucket to get data (e.g. read a certificate from S3).

Finally, can you help clarify your goal in using this approach? This is a deprecated feature because it requires that you make your buckets public which is not recommended.

[dlvenable]

@simonelbaz , Also, can you give a sense of what the PR might involve? If this is changing one setting globally that is probably ok. But, if we have to change all the places we access S3, that may be a lot for a deprecated feature.

[simonelbaz]

@dlvenable

@simonelbaz ,

First, I want to be sure I understand the ask. I understand that you would like Data Prepper to access S3 using paths such as https://s3.<region>.amazonaws.com/<bucket-name>/<key>. Is that correct?

Yes, it is.

What is the scope of this request? For all access to S3 or for the S3 sink or source? For context, Data Prepper has several configurations that can read from an S3 bucket to get data (e.g. read a certificate from S3).

You pointed the issue I am facing. Data Prepper gets the following exception:

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Certificate for <bucket-name.endpoint_url> doesn't match any of the subject alternative names: [*.endpoint_url_subpart]
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)

Finally, can you help clarify your goal in using this approach? This is a deprecated feature because it requires that you make your buckets public which is not recommended.

My goal is to resolve the wildcard certificate issue. By using path style, the exact endpoint_url (without the bucket name) should match the wildcard certificate.
I hope I am not wrong.

[simonelbaz]

@simonelbaz , Also, can you give a sense of what the PR might involve? If this is changing one setting globally that is probably ok. But, if we have to change all the places we access S3, that may be a lot for a deprecated feature.

@dlvenable : Here is the work in progress for the path style access PR: https://github.com/simonelbaz/data-prepper/tree/dlqpathstyle

[dlvenable]

@simonelbaz ,

How many places do you think you need this change applied to? I think the approach you have generally looks good to me as it is simple code. We would also need unit tests for these.