Upgrade to AWS SDK V2

[jtuglu1]

Closes #16903.

Description

Upgrade to AWS SDK v2.40.0. Looks like dependencies hdfs-storage + ranger have transitive dependencies on V1. Those will need to be excluded or upgraded as well. Core modules/extensions required for runtime are not reliant on V1.

Migrated to strictly synchronous APIs (to maintain backwards-compatibility with V1). A follow-up PR should be able to make use of async APIs where suitable. S3TransferManager is the only case where an async client may be used.

New Tests

• This adds IngestionRollingUpgradeDockerTest which simulates ingestion of Druid segments using differing Druid versions, verifying that indexing processes can talk to each other and brokers/historicals can read the segments.

Hiccups

1. KinesisAggregatorV2 not available in Maven – Maven via JitPack: amazon-kinesis-aggregator-2.0.2 has v1 code, amazon-kinesis-deaggregator-2.0.2 unavailable awslabs/kinesis-aggregation#120. Doesn't look like anything uses aggregation=true, so added a warning message. Seems like [FLINK-32097][Connectors/Kinesis] Implement support for Kinesis deaggregation flink-connector-aws#188 also ran into similar issues, but implemented their own unmerged approach. Per the warning on the https://github.com/awslabs/kinesis-aggregation repo, looks like data loss can occur anyways when using this mode: https://github.com/awslabs/kinesis-aggregation/blob/master/potential_data_loss.md so don't think that's really a desired feature anyways.

Release note

Upgrade to AWS SDK v2.40.0

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Fly-Style]

Very good job @jtuglu1! I have no objections, PR looks good from my perspective!

[jtuglu1]

👍 I'd still like to add 1-2 ITs that fully cover the newer S3 changes, but overall think it's basically there.

[cryptoe]

Due to the criticality of the change, I think we should get 2 approvals for this PR.
cc @jtuglu1

[cryptoe]

Left a partial review.
Wanted to ask the following questions:
Did we test this patch on any cluster with the following things with we have s3 as the deep storage.

• Kinesis Ingestion.
• MSQ with Durable storage.
• MSQ select with durable storage as the destination.
• Segment getting nuked out.
• Task logs being viewable and getting purged out on S3.
• Segment ingestion.

Also have you tried this patch in any of the production clusters ?

[jtuglu1]

Hi @cryptoe. Yes, I'm testing this on our production clusters running Kafka + S3. We don't have production clusters using Kinesis. I've only been able to run tests locally. I wonder if you folks might have some Kinesis clusters to test the new logic out on?