Add DruidContainer to completely phase out revised and standard ITs

[kfaraz]

Changes

• Add extension druid-testcontainers that can be later published as an individual module for Testcontainers
• Add DruidContainer - Testcontainer impl for running individual Druid services
• Add DruidContainerResource to allow use of DruidContainer in embedded cluster tests
• Make minor modifications to EmbeddedDruidServer and EmbeddedDruidCluster to support
  running both embedded servers and container-based services in the same cluster
• Add job to run the *DockerTests (2 right now, this number should remain small over time).

Note: Usage of DruidContainer will not be the norm but the exception to test out backward compatibility
and/or Docker related changes only. All other use cases should continue to use EmbeddedDruidServers.

Pending items

• Populate all the test methods using the builders in Migrate several indexing and compaction integration tests to embedded-tests #18207 once it is merged

Advantages

• Can be used to run embedded tests against the distribution Docker image as opposed to the current ITs
  which use a custom IT-only image
• Takes us 1 step closer to completely removing the old IT frameworks
• Leverages all the benefits of the embedded test framework while also testing out the Docker containers
• Can be easily used for backward compatibility tests

Next steps

• Once we have phased out the old ITs, all integration tests will be based on EmbeddedClusterTestBase
• Most of these tests will just use EmbeddedDruidServer (and required containers only, like KafkaResource)
• These tests will use Indexer as worker as they are faster and lighter than MiddleManagers
• There will be only a small number of tests that use DruidContainer, some of which will use MiddleManagers

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kfaraz]

The license check keeps failing. Need to fix up licenses.yml.

for reference:

Error: found 6 missing licenses. These licenses are reported, but missing in the registry
druid_module: druid-testcontainers, groupId: junit, artifactId: junit, version: 4.13.2, license: Eclipse Public License 1.0
druid_module: druid-testcontainers, groupId: org.testcontainers, artifactId: testcontainers, version: 1.21.3, license: None
druid_module: druid-testcontainers, groupId: com.github.docker-java, artifactId: docker-java-api, version: 3.3.4, license: Apache License version 2.0
druid_module: druid-testcontainers, groupId: com.github.docker-java, artifactId: docker-java-transport, version: 3.3.4, license: Apache License version 2.0
druid_module: druid-testcontainers, groupId: com.github.docker-java, artifactId: docker-java-transport-zerodep, version: 3.3.4, license: Apache License version 2.0
druid_module: druid-testcontainers, groupId: org.rnorth.duct-tape, artifactId: duct-tape, version: 1.0.8, license: None

[cecemei]

curious why does LatchableEmitter has waitForEvent(UnaryOperator<EventMatcher> condition), it seems like it can just do waitForEvent(EventMatcher matcher)?

[kfaraz]

I did have that initially but when I used the lamda, the tests became much more readable. So we kept this syntax.

[kfaraz]

The difference is very slight but I preferred the second syntax:

waitForEvent(EventMatcher.matchesA(...).matchesB(...))

and

waitForEvent(event -> event.matchesA().matchesB(...))

[cecemei]

nit: might be more readable if we use a builder, or mapper.readValue.

[kfaraz]

Yeah, that makes sense. There's another PR that needs to be merged. It has some builders which will be used here as well.

[kfaraz]

#18207

[cecemei]

where is DockerTest defined? is this some kind of smoke test?

[kfaraz]

There is an IngestionDockerTest that is being newly added in this PR. This command will run future tests also which have the same suffix. But yes, for the most part, it is meant to be a smoke test.

[kfaraz]

Closing in favor of #18302