Skip to content

Dbz 9419 add connection validator pulsar#295

Open
pxcamus wants to merge 16 commits intodebezium:mainfrom
pxcamus:DBZ-9419-add-connection-validator-pulsar
Open

Dbz 9419 add connection validator pulsar#295
pxcamus wants to merge 16 commits intodebezium:mainfrom
pxcamus:DBZ-9419-add-connection-validator-pulsar

Conversation

@pxcamus
Copy link
Copy Markdown

@pxcamus pxcamus commented Mar 11, 2026

WIP PR to add Apache Pulsar connection validation.

image

@github-actions
Copy link
Copy Markdown

Hi @pxcamus, thanks for your contribution. Please prefix the commit message(s) with the debezium/dbz#xxx GitHub issue key.

@mfvitale
Copy link
Copy Markdown
Member

Hi @pxcamus thanks for the PR. Could you please prefix the commit messages with debezium/dbz#1083? Thanks.

@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch from 3d17fb9 to 1ef50b0 Compare March 12, 2026 09:13
},
{
"type": "KAFKA",
"type": "APACHE_PULSAR",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you changed the KAFKA?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, I will put it back!

Copy link
Copy Markdown
Member

@mfvitale mfvitale left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pxcamus The general approach looks good. Let's continue with the other auth and then I'll do another review pass.

Comment on lines +25 to +30
// private PulsarConnectionValidator validator;
//
// @BeforeEach
// void setUp() {
// validator = new PulsarConnectionValidator(DEFAULT_30_SECONDS_TIMEOUT);
// }
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this isn't needed, let's remove it.

Comment on lines +45 to +47
// ConnectionValidationResult result = validator.validate(connection);

// assertTrue(result.valid(), "Connection validation should succeed");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, remove if its no longer needed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I finally got around to fix the tests. I will create a PulsarConnectionValidatorTest.java very soon as well, and hoping later today to add more tests for the different auth handlers.

@mfvitale
Copy link
Copy Markdown
Member

@pxcamus Could you please rebase with latest main?

@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch from 528da49 to 53812e8 Compare March 17, 2026 14:46
@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 17, 2026

@pxcamus Could you please rebase with latest main?

Done!

@pxcamus pxcamus requested a review from Naros March 17, 2026 15:30
@github-actions
Copy link
Copy Markdown

Hi @pxcamus, thanks for your contribution. Please prefix the commit message(s) with the debezium/dbz#xxx GitHub issue key.

@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch from 11667a4 to f58ffe3 Compare March 18, 2026 11:02
@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 18, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@mfvitale
Copy link
Copy Markdown
Member

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 19, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

@mfvitale
Copy link
Copy Markdown
Member

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 25, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@mfvitale Should the OAuth2 connection validation include the retrieval of the token? I.e. do I request the client credentials and request a token first? My concern is that the OAuth2 provider could be un-reachable from the platform conductor, right?

The other option is to just collect the token through the stage UI and validate the connection.

I have a working OAuth2 provider implementation for the integration tests, so can use the actual flow.

Copy link
Copy Markdown
Member

@Naros Naros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my PoV, once the following is addressed then LGTM.

}

public PulsarAuthHandler getAuthHandler(String authType) {
if (authType == null || authType.trim().isEmpty()) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (authType == null || authType.trim().isEmpty()) {
if (Strings.isNullorEmpty(authType)) {

@mfvitale
Copy link
Copy Markdown
Member

mfvitale commented Mar 26, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@mfvitale Should the OAuth2 connection validation include the retrieval of the token? I.e. do I request the client credentials and request a token first? My concern is that the OAuth2 provider could be un-reachable from the platform conductor, right?

The other option is to just collect the token through the stage UI and validate the connection.

I have a working OAuth2 provider implementation for the integration tests, so can use the actual flow.

This is a good point. We should not only think about the validation phase but also what happens when a connection is then validated and used in the DS instance deployed by the platform.

@pxcamus So for now since there is no support for loading credentials file.
In https://pulsar.apache.org/docs/next/security-oauth2/ I see that the client requires to use the keyFile so I will not support it for now in the validation.

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 26, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@mfvitale Should the OAuth2 connection validation include the retrieval of the token? I.e. do I request the client credentials and request a token first? My concern is that the OAuth2 provider could be un-reachable from the platform conductor, right?
The other option is to just collect the token through the stage UI and validate the connection.
I have a working OAuth2 provider implementation for the integration tests, so can use the actual flow.

This is a good point. We should not only think about the validation phase but also what happens when a connection is then validated and used in the DS instance deployed by the platform.

@pxcamus So for now since there is no support for loading credentials file. In https://pulsar.apache.org/docs/next/security-oauth2/ I see that the client requires to use the keyFile so I will not support it for now in the validation.

@mfvitale unless I missed something, it can be loaded as a base64 encoded string, right?

image

But to clarify, do I then wrap up the PR by passing the token for the validation?

@mfvitale
Copy link
Copy Markdown
Member

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@mfvitale Should the OAuth2 connection validation include the retrieval of the token? I.e. do I request the client credentials and request a token first? My concern is that the OAuth2 provider could be un-reachable from the platform conductor, right?
The other option is to just collect the token through the stage UI and validate the connection.
I have a working OAuth2 provider implementation for the integration tests, so can use the actual flow.

This is a good point. We should not only think about the validation phase but also what happens when a connection is then validated and used in the DS instance deployed by the platform.
@pxcamus So for now since there is no support for loading credentials file. In https://pulsar.apache.org/docs/next/security-oauth2/ I see that the client requires to use the keyFile so I will not support it for now in the validation.

@mfvitale unless I missed something, it can be loaded as a base64 encoded string, right?

image But to clarify, do I then wrap up the PR by passing the token for the validation?

Where you get this infos?

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 26, 2026

@mfvitale If you could please take a look at the approach I used for testing JWT auth. I am going to add IT tests with expired token, etc... then switch to OAuth2. Thanks!

@pxcamus It LGTM until now. 🚀

One small question please @mfvitale : for integration tests, should I spinnup an actual OAuth2 provider, such as Keycloak or Hydra? Or mock one to return hardcoded tokens? If using an actual provider, any preference? Thanks!

Honestly for now I'll go with a mocked one.

@mfvitale Should the OAuth2 connection validation include the retrieval of the token? I.e. do I request the client credentials and request a token first? My concern is that the OAuth2 provider could be un-reachable from the platform conductor, right?
The other option is to just collect the token through the stage UI and validate the connection.
I have a working OAuth2 provider implementation for the integration tests, so can use the actual flow.

This is a good point. We should not only think about the validation phase but also what happens when a connection is then validated and used in the DS instance deployed by the platform.
@pxcamus So for now since there is no support for loading credentials file. In https://pulsar.apache.org/docs/next/security-oauth2/ I see that the client requires to use the keyFile so I will not support it for now in the validation.

@mfvitale unless I missed something, it can be loaded as a base64 encoded string, right?
image
But to clarify, do I then wrap up the PR by passing the token for the validation?

Where you get this infos?

https://pulsar.apache.org/docs/4.1.x/security-oauth2/

@mfvitale
Copy link
Copy Markdown
Member

@mfvitale unless I missed something, it can be loaded as a base64 encoded string, right?
image
But to clarify, do I then wrap up the PR by passing the token for the validation?

Where you get this infos?

https://pulsar.apache.org/docs/4.1.x/security-oauth2/

Well then it should work but is not the best from the security point, since this base64 string will be then passed in the CRD definition.

Let's add the support for the validation for now.

@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch from 266eff9 to b75c041 Compare March 30, 2026 15:08
@pxcamus pxcamus requested a review from Naros March 30, 2026 15:21
@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Mar 30, 2026

@mfvitale I am done AFAIK. One question (hopefully not too stupid): when rebasing the latest main on my branch, I saw the following entry added to POM.xml:

<dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-common</artifactId> <version>${version.debezium}</version> </dependency>

But I cannot resolve this dependency. How do I build it locally? Thanks!

@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch 2 times, most recently from b00b1ea to 0db866b Compare March 31, 2026 14:07
@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 1, 2026

@pxcamus Could you please check the failure?

@mfvitale I am struggling so much to run my branch to update the ITs 😅 When trying to build the project:

Downloading from confluent: https://packages.confluent.io/maven/io/debezium/debezium-quarkus-outbox/3.6.0-SNAPSHOT/debezium-quarkus-outbox-3.6.0-SNAPSHOT.jar Downloading from confluent: https://packages.confluent.io/maven/io/debezium/debezium-operator-api/3.6.0-SNAPSHOT/debezium-operator-api-3.6.0-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 22.820 s [INFO] Finished at: 2026-04-01T19:02:04+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project debezium-platform-conductor: Could not resolve dependencies for project io.debezium:debezium-platform-conductor:jar:3.6.0-SNAPSHOT [ERROR] dependency: io.debezium:debezium-quarkus-outbox:jar:3.6.0-SNAPSHOT (compile) [ERROR] Could not find artifact io.debezium:debezium-quarkus-outbox:jar:3.6.0-SNAPSHOT in confluent (https://packages.confluent.io/maven/) [ERROR] dependency: io.debezium:debezium-operator-api:jar:3.6.0-SNAPSHOT (compile) [ERROR] Could not find artifact io.debezium:debezium-operator-api:jar:3.6.0-SNAPSHOT in confluent (https://packages.confluent.io/maven/)

But then when trying to build debezium-quarkus-outbox, I get an issue with something else (debezium-connector-informix). I suspect I am doing something stupid, but can't figure out what exactly...

@Naros
Copy link
Copy Markdown
Member

Naros commented Apr 2, 2026

There are now a few cross-repo dependencies, and it's probably something we should try to document clearly, particularly for first-time contributors. The debezium-quarkus repo now requires debezium-connector-informix to be built up-front. So if you clone that repository and build it, then that should solve your issue. Just be mindful that the Informix repo likely will require the main debezium/debezium repo to be built as well.

So build

  1. debezium/debezium
  2. debezium/debezium-connector-informix
  3. debezium/debezium-quarkus
  4. debezium/debezium-platform

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 2, 2026

There are now a few cross-repo dependencies, and it's probably something we should try to document clearly, particularly for first-time contributors. The debezium-quarkus repo now requires debezium-connector-informix to be built up-front. So if you clone that repository and build it, then that should solve your issue. Just be mindful that the Informix repo likely will require the main debezium/debezium repo to be built as well.

So build

  1. debezium/debezium
  2. debezium/debezium-connector-informix
  3. debezium/debezium-quarkus
  4. debezium/debezium-platform

Hi @Naros , I was able to resolve everything last night, except the Informix connector:

[INFO] DOCKER> [debezium/informix-test-database:15]: Created docker-build.tar in 26 milliseconds
[INFO] DOCKER> Credentials helper reply for "docker-credential-desktop" is docker-credential-osxkeychain (github.com/docker/docker-credential-helpers) v0.9.5
[ERROR] DOCKER> Unable to pull 'icr.io/informix/informix-developer-database:15.0' from registry 'icr.io' : {"message":"failed to resolve reference "icr.io/informix/informix-developer-database:15.0": icr.io/informix/informix-developer-database:15.0: not found"} (Not Found: 404) [{"message":"failed to resolve reference "icr.io/informix/informix-developer-database:15.0": icr.io/informix/informix-developer-database:15.0: not found"} (Not Found: 404)]

@Naros
Copy link
Copy Markdown
Member

Naros commented Apr 2, 2026

@pxcamus for the meantime can you try -Pinformix14 or -Pinformix12 to use the 14 or 12 images, do that work for you?

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 2, 2026

@pxcamus for the meantime can you try -Pinformix14 or -Pinformix12 to use the 14 or 12 images, do that work for you?

That worked, thanks!

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 2, 2026

@pxcamus for the meantime can you try -Pinformix14 or -Pinformix12 to use the 14 or 12 images, do that work for you?

@Naros actually a few more issues but on macOS ARM. the debezium-quarkus-testsuite-deployment produces a test JAR. So I could not run mvn install -Dmaven.test.skip=true. But when trying to skip just the execution of tests with mvn install -DskipTests I am getting this error:

[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.48.0:start (start-mongodb) on project debezium-quarkus-mongodb-integration-tests: I/O Error: Unable to pull 'quay.io/debezium/mongo:6.0' from registry 'quay.io' : {"message":"no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found"} (Not Found: 404) -> [Help 1]

I tried everything I could think of: setting up a default profile in ~/.m2/settings.xml:

<profiles> <profile> <id>default</id> <properties> <docker.host>unix://${user.home}/.docker/run/docker.sock</docker.host> <docker.platform>linux/amd64</docker.platform> </properties> </profile> </profiles>

I downloaded the image before running the Maven command:

docker pull --platform linux/amd64 quay.io/debezium/mongo:6.0

Then tried passing in various way a flag: DOCKER_DEFAULT_PLATFORM=linux/amd64 mvn install -DskipTests.

I think next I will have to modify the POM files to pass <platform>${docker.platform}</platform>.

Good news is that I am saving to buy a Linux workstation 😅

@mfvitale
Copy link
Copy Markdown
Member

mfvitale commented Apr 2, 2026

@pxcamus snapshots should now be published on maven central so you could just try to run test on your branch and dependencies should be resolved.

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 2, 2026

@pxcamus snapshots should now be published on maven central so you could just try to run test on your branch and dependencies should be resolved.

Still seeing this @mfvitale :

[ERROR] Failed to execute goal on project debezium-platform-conductor: Could not resolve dependencies for project io.debezium:debezium-platform-conductor:jar:3.6.0-SNAPSHOT [ERROR] dependency: io.debezium:debezium-operator-api:jar:3.6.0-SNAPSHOT (compile) [ERROR] Could not find artifact io.debezium:debezium-operator-api:jar:3.6.0-SNAPSHOT in confluent (https://packages.confluent.io/maven/)

@vsantonastaso
Copy link
Copy Markdown

@pxcamus for the meantime can you try -Pinformix14 or -Pinformix12 to use the 14 or 12 images, do that work for you?

@Naros actually a few more issues but on macOS ARM. the debezium-quarkus-testsuite-deployment produces a test JAR. So I could not run mvn install -Dmaven.test.skip=true. But when trying to skip just the execution of tests with mvn install -DskipTests I am getting this error:

[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.48.0:start (start-mongodb) on project debezium-quarkus-mongodb-integration-tests: I/O Error: Unable to pull 'quay.io/debezium/mongo:6.0' from registry 'quay.io' : {"message":"no matching manifest for linux/arm64/v8 in the manifest list entries: no match for platform in manifest: not found"} (Not Found: 404) -> [Help 1]

I tried everything I could think of: setting up a default profile in ~/.m2/settings.xml:

<profiles> <profile> <id>default</id> <properties> <docker.host>unix://${user.home}/.docker/run/docker.sock</docker.host> <docker.platform>linux/amd64</docker.platform> </properties> </profile> </profiles>

I downloaded the image before running the Maven command:

docker pull --platform linux/amd64 quay.io/debezium/mongo:6.0

Then tried passing in various way a flag: DOCKER_DEFAULT_PLATFORM=linux/amd64 mvn install -DskipTests.

I think next I will have to modify the POM files to pass <platform>${docker.platform}</platform>.

Good news is that I am saving to buy a Linux workstation 😅

Hi @pxcamus,
The ~/.m2/settings.xml approach couldn't work because docker-maven-plugin doesn't
automatically inherit properties from there.
You could try with:

  1. Add <docker.platform>linux/amd64</docker.platform> to the properties section
  2. Add ${docker.platform} inside each block
  3. run mvn command with -Ddocker.platform. e.g. mvn clean install -Ddocker.platform=linux/amd64

Ah for sure you have to enable Rosetta emulation in your docker engine

@pxcamus
Copy link
Copy Markdown
Author

pxcamus commented Apr 2, 2026

Thanks @vsantonastaso , that's what I thought I would have to do, but @mfvitale helped me out, and I was able to build.

pxcamus added 16 commits April 2, 2026 20:43
Signed-off-by: Philippe Camus <pxcamus@pm.me>
…min` integration and improved error handling. Update dependencies for Pulsar and Debezium.

Signed-off-by: Philippe Camus <pxcamus@pm.me>
… validation using Testcontainers

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…emes through `PulsarAuthHandler` abstraction

Signed-off-by: Philippe Camus <pxcamus@pm.me>
… `JwtAuthHandler` implementation and enhance connection validation logic

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…inProvider` interfaces to document authentication configuration and builder abstraction

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…s and minor code formatting adjustments

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…tion in interface

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…rFactory` and comprehensive unit tests for Pulsar authentication handlers.

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…tion with JWT authentication and enhance test coverage.

Signed-off-by: Philippe Camus <pxcamus@pm.me>
… scenarios for expired JWT tokens and inject missing `authScheme` configuration. Add constant for JWT token secret key.

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…uth2AuthHandler` and extend connection validation tests with integration support

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…ort order inconsistencies, and include missing license headers in Pulsar-related classes and tests

Signed-off-by: Philippe Camus <pxcamus@pm.me
Signed-off-by: Philippe Camus <pxcamus@pm.me>
…ence correct Apache Software License version

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…Test` to reflect authority component validation in service URI.

Signed-off-by: Philippe Camus <pxcamus@pm.me>
…ion tests to provide more specific failure reasons

Signed-off-by: Philippe Camus <pxcamus@pm.me>
@pxcamus pxcamus force-pushed the DBZ-9419-add-connection-validator-pulsar branch from 458c73f to 2e25131 Compare April 2, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants