Add support for PLAIN authn for Shadow Linking #28708

michael-redpanda · 2025-11-23T19:28:53Z

This PR adds support for authenticating using SASL/PLAIN with Shadow Linking.

This adds a new authentication configuration called PlainConfig:

// Authentication config.  Supports:
// * SASL/SCRAM
// * SASL/PLAIN
message AuthenticationConfiguration {
    oneof authentication {
        // SASL/SCRAM configuration
        ScramConfig scram_configuration = 1;
        // SASL/PLAIN configuration
        PlainConfig plain_configuration = 2;
    }
}
// PLAIN settings
message PlainConfig {
    // PLAIN username
    string username = 1;
    // Password
    string password = 2 [(google.api.field_behavior) = INPUT_ONLY];
    // Indicates that the password has been set
    bool password_set = 3 [(google.api.field_behavior) = OUTPUT_ONLY];
    // Timestamp of when the password was last set - only valid if password_set
    // is true
    google.protobuf.Timestamp password_set_at = 4
        [(google.api.field_behavior) = OUTPUT_ONLY];
}

Backports Required

Release Notes

Improvements

Adds support for SASL/PLAIN authentication for Shadow Linking

Copilot

Pull request overview

This PR adds support for SASL/PLAIN authentication to Shadow Linking, which previously only supported SASL/SCRAM. The implementation introduces a new PlainConfig message type in the protobuf schema, extends the authentication handling in the converter layer, and adds test coverage for PLAIN authentication.

Adds PlainConfig protobuf message with username/password fields similar to ScramConfig
Updates authentication handling to support PLAIN alongside existing SCRAM mechanisms
Implements PLAIN authentication flow in the Kafka client broker

Reviewed changes

Copilot reviewed 25 out of 43 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
proto/redpanda/core/admin/v2/shadow_link.proto	Adds PlainConfig message and extends AuthenticationConfiguration oneof
src/v/redpanda/admin/services/shadow_link/converter.cc	Implements conversion logic between PlainConfig proto and internal credentials
src/v/kafka/client/broker.cc	Adds PLAIN authentication implementation with proper message formatting
src/v/kafka/client/broker.h	Declares do_authenticate_plain method
src/v/cluster/cluster_link/frontend.cc	Extends validation to accept "PLAIN" mechanism
src/v/redpanda/admin/services/shadow_link/tests/converter_test.cc	Adds test coverage for PLAIN config creation and conversion
tests/rptest/tests/cluster_linking_topic_syncing_test.py	Adds test class for PLAIN authentication scenario
tests/rptest/clients/admin/proto/*	Generated Python protobuf bindings updates

tests/rptest/tests/cluster_linking_topic_syncing_test.py

src/v/redpanda/admin/services/shadow_link/converter.cc

src/v/kafka/client/broker.cc

rockwotj

you need to regenerate protos using the script in the tools directory of the redpanda repository. The old one in vtools is out of date (I will remove)

vbotbuildovich · 2025-11-23T22:01:48Z

CI test results

test results on build#76858

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
EndToEndCloudTopicsTest	test_delete_records	null	integration	https://buildkite.com/redpanda/redpanda/builds/76858#019ab25a-7b1c-4d14-abb0-ea89bebbed5b	FLAKY	20/21	upstream reliability is '99.75550122249389'. current run reliability is '95.23809523809523'. drift is 4.51741 and the allowed drift is set to 50. The test should PASS	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=EndToEndCloudTopicsTest&test_method=test_delete_records
MountUnmountIcebergTest	test_simple_remount	{"cloud_storage_type": 1}	integration	https://buildkite.com/redpanda/redpanda/builds/76858#019ab260-387a-45e2-b5f2-f27284937fc1	FLAKY	14/21	upstream reliability is '78.63910422049956'. current run reliability is '66.66666666666666'. drift is 11.97244 and the allowed drift is set to 50. The test should PASS	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=MountUnmountIcebergTest&test_method=test_simple_remount

test results on build#76874

test_class	test_method	test_arguments	test_kind	job_url	test_status	passed	reason	test_history
ConsumerOffsetsConsistencyTest	test_flipping_leadership	null	integration	https://buildkite.com/redpanda/redpanda/builds/76874#019ab5b7-de24-4933-8ffe-64f0e40e449f	FLAKY	20/21	upstream reliability is '100.0'. current run reliability is '95.23809523809523'. drift is 4.7619 and the allowed drift is set to 50. The test should PASS	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ConsumerOffsetsConsistencyTest&test_method=test_flipping_leadership
MountUnmountIcebergTest	test_simple_remount	{"cloud_storage_type": 1}	integration	https://buildkite.com/redpanda/redpanda/builds/76874#019ab5b2-0624-458f-ac3f-ae78e5bd69c7	FLAKY	18/21	upstream reliability is '78.66071428571428'. current run reliability is '85.71428571428571'. drift is -7.05357 and the allowed drift is set to 50. The test should PASS	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=MountUnmountIcebergTest&test_method=test_simple_remount

michael-redpanda · 2025-11-24T11:28:40Z

you need to regenerate protos using the script in the tools directory of the redpanda repository. The old one in vtools is out of date (I will remove)

thanks! is that new?

michael-redpanda · 2025-11-24T11:30:45Z

Force push:

Used correct proto gen script

BenPope · 2025-11-24T12:36:52Z

src/v/kafka/client/broker.cc

+    std::string bytes;
+    // 2 - number of null characters in the PLAIN auth message
+    bytes.reserve(2 + username.size() + password.size());
+    bytes.push_back('\0');
+    bytes.append(username.cbegin(), username.cend());
+    bytes.push_back('\0');
+    bytes.append(password.cbegin(), password.cend());
+    req.data.auth_bytes = bytes::from_string(std::move(bytes));


nitpick:

Suggested change

std::string bytes;

// 2 - number of null characters in the PLAIN auth message

bytes.reserve(2 + username.size() + password.size());

bytes.push_back('\0');

bytes.append(username.cbegin(), username.cend());

bytes.push_back('\0');

bytes.append(password.cbegin(), password.cend());

req.data.auth_bytes = bytes::from_string(std::move(bytes));

auto bytes = fmt::format("\0{}\0{}", username, password);

req.data.auth_bytes = bytes::from_string(bytes);

So I tried this but I think the std::string_view that gets created via the bytes::from_string reports a length of 0 since the string begins with a null character

You, my bad, you'll need:

auto bytes = fmt::format("\0{}\0{}", username, password); req.data.auth_bytes = bytes::from_string(std::string_view{bytes.begin(), bytes.end()});

That unfortunately won't work either... end() is start() + size() and in this case size() is 0

It should work if the format string is a std::string_view!

Can you provide an example of what you mean please

src/v/cluster/cluster_link/frontend.cc

src/v/redpanda/admin/services/shadow_link/converter.cc

BenPope · 2025-11-24T12:51:36Z

tests/rptest/tests/cluster_linking_topic_syncing_test.py


+class ClusterLinkingTopicSyncingWithPlain(ClusterLinkingTopicSyncingTestBase):
+    """
+    Run the same battery of tests with PLAIN


The same as what?

Is it necessary to run the whole test suite with each auth mechanism?

rockwotj · 2025-11-24T15:06:39Z

you need to regenerate protos using the script in the tools directory of the redpanda repository. The old one in vtools is out of date (I will remove)

thanks! is that new?

Yes it is, see:

Also the rpk protogen stuff is required for now, although hopefully we can switch those to work off of the bufbuild protos instead of the checked in ones (cc: @r-vasquez)

For now you can fix those via:

buf generate --path proto/redpanda/core/admin/v2/shadow_link.proto --path proto/redpanda/core/common/v1/acl.proto --path proto/redpanda/core/common/v1/tls.proto

michael-redpanda · 2025-11-24T21:16:39Z

Force push:

Addressed some PR comments

Second force push:

Rebased off of dev

Signed-off-by: Michael Boquard <[email protected]>

michael-redpanda · 2025-11-24T21:18:38Z

Force push:

another rebase

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Copilot · 2025-11-24T21:19:00Z

src/v/redpanda/admin/services/shadow_link/converter.cc

+              } else {
+                  throw std::invalid_argument(
+                    ssx::sformat(
+                      "Unknown SCRAM mechanism: {}", scram.mechanism));


The error message refers to 'SCRAM mechanism' but this error can now be triggered when using PLAIN authentication (which is not a SCRAM mechanism). Consider changing to 'Unknown authentication mechanism: {}' or adding separate validation to distinguish between SCRAM and PLAIN cases.

Suggested change

"Unknown SCRAM mechanism: {}", scram.mechanism));

"Unknown authentication mechanism: {}", scram.mechanism));

If we're gonna do that, it can go in another else on the outer scope. It's unreachable, though.

Copilot · 2025-11-24T21:19:00Z

src/v/kafka/client/broker.cc

    } else if (mechanism == security::scram_sha512_authenticator::name) {
        co_await do_authenticate_scram512(username, password);
    } else if (mechanism == security::oidc::sasl_authenticator::name) {
        co_await do_authenticate_oauthbearer(password);
+    } else if (mechanism == security::plain_authenticator::name) {
+        co_await do_authenticate_plain(username, password);
+    } else {
+        throw broker_error{
+          _node_id,
+          error_code::sasl_authentication_failed,
+          fmt_with_ctx(ssx::sformat, "Unknown mechanism: {}", mechanism)};
    }


The newly added else-if at line 357-358 makes the earlier check at lines 321-328 redundant. The earlier check validates that mechanism must be one of scram_sha256, scram_sha512, oidc, or plain, but the else block at lines 359-364 now provides the same validation with a better error message. Consider removing the earlier validation block to avoid duplication.

Copilot · 2025-11-24T21:19:01Z

src/v/cluster/cluster_link/frontend.cc

+                && c.mechanism != "PLAIN") {
                  vlog(
                    cluster::clusterlog.warn,
                    "Unsupported SCRAM mechanism: {}",


The log message at line 988 still refers to 'Unsupported SCRAM mechanism' but PLAIN is not a SCRAM mechanism. The message should be updated to 'Unsupported authentication mechanism' to accurately reflect that both SCRAM and PLAIN mechanisms are now supported.

Suggested change

"Unsupported SCRAM mechanism: {}",

"Unsupported authentication mechanism: {}",

vbotbuildovich · 2025-11-25T20:12:08Z

/backport v25.3.x

vbotbuildovich · 2025-11-25T20:13:31Z

Failed to create a backport PR to v25.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-28708-v25.3.x-922 remotes/upstream/v25.3.x
git cherry-pick -x 31614c7f68 61430f113e 8acce1791f

Workflow run logs.

michael-redpanda requested review from BenPope and r-vasquez November 23, 2025 19:28

michael-redpanda self-assigned this Nov 23, 2025

Copilot AI review requested due to automatic review settings November 23, 2025 19:28

michael-redpanda requested review from a team and rockwotj as code owners November 23, 2025 19:28

github-actions bot added the area/redpanda label Nov 23, 2025

Copilot AI reviewed Nov 23, 2025

View reviewed changes

tests/rptest/tests/cluster_linking_topic_syncing_test.py Outdated Show resolved Hide resolved

src/v/redpanda/admin/services/shadow_link/converter.cc Outdated Show resolved Hide resolved

src/v/kafka/client/broker.cc Show resolved Hide resolved

rockwotj reviewed Nov 23, 2025

View reviewed changes

michael-redpanda force-pushed the sl/support-plain branch from 9109458 to 9852249 Compare November 24, 2025 11:30

michael-redpanda requested a review from rockwotj November 24, 2025 11:30

BenPope reviewed Nov 24, 2025

View reviewed changes

michael-redpanda force-pushed the sl/support-plain branch from 9852249 to f57056a Compare November 24, 2025 21:13

michael-redpanda requested review from a team and kbatuigas as code owners November 24, 2025 21:13

github-actions bot added the area/rpk label Nov 24, 2025

michael-redpanda force-pushed the sl/support-plain branch from f57056a to 81b8dd8 Compare November 24, 2025 21:16

michael-redpanda added 3 commits November 24, 2025 16:17

k/c: Add support for PLAIN authentication

31614c7

Signed-off-by: Michael Boquard <[email protected]>

proto: Add support for PLAIN authn to Shadow Link

61430f1

Signed-off-by: Michael Boquard <[email protected]>

dt: Add PLAIN tests

8acce17

Signed-off-by: Michael Boquard <[email protected]>

michael-redpanda force-pushed the sl/support-plain branch from 81b8dd8 to 8acce17 Compare November 24, 2025 21:17

michael-redpanda requested review from BenPope and Copilot November 24, 2025 21:18

Copilot AI reviewed Nov 24, 2025

View reviewed changes

BenPope approved these changes Nov 25, 2025

View reviewed changes

rockwotj approved these changes Nov 25, 2025

View reviewed changes

michael-redpanda merged commit 51fd551 into redpanda-data:dev Nov 25, 2025
31 checks passed

vbotbuildovich mentioned this pull request Nov 25, 2025

[v25.3.x] Add support for PLAIN authn for Shadow Linking #28744

Closed

michael-redpanda mentioned this pull request Dec 11, 2025

[v25.3.x] Add support for PLAIN authn for Shadow Linking #28942

Merged

	"Unknown SCRAM mechanism: {}", scram.mechanism));
	"Unknown authentication mechanism: {}", scram.mechanism));

	"Unsupported SCRAM mechanism: {}",
	"Unsupported authentication mechanism: {}",

Add support for PLAIN authn for Shadow Linking #28708

Add support for PLAIN authn for Shadow Linking #28708

Conversation

michael-redpanda commented Nov 23, 2025

Backports Required

Release Notes

Improvements

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rockwotj left a comment

Choose a reason for hiding this comment

Uh oh!

vbotbuildovich commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI test results

Uh oh!

michael-redpanda commented Nov 24, 2025

Uh oh!

michael-redpanda commented Nov 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rockwotj commented Nov 24, 2025

Uh oh!

michael-redpanda commented Nov 24, 2025

Uh oh!

michael-redpanda commented Nov 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vbotbuildovich commented Nov 25, 2025

Uh oh!

vbotbuildovich commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vbotbuildovich commented Nov 23, 2025 •

edited

Loading