[DRAFT][mysql] Clean-room auth handshake codec (scramble + handshake + packet, Stage 1a of #2093)

[rajvarun77]

[mysql] Clean-room MySQL client auth handshake (scramble + handshake + packet, Stage 1a of #2093)

Picks up MySQL protocol support per the staged plan announced on
dev@brpc.apache.org
and on PR #2093.
This is Stage 1a — the connection-phase authentication layer as
pure functions, with no integration into Socket yet (that lands in
the follow-up CL).

What's in this CL

New subsystem under src/brpc/policy/mysql_auth/, three modules:

Module	Public API
mysql_auth_scramble.{h,cpp}	NativePasswordScramble, CachingSha2PasswordScramble, CachingSha2PasswordRsaEncrypt, CachingSha2PasswordCleartext, CachingSha2PasswordSlowPath(..., bool is_ssl = false)
mysql_auth_packet.{h,cpp}	length-encoded int/string, packet header, NUL-terminated string
mysql_auth_handshake.{h,cpp}	ParseHandshakeV10, BuildHandshakeResponse41, ParseAuthSwitchRequest, ParseAuthMoreData

All written clean-room from the public MySQL protocol documentation;
SHA-256 and RSA-OAEP use OpenSSL EVP (works under both OpenSSL and
BoringSSL — verified in CI).

Test coverage — by-name mapping to upstream

75 unit tests across 3 files under test/mysql_auth/:

• brpc_mysql_auth_scramble_unittest.cpp — 36 tests
• brpc_mysql_auth_handshake_unittest.cpp — 19 tests
• brpc_mysql_auth_packet_unittest.cpp — 20 tests

mysql/mysql-server — unittest/gunit/sha2_password-t.cc (GPLv2)

#	Upstream test	brpc equivalent (or N/A)
1	SHA256_digestTest.InitDigestContext	N/A — exercises a server-internal SHA-256 context wrapper
2	SHA256_digestTest.DigestSingleStage	N/A — server-side digest helper
3	SHA256_digestTest.DigestMultiStage	N/A — server-side digest helper
4	SHA256_digestTest.GenerateScramble	MysqlCachingSha2PasswordTest.KnownVector_UpstreamMysqlServerTest + KnownVector_PasswordPassword_AsciiSalt + KnownVector_PasswordSecret_BinarySalt (independently regenerated vectors, including the upstream's own pw="Ab12#$Cd56&*" / nonce="eF!@34gH%^78" pair)
5	SHA256_digestTest.ValidateScramble	N/A — server-side validation of client's scramble
6	SHA256_digestTest.generate_sha256_scramble	Same as #4 — C-API alias for the same algorithm
7	SHA256_digestTest.validate_sha256_scramble	N/A — server-side
8	SHA256_digestTest.SHA2_password_cache	N/A — server-side credential cache
9	SHA256_digestTest.Caching_sha2_password_Serialize_Deserialize	N/A — mysql.user table storage format
10	SHA256_digestTest.Caching_sha2_password_generate_fast_digest	N/A — server-side fast-digest for cache lookup
11	SHA256_digestTest.Caching_sha2_password_generate_sha2_multi_hash	N/A — server-side iterated hash
12	SHA256_digestTest.Caching_sha2_password_authenticate_fast_authenticate	N/A — server-side fast-path authenticator
13	SHA256_digestTest.Caching_sha2_password_authenticate_sanity	MysqlCachingSha2PasswordTest.EmptyPasswordReturnsEmptyString + LongPassword (empty + overlong-input edge cases — the parts of the upstream sanity test that are client-relevant)

mysql/mysql-server — unittest/gunit/xplugin/xcl/sha256_scramble_t.cc (GPLv2)

#	Upstream test	brpc equivalent (or N/A)
14	SHA256_digest_test.init_digest_context	N/A — X-protocol-side context wrapper
15	SHA256_digest_test.digest_single_stage	N/A — X-protocol helper
16	SHA256_digest_test.digest_multi_stage	N/A — X-protocol helper
17	SHA256_digest_test.generate_scramble	Same as #4 — X-protocol wrapper invokes the same algorithm
18	SHA256_digest_test.generate_sha256_scramble	Same as #4

mysql/mysql-server — unittest/gunit/xplugin/xpl/user_password_verification_t.cc (GPLv2)

#	Upstream test	brpc equivalent (or N/A)
19	User_password_verification.native_plain_verification_get_salt	N/A — server X-protocol verifier salt extraction
20	User_password_verification.native_plain_verification_pass	N/A — server verifies a client scramble succeeds
21	User_password_verification.native_plain_verification_fail	N/A — server rejects a wrong scramble
22	User_password_verification.sha256_plain_verification_get_salt	N/A — server X-protocol sha256 salt extraction
23	User_password_verification.sha256_plain_verification_pass	N/A — server sha256 success path
24	User_password_verification.sha256_plain_verification_fail	N/A — server sha256 failure path
25	User_password_verification.sha256_memory_verification_get_salt	N/A — server in-memory cache salt
26	User_password_verification.sha256_memory_verification_pass	N/A — server cache hit success
27	User_password_verification.sha256_memory_verification_no_entry	N/A — server cache miss
28	User_password_verification.sha256_memory_verification_fail	N/A — server cache hit wrong-scramble

Subtotal: 28 upstream unit tests. 5 are client-relevant (#4, #6, #13, #17, #18 — all flavors of the fast-path scramble + the sanity edge cases) and are mirrored in MysqlCachingSha2PasswordTest.* with independently re-derived hex vectors. 23 are server-side-only (cache, storage, verification) and out of scope for a client library.

mysql/mysql-connector-cpp

Inspected cdk/mysqlx/auth_hash.cc — implementation only, no unit tests. Coverage in this repo is via Boost.Test integration suites only.

MariaDB/mariadb-connector-c — unittest/libmariadb/connection.c (integration only)

Source file (GPLv2 + FLOSS Exception).
These are live-server tests against mysql_real_connect(). Cannot run without a real mysqld/mariadbd; deferred to the integration-test CL (see "Deferred work" below).

#	Upstream integration test	What it exercises	Plan
29	test_auth256	caching_sha2_password full handshake — fast-auth-ok and cache-miss + RSA pubkey paths	Deferred to integration CL
30	test_default_auth	Server default auth plugin selection at greeting	Deferred
31	test_conc312	Auth-switch mid-handshake (native → caching_sha2 or vice-versa)	Deferred
32	test_parsec	MariaDB-only parsec (ed25519) auth plugin	Out of scope — see Limitations
33	test_change_user	COM_CHANGE_USER mid-session re-auth	Deferred
34	test_expired_pw	Password-expired server response	Deferred

Subtotal: 6 client-relevant integration tests. 0 mirrored in this CL — running them requires system("which mysqld")-style server bring-up, which I'd like to add as a follow-up commit on this same PR (see reviewer-ping comment asking about preferred pattern).

percona/percona-server

No auth tests beyond what upstream mysql-server already provides.

brpc test inventory

Test suite	Tests	Module covered
MysqlNativePasswordTest	10	scramble — native_password (SHA1 XOR)
MysqlCachingSha2PasswordTest	10	scramble — caching_sha2_password fast path (SHA256 XOR)
MysqlCachingSha2RsaTest	5	scramble — caching_sha2_password slow path, RSA-OAEP branch
MysqlCachingSha2CleartextTest	5	scramble — caching_sha2_password slow path, TLS-cleartext branch
MysqlCachingSha2SlowPathTest	6	scramble — slow-path dispatcher (is_ssl flag routing)
HandshakeV10Test	7	handshake — server greeting parser
HandshakeResponse41Test	5	handshake — client reply builder
AuthSwitchRequestTest	3	handshake — mid-handshake plugin switch
AuthMoreDataTest	4	handshake — server-side extra data (pubkey, fast-auth status)
LenencIntTest	9	packet — length-encoded int codec
LenencStringTest	4	packet — length-encoded string codec
PacketHeaderTest	4	packet — 4-byte packet header codec
NullTermStringTest	3	packet — NUL-terminated string parser
Total	75

Limitations of this CL

1. Only mysql_native_password + caching_sha2_password supported.
   No sha256_password (deprecated), no MariaDB parsec/ed25519, no
   auth_pam/LDAP/Kerberos. Callers must refuse unknown plugin names
   they see in an AuthSwitchRequest.
2. caching_sha2_password slow path supports both RSA-OAEP and
   TLS-cleartext branches. The CachingSha2PasswordSlowPath()
   dispatcher takes bool is_ssl = false as its trailing argument:
   when the caller has confirmed the channel is secure
   (TLS/unix-socket/shared-mem) it returns the cleartext payload
   (1 round trip); otherwise it falls through to RSA-OAEP
   (3 round trips). Callers that haven't yet been threaded with
   the TLS flag get the safe default (RSA-OAEP, works everywhere).
   See "TLS shortcut via is_ssl flag" section below.
3. No Socket integration. No PROTOCOL_MYSQL registration, no
   MysqlChannel. Auth codec is pure functions. Wiring is the next CL.
4. No compressed packets, multi-statement, prepared statements, or
   transactions. All planned for later CLs in the Add Mysql Protocol #2093 takeover
   sequence.
5. No packet fragmentation. The auth handshake fits in one packet,
   so the codec assumes the caller has already reassembled.
6. No integration tests. End-to-end auth correctness can only be
   verified against a real mysqld. Deferred to a follow-up commit on
   this PR — pattern question open with reviewers (see comment thread).

TLS shortcut via is_ssl flag

What libmysqlclient does

libmysqlclient picks the slow-path strategy at runtime via
is_secure_transport(mysql) in
sql-common/client_authentication.cc:770-787,
which returns true for VIO_TYPE_SSL (with a non-null cipher),
VIO_TYPE_SHARED_MEMORY, or VIO_TYPE_SOCKET (unix socket). Its
result is stored at line 810 and gated at
line 871:

/* If connection isn't secure attempt to get the RSA public key file */
if (!connection_is_secure) {
  public_key = rsa_init(mysql);
  ...
}

So: TLS / unix-socket / shared-mem → send cleartext password (one
round trip); plain TCP → request RSA pubkey, encrypt, send ciphertext
(three round trips).

How this CL exposes the same branch

The codec layer keeps the pure-function discipline: it does not
include brpc/socket.h or accept a Socket*. Instead it accepts a
defaulted bool is_ssl parameter and lets the caller (the future
Stage-1c protocol dispatcher) make the policy decision.

// mysql_auth_scramble.h

// TLS / unix-socket / shared-mem path. Sends "<password>\0".
std::string CachingSha2PasswordCleartext(
    const butil::StringPiece& password);

// Plain-TCP path. Sends RSA-OAEP-encrypted (password ^ salt).
std::string CachingSha2PasswordRsaEncrypt(
    const butil::StringPiece& server_pubkey_pem,
    const butil::StringPiece& salt,
    const butil::StringPiece& password);

// Dispatcher. Defaults is_ssl=false (safe — RSA-OAEP works
// everywhere) so callers that aren't yet threaded with TLS state
// keep working. Callers that know their channel is secure pass
// is_ssl=true to skip the RSA round trip.
std::string CachingSha2PasswordSlowPath(
    const butil::StringPiece& password,
    const butil::StringPiece& salt,
    const butil::StringPiece& server_pubkey_pem,
    bool is_ssl = false);

When the Stage-1c CL wires PROTOCOL_MYSQL and gains a Socket* at
the dispatcher layer, it threads
Socket::is_ssl()
straight into the trailing argument — no API change required.

Test coverage of the new branch

The is_ssl=true path adds 11 tests in two suites:

• MysqlCachingSha2CleartextTest (5 tests) — directly exercises the
  cleartext payload helper: NUL terminator appending, empty-password
  early return, embedded-NUL passthrough, UTF-8 multibyte passthrough,
  300-character overlong-password sanity.
• MysqlCachingSha2SlowPathTest (6 tests) — exercises the
  dispatcher: default is_ssl=false routes to RSA (verified by
  decrypt round-trip), explicit is_ssl=false likewise, is_ssl=true
  returns the cleartext payload, is_ssl=true ignores bad salt + bad
  pubkey arguments, is_ssl=true empty password returns empty,
  is_ssl=false with bad pubkey returns empty.

Cost comparison (per fresh login, server cache cold)

Path	Round trips	Wire bytes (typical)
is_ssl=true (cleartext)	1	password length + 1
is_ssl=false (RSA-OAEP)	3	1 + 270 (PEM) + 256 (ciphertext) ≈ 527

Steady-state queries take the fast-auth path (server cache hit, no
RSA, no cleartext) regardless of is_ssl. The dispatcher only fires
on cold-cache fresh logins.

Deferred work (in order)

1. CL 1b — Wire PROTOCOL_MYSQL registration + MysqlChannel +
   Socket::WriteOptions::auth_flags rebase. Threads
   Socket::is_ssl() straight into CachingSha2PasswordSlowPath's
   trailing argument.
2. CL 1c (or appended to this PR) — Real-mysqld integration
   tests mirroring test_auth256, test_default_auth, test_conc312,
   test_change_user, test_expired_pw using the redis-style
   system()-spawn pattern from test/brpc_redis_unittest.cpp.
3. CL 2 — Text protocol (COM_QUERY), basic result-set parsing.
4. CL 3 — Transactions (MysqlTransaction).
5. CL 4 — Prepared statements (COM_STMT_*).

Per @yanglimingcn — relationship to #2093

This CL covers the auth-handshake slice of PR #2093
with two key changes:

• mysql_auth_hash.cpp from the original PR (which was flagged
  GPL-licensed by @wwbmmm at
  #2093 (discussion))
  is fully reimplemented from the public spec.
• The scope is expanded to include caching_sha2_password (both
  fast and slow paths), which Add Mysql Protocol #2093 did not cover — this lets the
  follow-up CL connect to MySQL 8.0+ servers under their default
  auth plugin without an extra round trip.

I am not pushing to #2093's branch; the original commit history stays
intact.

Per @wwbmmm — review request

This is ready for review whenever you have bandwidth. The protocol
choices to focus on:

• Wire-format scramble output (3 algorithms, golden-vector tests).
• HandshakeV10 / HandshakeResponse41 codec round-trips.
• RSA-OAEP padding choice (matches what libmysqlclient uses).
• Limitations above — please flag anything you'd expect a client to
  do at this layer that I've punted.

External spec gist with the full mapping table (this section, plus
algorithm references and license posture):
https://gist.github.com/rajvarun77/e84f97360c553230966e305438dfbd0e

CI: all 17 jobs passed on the previous draft (bc95c031). Current
push (d8752f14) is in action_required state pending committer
approval — see the comment thread.

[rajvarun77]

Hi @wwbmmm @chenBright @yanglimingcn — process question on integration testing for this codec, before I write more code.

The pure-codec unit tests (59 across scramble / handshake / packet) are in place. For end-to-end auth verification I'd like to add an integration test that drives the full handshake against a real mysqld or mariadbd.

brpc already has a precedent for this in test/brpc_redis_unittest.cpp (L62–L87) — system("which redis-server") at startup, spawn it with a temp datadir if found, otherwise each test does puts("Skipped due to absence of redis-server"); return;. That gives real-protocol coverage on contributor machines and CI runners that have the binary, while staying green on lean environments.

Two questions before I mirror the pattern:

1. Is the redis-style system()-spawn approach the one you'd recommend for MySQL, or would you rather something different — Docker-based service, mtr-style harness, mock server only, etc.?
2. Is CI expected to have mysqld/mariadbd available (so the tests will actually run in Apache CI), or should the test always self-skip in CI and only execute locally?

The upstream MariaDB integration tests I'd want to mirror (test_auth256 cache-hit + cache-miss + RSA pubkey paths, test_conc312 mid-handshake auth-switch) are listed at the bottom of the spec gist. Happy to mirror whatever pattern you'd want long-term.

Also flagging — CI runs on this branch are currently in action_required state (workflow-approval gate for outside contributors). Could either of you click Approve and run on the Checks tab when convenient?

[rajvarun77]

Update — pushed 38a01f33 (force-push, rebased on master at 477fa492).

Two changes since the last comment:

1. TLS shortcut for caching_sha2_password slow path is now supported. Added CachingSha2PasswordCleartext() + a dispatcher CachingSha2PasswordSlowPath(password, salt, server_pubkey_pem, bool is_ssl = false). Default is_ssl=false preserves the existing RSA-OAEP behavior; callers that have confirmed the channel is secure pass is_ssl=true and skip the 3-round-trip RSA path. The codec layer stays pure-functional — Stage-1c's Socket integration just threads Socket::is_ssl() straight into the trailing argument. Full rationale + the libmysqlclient reference (sql-common/client_authentication.cc:770-787) is in the PR description under "TLS shortcut via is_ssl flag".

2. Test count: 75 (+11 from the previous push) — 5 new in MysqlCachingSha2CleartextTest + 6 new in MysqlCachingSha2SlowPathTest. PR description has the by-name mapping table to all 28 upstream unit tests in mysql/mysql-server plus the 6 client-relevant MariaDB integration tests (deferred).

Asks:

• @wwbmmm @chenBright @yanglimingcn — could you click Approve and run on the Checks tab? The previous 17-job suite all passed on bc95c031; the new commit is gated on the outside-contributor approval workflow.
• Still open: the integration-test approach question from the previous comment (redis-style system()-spawn of mysqld vs. Docker vs. mtr — whichever pattern you prefer).

PR ready for first-pass review whenever you have bandwidth.