Merge #147318

147318: jwtauthccl, securityccl: sync SQL roles from JWT groups claim r=pritesh-lahoti,souravcrl a=shriramters

Previously, JWT login authenticated users but did not mirror group membership into SQL roles.  

This was inadequate because one would like to also authorize the user based on group membership information present in the JWT.  

A similar role-sync is already in place in the LDAP auth flow.

To address this, this patch adds an authorizer that grants/revokes roles listed in the token’s "groups" claim, or any other claim as configured.  If the JWT does not contain a field with groups, the authorizer tries to fetch the groups from the IdP's userinfo endpoint.

If the fetched groups list is empty, meaning the claim exists but contains `[]` or the userinfo response yields no groups, the server:
1. revokes all existing SQL role memberships for the user, and
2. rejects the login with `JWT authorization: empty group list`.

The feature is disabled by default and gated by `server.jwt_authentication.authorization.enabled`.

Additionally, the field names of the JWT and userinfo JSONs where cockroach looks for the groups list can be customized using `server.jwt_authentication.group_claim` and `server.jwt_authentication.userinfo_group_key` respectively.

This commit also provides a new package `securityccl/jwthelper` that provides `ExtractGroups()`, a small helper that accepts a JWT and extracts and normalizes group information.
This helper is now used by jwtauthccl and can be reused by oidcccl for supporting authorization in the OIDC flow.

Fixes: #104671
Epic: [CRDB-48763](https://cockroachlabs.atlassian.net/browse/CRDB-48763)

Release note (security update): CockroachDB can now synchronise SQL role membership from the groups claim contained in a JWT when `server.jwt_authentication.authorization.enabled = true`.  The claim name and the fallback *userinfo* JSON key are configurable by `server.jwt_authentication.group_claim` and `server.jwt_authentication.userinfo_group_key` respectively. Behaviour matches the existing LDAP role-sync feature.

Common IdP examples
-------------------
### Okta (default mapping)
```sql
  SET CLUSTER SETTING server.jwt_authentication.group_claim = 'groups';
  SET CLUSTER SETTING server.jwt_authentication.userinfo_group_key = 'groups';
```
### Keycloak
If you map Keycloak Groups to a JWT claim (default):
  - `group_claim = 'groups'             -- e.g. "groups": ["team-a","team-b"]`
If you prefer Keycloak’s built-in realm roles, CockroachDB does not follow nested paths like "realm_access.roles".  Add a Keycloak protocol mapper that flattens roles into a top-level claim (for example "roles") and then
configure:
- `group_claim = 'roles';             -- e.g. "roles": ["team-a","team-b"]`

The userinfo endpoint usually returns the same JSON keys, so the same values
work for `userinfo_group_key` as well.


Co-authored-by: Shriram Ravindranathan <[email protected]>

Author: craig[bot]

docs/generated/settings/settings-for-tenants.txt

@@ -125,13 +125,16 @@ server.hsts.enabled	boolean	false	if true, HSTS headers will be sent along with
 server.http.base_path	string	/	path to redirect the user to upon succcessful login	application
 server.identity_map.configuration	string		system-identity to database-username mappings	application
 server.jwt_authentication.audience	string		sets accepted audience values for JWT logins over the SQL interface	application
+server.jwt_authentication.authorization.enabled	boolean	false	enables role synchronisation based on group claims in JWTs	application
 server.jwt_authentication.claim	string		sets the JWT claim that is parsed to get the username	application
 server.jwt_authentication.client.timeout	duration	15s	sets the client timeout for external calls made during JWT authentication (e.g. fetching JWKS, etc.)	application
 server.jwt_authentication.enabled	boolean	false	enables or disables JWT login for the SQL interface	application
+server.jwt_authentication.group_claim	string	groups	sets the name of the JWT claim that contains groups used for role mapping	application
 server.jwt_authentication.issuers.configuration (alias: server.jwt_authentication.issuers)	string		sets accepted issuer values for JWT logins over the SQL interface which can be a single issuer URL string or a JSON string containing an array of issuer URLs or a JSON object containing map of issuer URLS to JWKS URIs	application
 server.jwt_authentication.issuers.custom_ca	string		sets the PEM encoded custom root CA for verifying certificates while fetching JWKS	application
 server.jwt_authentication.jwks	string	"{""keys"":[]}"	sets the public key set for JWT logins over the SQL interface (JWKS format)	application
 server.jwt_authentication.jwks_auto_fetch.enabled	boolean	false	enables or disables automatic fetching of JWKS from the issuer's well-known endpoint or JWKS URI set in JWTAuthIssuersConfig. If this is enabled, the server.jwt_authentication.jwks will be ignored.	application
+server.jwt_authentication.userinfo_group_key	string	groups	sets the field name to look for in userinfo JSON that lists groups when groups claim is absent from JWT	application
 server.ldap_authentication.client.tls_certificate	string		sets the client certificate PEM for establishing mTLS connection with LDAP server	application
 server.ldap_authentication.client.tls_key	string		sets the client key PEM for establishing mTLS connection with LDAP server	application
 server.ldap_authentication.domain.custom_ca	string		sets the PEM encoded custom root CA for verifying domain certificates when establishing connection with LDAP server	application

docs/generated/settings/settings.html

@@ -157,13 +157,16 @@
 <tr><td><div id="setting-server-http-base-path" class="anchored"><code>server.http.base_path</code></div></td><td>string</td><td><code>/</code></td><td>path to redirect the user to upon succcessful login</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-identity-map-configuration" class="anchored"><code>server.identity_map.configuration</code></div></td><td>string</td><td><code></code></td><td>system-identity to database-username mappings</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-audience" class="anchored"><code>server.jwt_authentication.audience</code></div></td><td>string</td><td><code></code></td><td>sets accepted audience values for JWT logins over the SQL interface</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
+<tr><td><div id="setting-server-jwt-authentication-authorization-enabled" class="anchored"><code>server.jwt_authentication.authorization.enabled</code></div></td><td>boolean</td><td><code>false</code></td><td>enables role synchronisation based on group claims in JWTs</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-claim" class="anchored"><code>server.jwt_authentication.claim</code></div></td><td>string</td><td><code></code></td><td>sets the JWT claim that is parsed to get the username</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-client-timeout" class="anchored"><code>server.jwt_authentication.client.timeout</code></div></td><td>duration</td><td><code>15s</code></td><td>sets the client timeout for external calls made during JWT authentication (e.g. fetching JWKS, etc.)</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-enabled" class="anchored"><code>server.jwt_authentication.enabled</code></div></td><td>boolean</td><td><code>false</code></td><td>enables or disables JWT login for the SQL interface</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
+<tr><td><div id="setting-server-jwt-authentication-group-claim" class="anchored"><code>server.jwt_authentication.group_claim</code></div></td><td>string</td><td><code>groups</code></td><td>sets the name of the JWT claim that contains groups used for role mapping</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-issuers" class="anchored"><code>server.jwt_authentication.issuers.configuration<br />(alias: server.jwt_authentication.issuers)</code></div></td><td>string</td><td><code></code></td><td>sets accepted issuer values for JWT logins over the SQL interface which can be a single issuer URL string or a JSON string containing an array of issuer URLs or a JSON object containing map of issuer URLS to JWKS URIs</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-issuers-custom-ca" class="anchored"><code>server.jwt_authentication.issuers.custom_ca</code></div></td><td>string</td><td><code></code></td><td>sets the PEM encoded custom root CA for verifying certificates while fetching JWKS</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-jwks" class="anchored"><code>server.jwt_authentication.jwks</code></div></td><td>string</td><td><code>{"keys":[]}</code></td><td>sets the public key set for JWT logins over the SQL interface (JWKS format)</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-jwt-authentication-jwks-auto-fetch-enabled" class="anchored"><code>server.jwt_authentication.jwks_auto_fetch.enabled</code></div></td><td>boolean</td><td><code>false</code></td><td>enables or disables automatic fetching of JWKS from the issuer&#39;s well-known endpoint or JWKS URI set in JWTAuthIssuersConfig. If this is enabled, the server.jwt_authentication.jwks will be ignored.</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
+<tr><td><div id="setting-server-jwt-authentication-userinfo-group-key" class="anchored"><code>server.jwt_authentication.userinfo_group_key</code></div></td><td>string</td><td><code>groups</code></td><td>sets the field name to look for in userinfo JSON that lists groups when groups claim is absent from JWT</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-ldap-authentication-client-tls-certificate" class="anchored"><code>server.ldap_authentication.client.tls_certificate</code></div></td><td>string</td><td><code></code></td><td>sets the client certificate PEM for establishing mTLS connection with LDAP server</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-ldap-authentication-client-tls-key" class="anchored"><code>server.ldap_authentication.client.tls_key</code></div></td><td>string</td><td><code></code></td><td>sets the client key PEM for establishing mTLS connection with LDAP server</td><td>Serverless/Dedicated/Self-Hosted</td></tr>
 <tr><td><div id="setting-server-ldap-authentication-domain-custom-ca" class="anchored"><code>server.ldap_authentication.domain.custom_ca</code></div></td><td>string</td><td><code></code></td><td>sets the PEM encoded custom root CA for verifying domain certificates when establishing connection with LDAP server</td><td>Serverless/Dedicated/Self-Hosted</td></tr>

pkg/BUILD.bazel

@@ -995,6 +995,7 @@ GO_TARGETS = [
     "//pkg/ccl/schemachangerccl:schemachangerccl",
     "//pkg/ccl/schemachangerccl:schemachangerccl_test",
     "//pkg/ccl/securityccl/fipsccl:fipsccl",
+    "//pkg/ccl/securityccl/jwthelper:jwthelper",
     "//pkg/ccl/serverccl/adminccl:adminccl_test",
     "//pkg/ccl/serverccl/diagnosticsccl:diagnosticsccl_test",
     "//pkg/ccl/serverccl/statusccl:statusccl_test",

pkg/ccl/jwtauthccl/BUILD.bazel

@@ -4,11 +4,13 @@ go_library(
     name = "jwtauthccl",
     srcs = [
         "authentication_jwt.go",
+        "authorization_jwt.go",
         "settings.go",
     ],
     importpath = "github.com/cockroachdb/cockroach/pkg/ccl/jwtauthccl",
     visibility = ["//visibility:public"],
     deps = [
+        "//pkg/ccl/securityccl/jwthelper",
         "//pkg/ccl/utilccl",
         "//pkg/security/username",
         "//pkg/server/telemetry",
@@ -34,6 +36,7 @@ go_test(
     size = "medium",
     srcs = [
         "authentication_jwt_test.go",
+        "authorization_jwt_test.go",
         "main_test.go",
         "settings_test.go",
     ],
@@ -47,6 +50,8 @@ go_test(
         "//pkg/security/securitytest",
         "//pkg/security/username",
         "//pkg/server",
+        "//pkg/settings",
+        "//pkg/settings/cluster",
         "//pkg/sql/pgwire/identmap",
         "//pkg/testutils",
         "//pkg/testutils/serverutils",

pkg/ccl/jwtauthccl/authentication_jwt.go

@@ -10,6 +10,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"net/http"
 	"strings"
 
 	"github.com/cockroachdb/cockroach/pkg/ccl/utilccl"
@@ -146,6 +147,7 @@ func (authenticator *jwtAuthenticator) ValidateJWTLogin(
 	tokenBytes []byte,
 	identMap *identmap.Conf,
 ) (detailedErrorMsg redact.RedactableString, authError error) {
+	authenticator.reloadConfig(ctx, st)
 	authenticator.mu.Lock()
 	defer authenticator.mu.Unlock()
 
@@ -370,8 +372,42 @@ func getOpenIdConfigEndpoint(issuerUrl string) string {
 	return openIdConfigEndpoint
 }
 
-var getHttpResponse = func(ctx context.Context, url string, authenticator *jwtAuthenticator) ([]byte, error) {
-	resp, err := authenticator.mu.conf.httpClient.Get(context.Background(), url)
+// getHTTPResponse issues a GET request using the authenticator’s configured
+// HTTP client, optionally setting the supplied headers.
+//
+//	ctx – caller’s context (for cancellation / deadlines)
+//	url – absolute URL to fetch
+//	a   – the *jwtAuthenticator whose client must be reused
+//	hdr – optional: pass one http.Header with any extra headers, or omit entirely
+//
+// The function returns the response body (fully read) so that callers can
+// inspect the payload without worrying about closing the body.
+//
+// Callers should wrap the returned error with errors.WithDetailf to tag the
+// operation they’re performing.
+var getHttpResponse = func(
+	ctx context.Context,
+	url string,
+	authenticator *jwtAuthenticator,
+	header ...http.Header, // optional variadic param for extra headers
+) ([]byte, error) {
+
+	// Reject misuse: only zero or one header may be supplied.
+	if len(header) > 1 {
+		return nil, errors.New("getHttpResponse: provide at most one extra header")
+	}
+
+	// Build the request
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	if len(header) == 1 && header[0] != nil {
+		req.Header = header[0].Clone()
+	}
+
+	client := authenticator.mu.conf.httpClient
+	resp, err := client.Do(req)
 	if err != nil {
 		return nil, err
 	}

pkg/ccl/jwtauthccl/authentication_jwt_test.go

@@ -672,7 +672,7 @@ func TestAudienceCheck(t *testing.T) {
 
 // mockGetHttpResponseWithLocalFileContent is a mock function for getHttpResponse. This is used to intercept the call to
 // getHttpResponse and return the content of a local file instead of making a http call.
-var mockGetHttpResponseWithLocalFileContent = func(ctx context.Context, url string, authenticator *jwtAuthenticator) ([]byte, error) {
+var mockGetHttpResponseWithLocalFileContent = func(ctx context.Context, url string, authenticator *jwtAuthenticator, _ ...http.Header) ([]byte, error) {
 	// remove https:// and replace / with _ in the url to get the testdata file name
 	fileName := "testdata/" + strings.ReplaceAll(strings.ReplaceAll(url, "https://", ""), "/", "_")
 	// read content of the file as a byte array
@@ -999,7 +999,7 @@ func TestJWTAuthClientTimeout(t *testing.T) {
 		testServer.Close()
 	}()
 
-	mockGetHttpResponse := func(ctx context.Context, url string, authenticator *jwtAuthenticator) ([]byte, error) {
+	mockGetHttpResponse := func(ctx context.Context, url string, authenticator *jwtAuthenticator, _ ...http.Header) ([]byte, error) {
 		if strings.Contains(url, "/.well-known/openid-configuration") {
 			return mockGetHttpResponseWithLocalFileContent(ctx, url, authenticator)
 		} else if strings.Contains(url, "/oauth2/v3/certs") {