catalog: index most of pg_authid

[teskje]

pgbouncer by default queries pg_authid to authenticate users. This query is slow because it reads from unindexed sources and computes an expensive dataflow. Ideally we would fix this by indexing it, but that's not possible because two columns are computed using unmaterializable functions. So instead we introduce a new pg_authid_core view that excludes these columns and can be indexed. pg_authid is then defined on top of this view.

The pg_authid query is written in a manually decorrelated way. This is necessary to ensure that queries that read from pg_authid but only materializable fields get optimized to simple lookups into the pg_authid_core index. See Slack thread.

Motivation

• This PR fixes a previously unreported bug.

pgbouncer's default auth queries are expensive.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

[teskje]

I was hoping querying the fields of mz_authid_core through mz_authid would be fast now because they could be optimized to a simple index lookup. However, seems like that's not actually the case:

materialize=> explain select oid from pg_authid;
                     Physical Plan
--------------------------------------------------------
 Explained Query:                                      +
   →With                                               +
     cte l0 =                                          +
       →Fused with Child Map/Filter/Project            +
         Project: #1                                   +
           →Arranged mz_internal.pg_authid_core        +
             Key: (#1{rolname})                        +
     cte l1 =                                          +
       →Distinct GroupAggregate                        +
         →Stream l0                                    +
     cte l2 =                                          +
       →Arrange (#0)                                   +
         →Fused with Child Map/Filter/Project          +
           Project: #1                                 +
             →Arranged mz_catalog.mz_roles             +
               Key: (#0{id})                           +
     cte l3 =                                          +
       →Differential Join %0 » %1                      +
         Join stage 0 in %1 with lookup key #0         +
         →Arranged l1                                  +
         →Arranged l2                                  +
     cte l4 =                                          +
       →Differential Join %0 » %1                      +
         Join stage 0 in %1 with lookup key #0         +
         →Arrange (#0)                                 +
           →Stream l0                                  +
         →Arrange (#0)                                 +
           →Union                                      +
             →Fused with Child Map/Filter/Project      +
               Project: #0, #1                         +
               Map: true                               +
                 →Read l3                              +
             →Map/Filter/Project                       +
               Project: #0, #1                         +
               Map: false                              +
                 →Consolidating Union                  +
                   →Negate Diffs                       +
                     →Stream l3                        +
                   →Unarranged Raw Stream              +
                     →Arranged l1                      +
     cte l5 =                                          +
       →Fused with Child Map/Filter/Project            +
         Project: #0                                   +
         Filter: (false = NOT(#1))                     +
           →Read l4                                    +
     cte l6 =                                          +
       →Union                                          +
         →Fused with Child Map/Filter/Project          +
           Project: #0                                 +
           Filter: NOT(#1)                             +
             →Read l4                                  +
         →Differential Join %1 » %0                    +
           Join stage 0 in %0 with lookup key #0       +
           →Arrange (#0)                               +
             →Stream l5                                +
           →Distinct GroupAggregate                    +
             →Stream l5                                +
     cte l7 =                                          +
       →Distinct GroupAggregate                        +
         →Stream l6                                    +
     cte l8 =                                          +
       →Differential Join %0 » %1                      +
         Join stage 0 in %1 with lookup key #0         +
         →Arranged l7                                  +
         →Arranged l2                                  +
     cte l9 =                                          +
       →Differential Join %0 » %1                      +
         Join stage 0 in %1 with lookup key #0         +
         →Arrange (#0)                                 +
           →Stream l6                                  +
         →Arrange (#0)                                 +
           →Union                                      +
             →Fused with Child Map/Filter/Project      +
               Project: #0, #1                         +
               Map: true                               +
                 →Read l8                              +
             →Map/Filter/Project                       +
               Project: #0, #1                         +
               Map: false                              +
                 →Consolidating Union                  +
                   →Negate Diffs                       +
                     →Stream l8                        +
                   →Unarranged Raw Stream              +
                     →Arranged l7                      +
     cte l10 =                                         +
       →Fused with Child Map/Filter/Project            +
         Project: #0                                   +
         Filter: (false = NOT(#1))                     +
           →Read l9                                    +
   →Return                                             +
     →Union                                            +
       →Fused with Child Map/Filter/Project            +
         Project: #0                                   +
         Filter: NOT(#1)                               +
           →Read l9                                    +
       →Differential Join %1 » %0                      +
         Join stage 0 in %0 with lookup key #0         +
         →Arrange (#0)                                 +
           →Stream l10                                 +
         →Distinct GroupAggregate                      +
           →Stream l10                                 +
                                                       +
 Source mz_catalog.mz_system_privileges                +
                                                       +
 Used Indexes:                                         +
   - mz_internal.pg_authid_core_ind (*** full scan ***)+
   - mz_catalog.mz_roles_ind (*** full scan ***)       +
                                                       +
 Target cluster: mz_catalog_server                     +

(1 row)

A missed optimization or am I missing something?

[teskje]

Fixed! This is the plan for the pgbouncer query now:

materialize=> explain SELECT rolname, CASE WHEN rolvaliduntil < now() THEN NULL ELSE rolpassword END FROM pg_authid WHERE rolname='foo' AND rolcanlogin;
                                              Physical Plan
---------------------------------------------------------------------------------------------------------
 Explained Query (fast path):                                                                           +
   →Map/Filter/Project                                                                                  +
     Project: #0, #11                                                                                   +
     Filter: #4{rolcanlogin}                                                                            +
     Map: case when (#9{rolvaliduntil} < 2025-12-16 17:56:35.584 UTC) then null else #8{rolpassword} end+
       →Index Lookup on mz_internal.pg_authid_core (using mz_internal.pg_authid_core_ind)               +
         Lookup values: ("foo")                                                                         +
                                                                                                        +
 Used Indexes:                                                                                          +
   - mz_internal.pg_authid_core_ind (lookup)                                                            +
                                                                                                        +
 Target cluster: mz_catalog_server                                                                      +

(1 row)

[SangJunBak]

Why a left join on extra and not just a single FROM with additional columns/projections for rolcreaterole and rolcreatedb?

[teskje]

I've added a comment now that explains this, but it's necessary to go from the bad EXPLAIN plan I posted above to the good EXPLAIN plan I also posted above. See also this thread.

[ggevay]

LGTM!

[teskje]

TFTRs!