Bug Report: `SET` Query Fails on `@replica` When `AnyKeyspace` Selects Keyspace Without Replica Tablets

[arthurschreiber]

Overview of the Issue

When a client connects to VTGate using @replica (global routing without specifying a keyspace) and executes a SET query, the query fails with no healthy tablet available if the alphabetically first serving keyspace has no REPLICA tablets.

There's different scenarios how users can run into this error. This can either happen on every SET query, if the alphabetically first keyspace does not have tablets of the requested type. Or it can happen only during specific conditions like a PRS / ERS / downtime, where the alphabetically first keyspace has tablets of the requested type but is unavailable for a short moment - thus causing the next keyspace in the list to be selected (which then might not have tablets of the requested type).

---

The issue is in how AnyKeyspace() selects a keyspace when no specific keyspace is targeted:

1. GetServingKeyspaces() (go/vt/discovery/keyspace_events.go:763-775) returns all keyspaces that have at least one serving shard, regardless of tablet type:

   func (kew *KeyspaceEventWatcher) GetServingKeyspaces() []string {
       for ksName, state := range kew.keyspaces {
           if state.isServing() {  // Only checks if ANY shard is serving
               servingKeyspaces = append(servingKeyspaces, ksName)
           }
       }
       return servingKeyspaces
   }

2. isServing() (go/vt/discovery/keyspace_events.go:606-616) returns true if any shard has serving: true, without considering tablet type:

   func (kss *keyspaceState) isServing() bool {
       for _, state := range kss.shards {
           if state.serving {
               return true  // Doesn't check tablet type!
           }
       }
       return false
   }

3. AnyKeyspace() (go/vt/vtgate/executorcontext/vcursor_impl.go:600-623) selects the first keyspace from the sorted list:

   func (vc *VCursorImpl) AnyKeyspace() (*vindexes.Keyspace, error) {
       keyspaces := vc.getSortedServingKeyspaces()  // Sorted alphabetically
       for _, ks := range keyspaces {
           if ks.Sharded {
               return ks, nil  // Returns first sharded keyspace
           }
       }
       return keyspaces[0], nil  // Or first keyspace if none sharded
   }

4. SET query execution uses the VCursor's tablet type (REPLICA in this case) when resolving destinations:

   // vcursor_impl.go:1003
   rss, values, err := vc.resolver.ResolveDestinations(ctx, keyspace, vc.tabletType, ids, destinations)

Reproduction Steps

N/A

Binary Version

N/A

Operating System and Environment details

N/A

Log Fragments

N/A

[arthurschreiber]

There's other query types where this bug manifests:

• Dual queries that need tablet execution
• Information schema queries
• Any Route where Routing.Keyspace() returns nil

[arthurschreiber]

The tablet selection for explicit and global routing seems to be differing. When a keyspace is explicitly targeted (e.g. via foobar@replica), then:

• *Resolver.GetKeyspaceShards is called, which returns a SrvKeyspace which is used with topoproto.SrvKeyspaceGetPartition.
• This is then used to resolve destinations for the given query.

Later, during query execution (tabletgateway.go:withRetry)), we try to find all matching tablets for the target (keyspace/shard/tabletType).

This means that for explicit targeting, we never actually look at the keyspace event watcher and whether it thinks is serving or not. Instead the code relies on health check data.

I'm wondering if we should make the AnyKeyspace logic more similar to this.

[mattlord]

/cc @vitessio/query-serving

[Devanshusharma2005]

@arthurschreiber @mattlord I am thinking to use what we already trust. vc.tabletType + ResolveDestinations().
Adding a helper function that does a lightweight resolve for the current tablet type. If it succeeds, the keyspace is eligible, if not, it's filtered out. wdyt ?