FCP-1434: Add resource-level wildcard support for snapshot cache #33

mohit-pal-kohli · 2025-12-03T20:43:06Z

Part 1 of implementing explicit wildcard support for xds-control-plane.

Context

We leverage the Snapshot Cache in go-control-plane, where delta for a node ID snapshot is proliferated to Envoys with that node ID when there is a change. This is acceptable in the current state where Fabric Proxy declares the destinations (clusters) it needs at start time in YAML.

The introduction of ODCDS makes the current state unpalatable, even though it is functional. The issue is that we do not want a mass proliferation of new clusters to all Envoys with node ID X when one pod with that node ID X requests a resource. Not solving this could create significant load on our control plane, especially for Fabric Proxy users with hundreds or thousands of destinations.

To solve this, we should add support for resource-level wildcard support for control plane. By default, a snapshot resource will be considered wildcard (OnDemandOnly = false) and proliferated to all Envoys with the relevant node ID. By contrast, we will allow certain resources to not be wildcard, allowing ODCDS clusters to only be proliferated on demand.

Implementation details

This PR aims to make changes in a way that are fully backward compatible for Delta and SoTW. Users using the deprecated cache will have all resources default to wildcard-eligible. Users using the new cache will be able to specify OnDemandOnly. If not specified, then OnDemandOnly=false, meaning that the resource is wildcard-eligible.

snapshot, _ := types.NewSnapshot("v1", map[string][]types.SnapshotResource{
    resource.ClusterType: {
        // Wildcard-eligible: Default case
        {
            Name: "core-service",
            Resource: &cluster.Cluster{Name: "core-service"},
        },
        // Wildcard-eligible: Explicitly defined
        {
            Name: "core-service",
            Resource: &cluster.Cluster{Name: "core-service"},
            OnDemandOnly: false,
        },
        // On-demand only: sent only when explicitly requested
        {
            Name: "analytics-cluster",
            Resource: &cluster.Cluster{Name: "analytics-cluster"},
            OnDemandOnly: true,
        },
    },
})

pkg/cache/types/snapshot.go

pkg/cache/v3/simple.go

pkg/cache/types/snapshot.go

pkg/cache/internal/cached_resource.go

zhiyanfoo

LGTM mostly some nits

zhiyanfoo · 2025-12-05T16:22:23Z

pkg/cache/internal/cached_resource.go

 	// cacheVersion is the version of the cache at the time of last update, used in sotw.
 	cacheVersion string

+	// onDemandOnly indicates if this resource is only sent when explicitly requested (ODCDS support).


I would remove reference to the ODCDS part. As Valerian mentioned this could apply to Listeners as well.

Suggested change

// onDemandOnly indicates if this resource is only sent when explicitly requested (ODCDS support).

// onDemandOnly indicates if this resource is only sent when explicitly requested: for example when performing ODCDS (on-demand cluster discovery).

That's fair - Making it more use case agnostic is probably better for when we upstream this.

zhiyanfoo · 2025-12-05T16:23:07Z

pkg/cache/internal/cached_resource.go

 	return func(r *CachedResource) { r.ttl = ttl }
 }

+// OnDemandOnly marks the resource as on-demand only (for ODCDS support).


Suggested change

// OnDemandOnly marks the resource as on-demand only (for ODCDS support).

// OnDemandOnly marks the resource as on-demand only

pkg/cache/types/snapshot.go

zhiyanfoo · 2025-12-05T16:24:18Z

pkg/cache/v3/simple.go

 				resourcesToReturn = make(map[string]*internal.CachedResource, len(resources))
 				for _, res := range resources {
-					addResource(res)
+					// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)


Suggested change

// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)

// Include wildcard-eligible resources for wildcard subscriptions

zhiyanfoo · 2025-12-05T16:24:38Z

pkg/cache/v3/simple.go

-				_, known := knownResources[name]
-				if known {
+			for name, res := range resources {
+				// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)


Suggested change

// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)

// Include wildcard-eligible resources for wildcard subscriptions

zhiyanfoo · 2025-12-05T17:04:43Z

pkg/cache/v3/simple.go

 		}

-		if watch.subscription.IsWildcard() {
+		if watch.subscription.IsWildcard() && !res.OnDemandOnly() {


Suggested change

if watch.subscription.IsWildcard() && !res.OnDemandOnly() {

// the only case where we can keep a resource if it's a not explicitly subscribed is when we are a wildcard subscription and the resource is also a wildcard resource.

if watch.subscription.IsWildcard() && !res.OnDemandOnly() {

zhiyanfoo · 2025-12-05T17:05:16Z

pkg/cache/v3/simple.go

 		for _, res := range resources {
-			if err := addIfChanged(res); err != nil {
-				return nil, fmt.Errorf("failed to compute resource version for %s: %w", res.Name, err)
+			// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)


Suggested change

// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)

// Include wildcard-eligible resources for wildcard subscriptions

zhiyanfoo · 2025-12-05T17:05:28Z

pkg/cache/v3/simple.go

 			for name, resource := range resources {
-				resourcesToReturn = append(resourcesToReturn, resource)
-				returnedResources[name] = version
+				// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)


Suggested change

// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)

// Include wildcard-eligible resources for wildcard subscriptions

zhiyanfoo · 2025-12-05T17:07:21Z

pkg/cache/v3/simple.go

 				continue
 			}

 			if watch.subscription.IsWildcard() {


I think we can do the same pattern here as for the delta wildcard requests

Suggested change

if watch.subscription.IsWildcard() {

// the only case where we can keep a resource if it's a not explicitly subscribed is when we are a wildcard subscription and the resource is also a wildcard resource.

if watch.subscription.IsWildcard() && !res.OnDemandOnly() {

Nice catch, updating

valerian-roche · 2025-12-07T04:42:29Z

@mohit-pal-kohli can you rebase the PR on the upstream branches and confirm when tested?
I raised envoyproxy#1356 for the first branch

mohit-pal-kohli self-assigned this Dec 3, 2025

mohit-pal-kohli requested review from a team and valerian-roche December 3, 2025 20:52

valerian-roche reviewed Dec 4, 2025

View reviewed changes

mohit-pal-kohli requested a review from a team December 4, 2025 21:44

zhiyanfoo approved these changes Dec 5, 2025

View reviewed changes

mohit-pal-kohli added 6 commits December 5, 2025 14:16

Add resource-level wildcard support for snapshot cache

bb804fb

Nit

4d50bc4

Remove unnecessary block

36f4fd2

Review comments and naming

ade308d

nestif fixes

8787025

review

988f432

mohit-pal-kohli force-pushed the mohit/snapshot-v2-work branch from 801d766 to 988f432 Compare December 5, 2025 19:45

mohit-pal-kohli changed the base branch from vr/snapshot-v2 to vr/delta-resource December 5, 2025 19:46

mohit-pal-kohli changed the base branch from vr/delta-resource to main December 22, 2025 18:28

mohit-pal-kohli merged commit 611973d into main Dec 22, 2025
4 checks passed

	// onDemandOnly indicates if this resource is only sent when explicitly requested (ODCDS support).
	// onDemandOnly indicates if this resource is only sent when explicitly requested: for example when performing ODCDS (on-demand cluster discovery).

	// OnDemandOnly marks the resource as on-demand only (for ODCDS support).
	// OnDemandOnly marks the resource as on-demand only

	// Include wildcard-eligible resources for wildcard subscriptions (ODCDS support)
	// Include wildcard-eligible resources for wildcard subscriptions

	if watch.subscription.IsWildcard() && !res.OnDemandOnly() {
	// the only case where we can keep a resource if it's a not explicitly subscribed is when we are a wildcard subscription and the resource is also a wildcard resource.
	if watch.subscription.IsWildcard() && !res.OnDemandOnly() {

	if watch.subscription.IsWildcard() {
	// the only case where we can keep a resource if it's a not explicitly subscribed is when we are a wildcard subscription and the resource is also a wildcard resource.
	if watch.subscription.IsWildcard() && !res.OnDemandOnly() {

FCP-1434: Add resource-level wildcard support for snapshot cache #33

FCP-1434: Add resource-level wildcard support for snapshot cache #33

Uh oh!

Conversation

mohit-pal-kohli commented Dec 3, 2025

Context

Implementation details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhiyanfoo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valerian-roche commented Dec 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

valerian-roche commented Dec 7, 2025 •

edited

Loading