GVL Startup Fetch Creates Thundering Herd at Scale — Need Cluster-Wide Caching

[scr-oath]

Problem

Every Prebid Server pod fetches the entire Global Vendor List history from vendor-list.consensu.org at startup. As of today, that's ~369 sequential HTTP requests per pod (TCF v2: 223 versions, TCF v3: 146 versions). Each pod builds its own in-memory cache from scratch — nothing is shared across pods, and the upstream cache-control: max-age=604800 (7 days) is not leveraged.

This creates a classic Thundering Herd problem during Kubernetes rollouts where many pods start simultaneously.

Impact — Back-of-Napkin Numbers

Deployment Size	Pods	GVL Fetches/Pod	Requests per Rollout	Requests/Week (daily deploys)
Small (20 pods)	20	~369	~7,400	~52,000
Medium (50 pods)	50	~369	~18,500	~130,000
Large (100 pods)	100	~369	~36,900	~258,000

With caching honored (7-day TTL), the same data would require only ~369 requests per week regardless of fleet size — a 99%+ reduction.

Observed Failures

• Bursts of concurrent requests to consensu.org (fronted by CloudFront) trigger transient failures (HTTP errors, timeouts, rate-limiting).
• Even one failure during the init window can leave a pod without a complete GVL, causing GDPR consent processing errors (see: "Cookie syncs may be affected" warnings in logs).
• In Kubernetes, a pod that can't process consent correctly may fail health checks and restart — creating a cascading restart loop that amplifies the herd further.
• Deployment reliability becomes coupled to an external third-party CDN's ability to absorb bursty traffic — a fragile dependency for production rollouts.

The data is highly cacheable

$ curl -I https://vendor-list.consensu.org/v2/vendor-list.json
cache-control: max-age=604800
x-cache: Hit from cloudfront

• Archived versions (e.g., v2/archives/vendor-list-v100.json) are immutable — they never change.
• Only the "latest" endpoint updates, roughly weekly.
• Yet today, every pod re-fetches all ~369 URLs from the origin on every startup.

What's Needed

A way to cache GVL data once for the cluster so that only the first requester fetches from origin, and all subsequent pods (and restarts) are served from a local, cluster-internal cache.

---

Technical Context

Current Implementation

In gdpr/vendorlist-fetching.go:

• preloadCache() loops over TCF v2 and v3, fetching every archived version sequentially via saveOne().
• saveOne() makes a plain http.GET — no retry, no backoff, no shared caching.
• VendorListURLMaker() is hardcoded to https://vendor-list.consensu.org/... with no configuration to override the base URL.
• The in-memory cache (sync.Map) is per-process only — lost on restart, not shared.

Proposed Solution: Cooperative Caching via Prebid Cache

Prebid Cache is already deployed as a cluster-wide microservice alongside Prebid Server in most production setups. It has mature storage backends (Redis, Aerospike, Memcache, etc.) and is owned by the Prebid project.

Proposal: Add a GVL caching endpoint to Prebid Cache that:

1. Exposes a GVL-compatible URL path (e.g., /gvl/v2/vendor-list.json, /gvl/v3/archives/vendor-list-v100.json) that Prebid Server can be configured to use instead of vendor-list.consensu.org.
2. Fetches from origin on cache miss, stores in its backend (Redis, etc.), and serves subsequent requests from cache.
3. Respects cache-control / TTL — archived versions cached indefinitely (immutable); latest version cached for up to 7 days per upstream headers (a library like pquerna/cachecontrol could help here).
4. Deduplicates concurrent origin fetches (singleflight pattern) to prevent the cache itself from herding against the origin.

On the Prebid Server side:

5. Make the GVL base URL configurable — e.g., a new config parameter gdpr.vendorlist_base_url (default: https://vendor-list.consensu.org) that can be pointed at the local Prebid Cache instance.

Why Prebid Cache?

• Already deployed: Most PBS operators already run Prebid Cache as a cluster-internal service — no new infrastructure needed.
• Both are Prebid-owned: This is a natural cooperation between two projects in the same ecosystem.
• Proven storage backends: Redis/Aerospike/Memcache already handle TTL-based caching at scale.
• Simple integration: PBS only needs a configurable base URL; all the caching intelligence lives in Prebid Cache.

Result

Scenario	External Requests to consensu.org
Today (100 pods, daily deploys)	~258,000/week
With cluster cache	~369/week (one cache fill)
Reduction	~99.9%

This eliminates the Thundering Herd, decouples deployment reliability from a third-party CDN, and is architecturally clean — leveraging infrastructure that already exists in the Prebid ecosystem.

Related Issues

• GDPR Endpoint Logic #504 — Original GVL fetching implementation
• Change GVL URL #1632 — Request to change GVL URL
• Failed to fetch Vendor-List (context deadline exceeded) #1687 — "Failed to fetch Vendor-List (context deadline exceeded)"

[bsardo]

We have the following questions for the IAB:

• Can all of the GVL versions be bundled so that we only need to make a single request per cluster?
• Is there a rate limit?

Discussed in the backlog meeting. This proposal looks reasonable and is an optional feature. @Net-burst, @SyntaxNode do you care to weigh in?

[Net-burst]

PBS-Java does both caching, fetch backoff, and sharing across a mutual Docker volume by default.

[bsardo]

@Net-burst can you add more detail here on Java's approach?

[Taxel]

After thinking about this some more, we are actually very interested in this (although we are yet to experience problems with the thundering herd). Reading from a shared volume sounds like a solid approach.

Proposed solution:

• Cluster Fetcher: A single fetcher per cluster (a separate, very small service) periodically checks the current vendor list, and keeps the files in a mounted RWX (ReadWriteMany) volume up-to-date
• Atomic Updates: The fetcher uses os.Rename to move the files into the final directory to keep updates atomic (assuming POSIX compliance on the volume)
• Manifest File: It writes an index file to the directory (index.json or manifest.json), which contains a list of all the downloaded vendor list files. This file is strictly updated last, after new files have been added.
• Cheap PBS Updates: On the PBS side, this enables extremely cheap update checks. We just need to keep the mtime of that index.json saved, and os.Stat() it at a configurable interval. When changed, we read newly detected files into the same sync.Map used currently.
• Startup Jitter: To avoid a secondary thundering herd problem on startup (this time hammering the shared volume's metadata server), we should add some startup jitter to the initial read.

Tradeoffs

• Extra Infrastructure. This requires deploying and maintaining an additional service
• RWX Complexity: RWX volumes in Kubernetes can be finicky. If the volume is not bound/schedulable when PBS starts, the pods will fail to start.
• HTTP Caching Proxies exist: Since prebid-cache was ruled out for this type of data, simply making the URI configurable in PBS and letting operators use a standard HTTP proxy (like NGINX with proxy_cache_lock on) would technically be simpler for those without RWX volumes.

On the other hand, we completely eliminate external HTTP calls to GVL from the PBS pods, and the initial thundering herd problem is cleanly solved.

Open questions

• Cold Starts: How do we want to handle scenarios where no files can be fetched yet? (e.g. a freshly provisioned volume where the cluster-fetcher hasn't completed its first run).
  • My proposal: We wire this into the status endpoint (readiness probe) so that the pod does not report as ready until a GVL fetch from the volume has succeeded at least once

What do you think @scr-oath - would that solve your problems? We are considering something similar for loading AccountConfig from a shared volume, so all the risks concerning the shared volume itself are effectively void for AY.

[bsardo]

Discussed at the backlog meeting. There are several ways to solve this problem but for the sake of simplicity we should look at making PBS-Go support a shared volume for storing fetched GVLs. I believe it is currently stored in RAM.

Java supports a shared volume. When a consent string referencing a given version is encountered and the GVL is available on the shared volume, it pulls it into RAM from disk. If not available, it will fetch it and stored it on the shared volume.

Java team acknowledged that there will be a lot of errors due to contention when a new GVL goes live with several pods attempting to fetch the GVLs but the impact is negligible.

There was also a suggestion to consider whether https://pkg.go.dev/golang.org/x/sync/singleflight would help here. Addresses case where multiple go routines are attempting to fetch at the same time and this package would help, particularly in a startup scenario but might not address our use case where we are fetching as GVLs first go live.

An alternative approach is adding a reverse proxy but that would not be baked into PBS. We can add a config option so that an alternative URL can be used to retrieve the GVLs to support this approach.

[bsardo]

@Net-burst can you point us to where the Java code lives?

[Net-burst]

The logic is somewhat split between different classes. But the following 2 should cover everything that is needed in this context:

• https://github.com/prebid/prebid-server-java/blob/master/src/main/java/org/prebid/server/privacy/gdpr/vendorlist/VendorListService.java
• https://github.com/prebid/prebid-server-java/blob/master/src/main/java/org/prebid/server/privacy/gdpr/vendorlist/VendorListFetchThrottler.java

[bsardo]

@Taxel is interested in contributing this feature.