perf(auth): opt-in in-memory cache for internal-secret account context

[pfreixes]

Note

Drafted pending #6800 (light internal auth context query for persist). Once that merges, this PR will be reworked so the cache wraps getInternalAuthContext and stores the narrow context (smaller entries, no secrets retained) instead of the full one.

• Adds an opt-in, per-process TTL cache in front of the internal-secret account-context lookup (persist/runner auth). This lookup runs an uncached 5-table join per request — ~3.3k req/s in production, roughly a third of all main-DB queries, re-authenticating the same small set of internal keys. With the cache on, that should drop to roughly one query per environment per TTL window per pod.
• New TTLFixedSizeMap in @nangohq/utils: bounded size (composes FixedSizeMap) plus TTL expiry on a monotonic clock, expired entries evicted on read.
• Rollout is config-driven and ships inert: AUTH_ACCOUNT_CONTEXT_CACHE_MODE defaults to off; dry records hit/miss metrics (nango.auth.accountContextCache) while every lookup still goes to the DB; on serves cached hits. AUTH_ACCOUNT_CONTEXT_CACHE_TTL_MS (default 60s) bounds staleness.

Staleness contract

With mode on, everything in the cached context — rotated/revoked internal keys, revoked accounts, plan changes, account metadata — keeps resolving for up to the TTL on each pod. The TTL should stay very short while still effective. We expect a non-negligible hit ratio (misses scale with environments × pods per TTL window rather than with request rate), which will be measured while the cache runs in dry mode before we serve from it. Only keys that previously passed a successful lookup can ever be served from cache.

Test plan

• Unit tests for TTLFixedSizeMap (TTL expiry, reset on rewrite, size-limit eviction, non-integer TTL)
• Integration tests for all three modes: serves from cache within TTL, refetches after TTL, dry mode records hits but stays on the DB path, off mode never touches the cache
• Verified the serve-from-cache test goes red when the cache is disabled
• Deploy with default off — confirm no behavior change and no nango.auth.accountContextCache metric
• Flip dry in production — measure hit ratio for a day
• Flip on — confirm the api_secrets join rate and nango-production-writer CPU drop

🤖 Generated with Claude Code

[cubic-dev-ai]

1 issue found across 6 files

Confidence score: 4/5

• In packages/shared/lib/services/account.service.ts, cache-expiry misses can trigger parallel expensive lookups (a stampede), which may spike database load and increase latency under hot-key traffic; add per-key in-flight request coalescing (e.g., store and await a shared promise) before merging to de-risk peak behavior.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/shared/lib/services/account.service.ts">

<violation number="1" location="packages/shared/lib/services/account.service.ts:662">
P2: Hot keys can stampede the database at cache expiry: concurrent misses all start the expensive lookup before any request reaches `set`. Track an in-flight lookup per hash (or cache a promise) so callers coalesce behind the first query.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Hot keys can stampede the database at cache expiry: concurrent misses all start the expensive lookup before any request reaches set. Track an in-flight lookup per hash (or cache a promise) so callers coalesce behind the first query.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/shared/lib/services/account.service.ts, line 662:

<comment>Hot keys can stampede the database at cache expiry: concurrent misses all start the expensive lookup before any request reaches `set`. Track an in-flight lookup per hash (or cache a promise) so callers coalesce behind the first query.</comment>

<file context>
@@ -646,6 +657,14 @@ class AccountService {
     private async getAccountContextByInternalSecret(secretKey: string): Promise<AccountContext | null> {
+        const cacheMode = envs.AUTH_ACCOUNT_CONTEXT_CACHE_MODE;
+        if (cacheMode !== 'off') {
+            const cached = getCachedAccountContext({ hash: hashLocalCache.get(secretKey), mode: cacheMode });
+            if (cached !== undefined && cacheMode === 'on') {
+                return cached;
</file context>

[pfreixes]

Quantified during review: at expiry, concurrent misses during the few-ms DB roundtrip amount to ~1–2 duplicate lookups per key per TTL window per pod — against a baseline where every request runs the query today. Coalescing would also need to be mode-conditional (dry must keep querying per request to stay observe-only), which isn't worth the complexity at a 60s TTL. If dry-mode metrics surface hot keys where this matters, in-flight dedup is a contained follow-up — same pattern as inFlightRefreshes in refresh.ts.

[pfreixes]

To make the worst case explicit: during a cache-miss stampede, an environment generates at most the same query traffic it generates today all the time. So the current approach adds no regression in terms of worst-case scenario — at most it leaves it at the same level, but contained at env level and to this specific expiry event.

[pfreixes]

And if we ever want to work on the stampede-miss side effect, it's an easy follow-up: either guard rails to reduce the blast radius, or — even simpler, mimicking what Memcached does — a new attribute on the entry that forces a refresh for one single request when we're approaching the TTL, while everyone else keeps being served the still-fresh value. That removes the expiry miss burst entirely without any in-flight promise tracking.