Maximum call stack size exceeded during denormalization with large bidirectional entity graphs

[jayseo5953]

Description

We're experiencing RangeError: Maximum call stack size exceeded in GlobalCache.getEntity when denormalizing large datasets (~3000 entities) with bidirectional schema references. The existing same-pk cycle detection doesn't prevent this because the traversal visits different entity instances of different types at each hop.

Reproduction

Schema setup

import { Entity } from '@data-client/rest';

class Department extends Entity {
  id = '';
  name = '';
  buildings: Building[] = [];
  parent: Department | null = null;
  children: Department[] = [];

  pk() { return this.id; }
  static key = 'Department';

  static schema = {
    buildings: [Building],          // Department → Building
    parent: Department,
    children: [Department],
  };
}

class Building extends Entity {
  id = '';
  name = '';
  departments: Department[] = [];
  rooms: Room[] = [];

  pk() { return this.id; }
  static key = 'Building';

  static schema = {
    departments: [Department],      // Building → Department (bidirectional)
    rooms: [Room],
  };
}

class Room extends Entity {
  id = '';
  name = '';
  building: Building | null = null;
  departments: Department[] = [];
  buildings: Building[] = [];

  pk() { return this.id; }
  static key = 'Room';

  static schema = {
    building: Building,
    departments: [Department],      // Room → Department (cross-type)
    buildings: [Building],          // Room → Building (cross-type)
  };
}

What happens

With small datasets (50 departments, 50 buildings), everything works. With large datasets (~3000 departments, ~3000 buildings, each with bidirectional references), useSuspense crashes:

RangeError: Maximum call stack size exceeded
    at GlobalCache.getEntity (@data-client/react)
    at unvisitEntity
    at unvisit
    at denormalize
    ...

Why the existing cycle detection doesn't help

GlobalCache.getEntity tracks visited pks per entity type (cycleCacheKey). It detects when the same pk for the same entity type is revisited. But with bidirectional cross-type references, the traversal visits different pks at each hop:

Department:dept-1 → Building:bldg-1 → Department:dept-2 → Building:bldg-2 → Department:dept-3 → ...

Each pk is unique, so the cycle detection never fires. With 3000 interconnected entities, the recursion depth exceeds the JS call stack (~10,000 frames).

Console log proof

We added depth tracking to GlobalCache.getEntity and captured the denormalization chain at depth 200:

[denorm] depth 200 chain:
[Department]:{"filter...
 → Department:a1b2c3d4
 → Building:e5f6g7h8
 → Department:i9j0k1l2
 → Building:m3n4o5p6
 → Department:q7r8s9t0
 → Building:u1v2w3x4
 → Department:y5z6a7b8
 → ... (200 unique entity pks, still going)

Every entity in the chain has a different pk. The chain alternates between entity types, visiting thousands of unique entities before overflowing.

Key observations

1. It doesn't crash on initial load — only after enough entities from related types accumulate in the cache (e.g., loading a list of buildings, then switching to a list of departments, so both are now in the cache)

2. page[size]: 500 × 5 pages = 2,500 primary entities + ~20,000 included entities in cache. useSuspense denormalizes 500 primary entities synchronously, traversing the full connected graph.

3. The problem is depth, not cycles — even without true cycles, 3000 entities connected via department.buildings → building.departments → ... creates a traversal chain thousands of frames deep through unique pks.

Expected behavior

Denormalization should not overflow the stack regardless of how many entities are in the cache or how they're connected.

Proposed solutions

Option A: Global depth limit

Add a configurable max denormalization depth. Beyond that depth, return entity pks instead of fully resolved entities (similar to how same-pk cycle detection returns early).

class Department extends Entity {
  static schema = {
    buildings: [Building],
  };
  static maxDenormDepth = 3; // Stop resolving after 3 levels
}

Option B: Lazy relationship resolution (like Ember Data)

Support { async: true } or { lazy: true } on schema relationships. Instead of eagerly resolving during denormalization, return a proxy/getter that resolves on first property access:

class Department extends Entity {
  static schema = {
    buildings: { schema: [Building], lazy: true },  // resolved on access, not during denormalize
  };
}

This follows Ember Data's pattern of { async: true } relationships and would fundamentally prevent deep traversal since relationships are only resolved when explicitly accessed.

Environment

• @data-client/react: 0.15.7
• @data-client/rest: 0.15.7
• React: 18
• Browser: Chrome 131+
• Dataset: ~3000 entities per type, bidirectional relationships, JSON:API with include strings

Related

• Entity.denormalize() is not yet written, so denormalizing a nested Resource is not possible #277 — Entity.denormalize() implementation (the initial workaround author noted: "I suspect it might have effects with infinite loops if the child Entity in turn contains a reference to the parent")
• feat: Entity fully denormalizes & Entity -> FlatEntity for non-nested #328 — "Entity fully denormalizes — including recursion" (added full denormalization but cross-type depth wasn't addressed)
• normalizr docs: Add note about renderRestHook() wrapping with provider #195 — Circular reference handling
• normalizr internal(deps): bump stringstream from 0.0.5 to 0.0.6 in /packages/normalizr/examples/redux #216 — Denormalize with recursive linked entities

[ntucker]

Can you provide a sample dataset that reproduces? Perhaps santizied?

[jayseo5953]

@ntucker
Thanks for the incredibly fast turnaround on PR #3823 — the depth limit is a solid pragmatic fix and we'll test it immediately.

Two follow-up ideas that might be worth considering for a more targeted solution:

1. Edge-aware cycle detection (instead of global depth counter)

Rather than counting total depth, track which {entityType, fieldName} relationship edges have been traversed in the current chain. Stop when the same edge is followed a second time:

Department:A → [buildings] → Building:1 → [departments] → Department:B → [buildings] → STOP
                ↑ first Dept→buildings     ↑ first Bldg→depts     ↑ Dept→buildings already crossed

This would:

• Allow the first hop in each direction (full Entity instances)
• Block only the second traversal of the same relationship pattern
• Be more precise than a global depth limit — it wouldn't cut short legitimate deep but non-cyclic paths (like a 20-level department hierarchy via parent/children)
• Cost: O(1) per relationship hop (Set lookup on ${schema.key}.${fieldName})

The depth=128 limit would never trigger for a deep org chart (Department.parent → Department.parent → ... 50 levels deep), but edge-aware would also not trigger because each parent hop is the same entity type and same-pk detection catches it. Edge-aware specifically targets the cross-type zigzag pattern (Dept → Building → Dept → Building) that the current same-pk detection misses.

2. Lazy relationship resolution (longer-term)

The deeper question is: why resolve relationships that nobody reads? In our case, useSuspense denormalizes 500 entities × all their relationships recursively, but the grid only renders ~20 visible rows and reads .name, .state, .sidc.

Something like Ember Data's { async: true } pattern — where Entity.denormalize() defines getters that resolve on first property access instead of eagerly calling unvisit() — would make the traversal depth driven by what the component actually reads, not by the schema graph.

We prototyped this as a consumer-side override of Entity.denormalize() in our base class. It works for the stack overflow fix, but we're concerned about memoization/dependency tracking — lazy getters skip the initial dependency registration in GlobalCache, which could affect cache invalidation for entities whose getters haven't been accessed yet.

Would either of these be worth exploring as follow-up improvements? Happy to contribute if helpful.

---

Re: sample dataset — I'll put together a sanitized reproduction script that generates the bidirectional Department/Building/Room graph with ~3000 entities and triggers the overflow via useSuspense. Will post it as a follow-up.

[jayseo5953]

Thanks for the quick fix! The depth limit solves our immediate crash.

Just wanted to follow up on the two ideas from my earlier comment — edge-aware cycle detection and lazy relationship resolution. Are either of these worth exploring as separate issues/discussions, or is the depth limit the intended long-term solution?

Also — when can we expect this to land in a release? We're on @data-client/react 0.15.7 and would love to pick it up.

[ntucker]

Edge-aware cycle detection doesn't seem safe, as there are many legitimate use cases to do this that don't explode.

Lazy is interesting, but since react patterns rely on immutable objects, a proxy mechanism won't work well. So it needs to be combined with an explicit viewing mechanism - a hook.

Lazy + useQuery()

import { Entity, Lazy } from '@data-client/rest';

class Department extends Entity {
  buildings: string[] = [];
  static schema = {
    buildings: new Lazy([Building]),
  };
}

// Parent denormalization: dept.buildings = ['bldg-1', 'bldg-2'] (raw PKs, not resolved)
const dept = useSuspense(DepartmentResource.get, { id: '1' });

// Array case: pass raw IDs directly — they're already available from the parent
const buildings = useQuery(Department.schema.buildings.query, dept.buildings);
// buildings: Building[] | undefined

class Department extends Entity {
  mainBuilding: string = '';
  static schema = {
    mainBuilding: new Lazy(Building),
  };
}

// Entity case: use the inner Entity's own args lookup
const building = useQuery(Department.schema.mainBuilding.query, { id: dept.mainBuilding });

class Department extends Entity {
  static schema = {
    buildings: new Lazy(buildingsCollection),
  };
}

// Collection case: use Collection's own args/queryKey
const buildings = useQuery(Department.schema.buildings.query, ...collectionArgs);

[jayseo5953]

Testing PR #3823 with ~20k bidirectional entities

@ntucker

We patched @data-client/normalizr@0.15.4 with the exact changes from this PR and tested against our production-scale dataset.

Observations:

1. The depth limit of 128 fires correctly — we see Entity depth limit of 128 reached for "Building" entity in the console.

2. The page still crashes, but with a different failure mode. Without the patch, we get RangeError: Maximum call stack size exceeded. With the patch at 128, the stack overflow is prevented but Chrome runs out of memory (Aw Snap, error code 5) as denormalization resolves and caches entity objects 128 levels deep across ~20k entities.

3. Each entity hop adds ~10 stack frames to the call stack (unvisit → arrayDenormalize → unvisit → Entity.denormalize → unvisitEntityObject → computeValue → getEntity → unvisitEntity → unvisit). At depth 128, that's ~1,280 frames.

4. With 10,500+ unique entities where every type back-references every other type, the denormalization visits different PKs at each hop (no cache hit), sustaining chains of 128+ unique entities easily.

---

Follow up

Lazy resolution seems like a more scalable/robust approach for this class of problem in my opinion. We experimented with overriding Entity.denormalize() to defer relationship resolution using ES5 getters — relationships resolve on first property access instead of during the denormalization pass:

class ApplicationEntity extends Entity {
  static denormalize(input, args, unvisit) {
    if (typeof input === 'symbol') return input;
    for (const key of Object.keys(this.schema)) {
      const schema = this.schema[key];
      if (!isEntityRelationship(schema)) {
        // Transform functions (dates, enums) — run eagerly
        const value = unvisit(schema, input[key]);
        input[key] = typeof value === 'symbol' ? undefined : value;
        continue;
      }
      // Entity relationships — defer to first access
      const rawValue = input[key];
      Object.defineProperty(input, key, {
        get() {
          const resolved = unvisit(schema, rawValue);
          Object.defineProperty(this, key, {
            value: typeof resolved === 'symbol' ? undefined : resolved,
            writable: true, configurable: true, enumerable: true,
          });
          return this[key];
        },
        configurable: true, enumerable: true,
      });
    }
    return input;
  }
}

This eliminated the crash entirely — only ~18% of relationships were actually accessed by the UI. However, unvisit also registers dependencies for cache invalidation during denormalization, not just entity resolution. Lazy getters bypass that registration since they fire after the dependency list is finalized — resulting in stale UI when related entities change.

I initially expected relationship dependencies to be tracked at normalization time (since the relationship graph is already known), but they're tracked at denormalization time — per query result, not per entity. This makes sense as a design decision since different queries traverse different subsets of the graph.

For lazy resolution to work, dependency registration would need to be decoupled from entity resolution — registering dependencies eagerly (shallow walk of schema fields) while deferring actual Entity construction to property access.

[ntucker]

@jayseo5953 do these need to be normalized?

[ntucker]

@jayseo5953 It might be best to move the 'lazy' to a new discussion in https://github.com/reactive/data-client/discussions so we can consider this design fully. For the memory issue probably just decreasing the depth is a good idea as 128 is already pretty ridiculous. Perhaps 32 is a better number.

[jayseo5953]

@ntucker sounds good I will create a new one.

but can you make that configurable on the Entity/dataprovider instead of hard coding internally?

Thanks for your prompt response!

Btw I am also in the discord channel if you'd prefer to further discuss there. we could

[jayseo5953]

@ntucker thank you for the suggestion.

But then we would have to remember which relationship is lazy and which is not and have to change the implementation to use useQuery instead of useSuspense alone.

I created a new discussion here and explained what we tried to do.
#3828 (comment)