Use timer coalescing instead of globalThis.setTimeout for GC to avoid performance issues

[justjake]

I'm investigating adopting tanstack/query in a large production application called Notion. We need to support tens of thousands of queries on screen at once. I'm concerned that the current implementation of query GC uses individual setTimeout per query. This means after a navigation, tanstack/query-core would call setTimeout tens of thousands of times, which can cause a lot of event loop delay.

Our solution to this issue with our current internal system is to use a very simple timer coalescing with a scheduler, so instead of 30k setTimeout calls, we have either 0 or 1 calls to setInterval. It would be great to update Removable#scheduleGc to do its scheduling via something like TimeoutGroup, similar to how the system uses notifyManager. Pseudocode below:

type PendingTimeout = {
  id: number
  deadline: number
  callback: () => void
}

class TimeoutGroup {
  constructor(private wakeupIntervalMs: number) {}
  private interval: typeof ReturnType<setInterval> | undefined
  private timeouts = new Map<number, PendingTimeout>
  private nextId = 1

  setTimeout(callback: () => void, ms: number) {
    const id = this.nextId++
    this.timeouts.set(id, { id, callback, deadline: ms + Date.now() })
    this.schedule()
    return id
  }

  clearTimeout(id: number) {
    try {
      return this.timeouts.delete(id)
    } finally {
      this.unschedule()
    }
  }

  flushPendingTimeouts() {
    const now = Date.now()
    for (const timer of this.timeouts.values()) {
      if (timer.deadline <= now) { this.flushTimeout(timer) }
    }
    this.unschedule()
  }

  private schedule() {
    if (!this.timeouts.size) { return }
    this.interval ??= setInterval(() => this.flushPendingTimeouts(), this.wakeupIntervalMs)
  }

  private unschedule() {
    if (this.timeouts.size || !this.interval) { return }
    clearInterval(this.interval)
    this.interval = undefined
  }

  private flushTimeout(timer: PendingTimeout) {
    this.timeouts.delete(timer.id)
    try {
      timer.callback()
    } catch (error) {
      Promise.resolve().then(() => { throw error })
    }
  }
}

to avoid huge event loop delay after navigation unmounts a bunch of QueryObservers.

[justjake]

I looked around for other open-source libraries and found timer-wheel on npm which implements an expiry data structure that appears high quality and fulfills our needs. We could implement a TimoutManager (similar to NotifyManager and OnlineManager) using different strategies based on the accuracy needs of the caller:

• Very small timeouts (<5s) continue using native setTimeout
• Small timeouts (< 60s) use a simple strategy and fire with 10s granularity
• Long timeouts (>= 60s) use TimerWheel and fire with 30s granularity
• We could increase or decrease the resolution of the long-timeout setInterval depending on the resolution of the remaining timeouts in the bucket. If they're all >60 minutes away, can switch to a 30 minute interval.

[TkDodo]

We need to support tens of thousands of queries on screen at once

I don’t want to discourage you from investigating react-query but I think this huge amount of queries is not something we’ve optimized for. After about 1k observers, staleTime timers alone are really troublesome, as we kick them off per-observer. So gcTime might be the least of your problems 😅.

But, if the TimeoutGroup could also reduce the amount of timers we trigger for staleTime, we could think about adopting it, but we have to keep bundle size in mind.

What you can try is to opt out of our own garbage collection by setting gcTime: Infinity globally (this won’t start any timers), and then do your own garbage collecting with a global cache subscription that listens to the observerRemoved event:

queryCache.subscribe(event => {
  if (event.type === 'observerRemoved' && event.query.getObserversCount() === 0) {
    // last observer was removed
    // trigger your own gc logic here
  }
})

Then, when you want to remove the query, call queryClient.removeQueries(...)

[TkDodo]

But I also really want to know why you’d need that many queries. Would every block / word in notion be its own query? That doesn’t sound like a good idea...