Adaptive JIT: ELU-aware background query compilation

[mcollina]

Problem

The current jit option is a simple counter threshold: compile a query after it has been seen N times. This has two issues:

1. Under load: JIT compilation is synchronous and blocks the event loop. Compiling during a traffic spike adds latency to the request that triggers compilation and to all concurrent requests waiting behind it.
2. Fixed threshold: jit: 1 compiles too eagerly (pays cost on 2nd hit), while jit: 100 delays the benefit. There's no universal correct value.

Benchmarks

We benchmarked 1000 unique queries sent twice each (so the 2nd hit triggers compileQuery with jit: 1):

Server	Cold ms/req	Warm ms/req	Overhead
fastify-mercurius	8.46	7.84	7.8%
fastify-mercurius-jit	9.40	6.18	52.1%

The 2nd execution of each unique query pays ~3ms for synchronous compilation, creating a 52% overhead vs warm cached queries. With jit: 1 and only 1 hit per query, JIT never fires (zero cost), but as soon as queries repeat, the compilation cost hits the request path.

Proposal: ELU-gated background JIT

Use Node.js Event Loop Utilization (ELU) to decide when to compile, and query frequency to decide what to compile.

Flow

Request arrives
     │
Query in LRU? ──no──► parse, validate, store with count=1, jit=null
     │
    yes
     │
cached.count++
     │
jit compiled? ──yes──► use jit.query()
     │
    no
     │
count >= minCount? ──no──► use graphql execute()
     │
    yes
     │
add to compilation queue (if not already queued)
     │
use graphql execute() for now


Background compilation loop (setImmediate / setTimeout)
     │
ELU < threshold? ──no──► back off (setTimeout 50ms)
     │
    yes
     │
dequeue highest-count query
     │
compileQuery()
     │
cached.jit = result
     │
more in queue? ──yes──► schedule next tick
     │
    no ──► idle

API

app.register(mercurius, {
  schema: typeDefs,
  resolvers,
  jit: {
    // Minimum hit count before a query becomes a compilation candidate.
    // Unlike today's jit option, reaching this count does NOT block the
    // request — it enqueues the query for background compilation.
    minCount: 3,

    // ELU threshold (0-1). Compilation only proceeds when the current
    // ELU is below this value. Default 0.8 = compile when event loop
    // is less than 80% utilized.
    eluThreshold: 0.8,

    // Maximum number of queries to compile per tick. Limits how long
    // the event loop is blocked per compilation batch.
    maxCompilePerTick: 1,

    // Optional: maximum queue size. If the queue is full, the least
    // popular candidate is dropped.
    maxQueueSize: 100,
  },
})

Backwards compatible: jit: 1 / jit: 0 keeps working as today.

Integration with Mercurius

The change to the request handler is minimal:

// Before (current):
const shouldCompileJit = cached && cached.count++ === minJit
// ...
if (shouldCompileJit) {
  cached.jit = compileQuery(schema, document, operationName)  // blocks!
}

// After (adaptive):
cached && cached.count++
adaptiveJit.maybeEnqueue(cached, document, operationName)
// compilation happens in background — this request uses execute()
// next request will find cached.jit populated and use the fast path

Reference implementation

import { performance } from 'node:perf_hooks'
import { compileQuery, isCompiledQuery } from 'graphql-jit'

function createAdaptiveJit (schema, opts) {
  const minCount = opts.minCount ?? 3
  const eluThreshold = opts.eluThreshold ?? 0.8
  const maxCompilePerTick = opts.maxCompilePerTick ?? 1
  const maxQueueSize = opts.maxQueueSize ?? 100

  const queue = []
  let compiling = false
  let elu1 = performance.eventLoopUtilization()

  function getELU () {
    const elu2 = performance.eventLoopUtilization(elu1)
    elu1 = performance.eventLoopUtilization()
    return elu2.utilization
  }

  function enqueue (cached, document, operationName) {
    if (cached.jitQueued) return
    cached.jitQueued = true

    const entry = { cached, document, operationName }
    let i = queue.findIndex((e) => e.cached.count < cached.count)
    if (i === -1) i = queue.length
    queue.splice(i, 0, entry)

    if (queue.length > maxQueueSize) {
      const dropped = queue.pop()
      dropped.cached.jitQueued = false
    }

    scheduleCompilation()
  }

  function scheduleCompilation () {
    if (compiling || queue.length === 0) return
    compiling = true
    setImmediate(compileTick)
  }

  function compileTick () {
    const utilization = getELU()

    if (utilization >= eluThreshold) {
      setTimeout(compileTick, 50)
      return
    }

    let compiled = 0
    while (queue.length > 0 && compiled < maxCompilePerTick) {
      const { cached, document, operationName } = queue.shift()
      cached.jit = compileQuery(schema, document, operationName)
      cached.jitQueued = false
      compiled++
    }

    if (queue.length > 0) {
      setImmediate(compileTick)
    } else {
      compiling = false
    }
  }

  function maybeEnqueue (cached, document, operationName) {
    if (cached.jit !== null) return
    if (cached.count < minCount) return
    enqueue(cached, document, operationName)
  }

  return { maybeEnqueue }
}

Behavior under different load patterns

Scenario	Behavior
Steady traffic, low load (ELU ~30%)	Queries compile almost immediately after reaching minCount. Effectively jit: 3 but non-blocking.
Traffic spike (ELU ~90%)	Compilation deferred. All requests use execute(). When load drops, most popular queries compile first. No added latency during the spike.
Many unique queries (gateway, no persisted queries)	Most queries never reach minCount — never compiled, zero overhead. Same as jit: 0 for one-off queries.
Cold start (deploy, restart)	All queries start at count=0. Popular queries compile in background as traffic ramps. No thundering herd of synchronous compilations.

Why setImmediate and not a worker thread

compileQuery is synchronous and returns V8 heap objects (functions). These can't be serialized across the thread boundary, so worker threads aren't an option. setImmediate yields to I/O between compilations; combined with the ELU gate, it ensures compilation only happens when there's headroom.

Why prioritize by count

Under sustained load with many candidate queries, we want to compile the queries that give the most benefit first. A query seen 1000 times will save more total CPU than one seen 5 times. The priority queue ensures the most impactful queries are compiled first when ELU headroom is limited.

---

Benchmarks and profiling data from https://github.com/platformatic/graphql-benchmarks

[mcollina]

Follow-up: Enable adaptive JIT by default

With the adaptive JIT implementation in #1224, there's no longer a downside to enabling JIT by default:

• Cold/unique queries: never reach minCount, zero compilation overhead — identical to jit: 0
• Hot queries: compiled in the background when the event loop is idle — no blocking
• Under load: compilation defers automatically when ELU exceeds the threshold

The benchmarks in #1224 confirm that adaptive JIT matches sync JIT throughput on hot queries while causing significantly less interference when cold queries arrive (+44.6% p99 degradation vs +62.3% for sync JIT).

A safe default would be:

jit: {
  minCount: 3,
  eluThreshold: 0.8,
}

This is strictly better than jit: 0 for the vast majority of workloads. The only cost is bounded memory for compiled functions in the LRU cache.

Once #1224 is merged, a follow-up PR could change the default from jit: 0 to adaptive JIT with the above settings.