[Bug]: Crash when enabling MAX_UTILIZATION

[Shang-Pin]

System Info

• TensorRT-LLM v1.2.0rc2

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

enabling MAX_UTILIZATION in the scheduler_config, specifically we saw it in deepseek v3

scheduler_config:
capacity_scheduler_policy: MAX_UTILIZATION

Expected behavior

No crash

actual behavior

2025-12-04 22:23:18.852
[12/05/2025-06:23:18] [TRT-LLM] [RANK 0] [I] iter = 4286, global_rank = 0, rank = 0, currank_total_requests = 0/0, host_step_time = 88.22870254516602ms, prev_device_step_time = 89.87372589111328ms, timestamp = 2025-12-05 06:23:18, num_scheduled_requests: 48, states = {'num_ctx_requests': 0, 'num_ctx_tokens': 0, 'num_generation_tokens': 96}
2025-12-04 22:23:18.869
[12/05/2025-06:23:18] [TRT-LLM] [RANK 7] [E] Error in event loop: vector::_M_default_append
2025-12-04 22:23:18.870
Exception in thread Thread-3 (_event_loop_wrapper):
2025-12-04 22:23:18.870
Traceback (most recent call last):
2025-12-04 22:23:18.870
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
2025-12-04 22:23:18.870
[12/05/2025-06:23:18] [TRT-LLM] [RANK 5] [E] Error in event loop: vector::_M_default_append
2025-12-04 22:23:18.870
[12/05/2025-06:23:18] [TRT-LLM] [RANK 0] [E] Error in event loop: vector::_M_default_append
2025-12-04 22:23:18.870
[12/05/2025-06:23:18] [TRT-LLM] [RANK 7] [E] Traceback (most recent call last):
2025-12-04 22:23:18.870
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 346, in _event_loop_wrapper
2025-12-04 22:23:18.870
self.event_loop()
2025-12-04 22:23:18.870
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1425, in _executor_loop_overlap
2025-12-04 22:23:18.870
self._pause_requests(scheduled_batch.paused_requests)
2025-12-04 22:23:18.870
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 2535, in _pause_requests
2025-12-04 22:23:18.870
req.pause(max_input_len)
2025-12-04 22:23:18.870
ValueError: vector::_M_default_append
2025-12-04 22:23:18.870
2025-12-04 22:23:18.871
Exception in thread Thread-3 (_event_loop_wrapper):
2025-12-04 22:23:18.871
Traceback (most recent call last):
2025-12-04 22:23:18.871
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
2025-12-04 22:23:18.871
[12/05/2025-06:23:18] [TRT-LLM] [RANK 5] [E] Traceback (most recent call last):
2025-12-04 22:23:18.871
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 346, in _event_loop_wrapper
2025-12-04 22:23:18.871
self.event_loop()
2025-12-04 22:23:18.871
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1425, in _executor_loop_overlap
2025-12-04 22:23:18.871
self._pause_requests(scheduled_batch.paused_requests)
2025-12-04 22:23:18.871
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 2535, in _pause_requests
2025-12-04 22:23:18.871
req.pause(max_input_len)
2025-12-04 22:23:18.871
ValueError: vector::_M_default_append
2025-12-04 22:23:18.871

additional notes

We start seeing the crash most likely when there are more requests getting paused when kv cache space runs out.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[pythongiant]

Hmm, seems like when MAX_UTILIZATION is enabled the scheduler over-schedules tokens to fully saturate SMs.
have you tried to reduce pause explosion:

scheduler_config:
  max_paused_requests: 8

(or even 4)

This essentially limits vector growth.

[pythongiant]

you can also set larger KV pages, and increase reserved KV headroom or disable overlap executor.
This bug happens in _executor_loop_overlap.

export TRTLLM_EXECUTOR_OVERLAP=0

[pcastonguay]

Could you share repro instructions?

[Funatiq]

Hi again @Shang-Pin, could you please provide repro instructions?

[Shang-Pin]

@Funatiq @pcastonguay Here's a simple script that can make it crash.

import aiohttp
import time
import os
from dataclasses import dataclass

@dataclass
class Result:
    success: bool
    status_code: int | None
    latency: float
    error: str | None = None

class ProgressTracker:
    def __init__(self, total: int):
        self.total = total
        self.completed = 0
        self.successful = 0
        self.failed = 0
        self.lock = asyncio.Lock()

    async def update(self, success: bool):
        async with self.lock:
            self.completed += 1
            if success:
                self.successful += 1
            else:
                self.failed += 1
            print(f"\rProgress: {self.completed}/{self.total} ({self.successful} ok, {self.failed} failed)", end="", flush=True)

async def send_request(session: aiohttp.ClientSession, url: str, headers: dict, payload: dict, progress: ProgressTracker) -> Result:
    start = time.perf_counter()
    try:
        async with session.post(url, headers=headers, json=payload) as response:
            await response.text()
            result = Result(
                success=response.status == 200,
                status_code=response.status,
                latency=time.perf_counter() - start
            )
            await progress.update(result.success)
            return result
    except Exception as e:
        result = Result(
            success=False,
            status_code=None,
            latency=time.perf_counter() - start,
            error=str(e)
        )
        await progress.update(result.success)
        return result

async def run_load_test(num_requests: int = 300, concurrency: int = 300):
    url = "http://your_end_point/v1/chat/completions"
    api_key = "123"

    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }

    payload = {
        "model": "model-name",
        "messages": [{"role": "user", "content": "Say hello"}],
        "max_tokens": 3000,
        "temperature":10
    }

    connector = aiohttp.TCPConnector(limit=concurrency)
    timeout = aiohttp.ClientTimeout(total=6000)

    async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
        print(f"Sending {num_requests} concurrent requests...")
        progress = ProgressTracker(num_requests)
        start_time = time.perf_counter()

        tasks = [send_request(session, url, headers, payload, progress) for _ in range(num_requests)]
        results = await asyncio.gather(*tasks)

        total_time = time.perf_counter() - start_time
        print()  # newline after progress

    # Analyze results
    successful = [r for r in results if r.success]
    failed = [r for r in results if not r.success]

    print(f"\n{'='*50}")
    print(f"Results Summary")
    print(f"{'='*50}")
    print(f"Total requests:    {num_requests}")
    print(f"Successful:        {len(successful)}")
    print(f"Failed:            {len(failed)}")
    print(f"Total time:        {total_time:.2f}s")
    print(f"Requests/sec:      {num_requests / total_time:.2f}")

    if successful:
        latencies = [r.latency for r in successful]
        print(f"\nLatency (successful requests):")
        print(f"  Min:    {min(latencies):.3f}s")
        print(f"  Max:    {max(latencies):.3f}s")
        print(f"  Avg:    {sum(latencies)/len(latencies):.3f}s")

    if failed:
        print(f"\nFailure breakdown:")
        status_codes = {}
        for r in failed:
            key = r.status_code or r.error
            status_codes[key] = status_codes.get(key, 0) + 1
        for code, count in sorted(status_codes.items(), key=lambda x: -x[1]):
            print(f"  {code}: {count}")

if __name__ == "__main__":
    asyncio.run(run_load_test(num_requests=300, concurrency=300))

I'm not getting the exact same crash, but it only happens with max_utilization enabled.

[01/15/2026-23:35:09] [TRT-LLM] [E] Error in event loop: [TensorRT-LLM][ERROR] Assertion failed: isLeaf() (../tensorrt_llm/batch_manager/kvCacheManager.cpp:465)
1       0x7688f366f87d tensorrt_llm::_v1::common::throwRuntimeError(char const*, int, char const*) + 97
2       0x7688b452db1a /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(+0x195cb1a) [0x7688b452db1a]
3       0x7688b5c53025 tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::getFreeBlock(tensorrt_llm::batch_manager::kv_cache_manager::GenerationRequest&, int, std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >, tensorrt_llm::executor::KvCacheTransferMode, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 181
4       0x7688b5c561a9 tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::allocateBlock(tensorrt_llm::batch_manager::kv_cache_manager::GenerationRequest&, bool) + 185
5       0x7688b5c564ac tensorrt_llm::batch_manager::kv_cache_manager::WindowBlockManager::adjustBlocksIfNeeded(tensorrt_llm::batch_manager::kv_cache_manager::GenerationRequest&) + 252
6       0x7688b5c56569 tensorrt_llm::batch_manager::kv_cache_manager::BlockManager::adjustBlocksIfNeeded(tensorrt_llm::batch_manager::kv_cache_manager::GenerationRequest&) + 137
7       0x768904f16f6e /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x50df6e) [0x768904f16f6e]
8       0x768904fb5e9c /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x5ace9c) [0x768904fb5e9c]
9             0x549cf5 PyObject_Vectorcall + 53
10            0x5d68bf _PyEval_EvalFrameDefault + 2783
11            0x54cea2 /usr/bin/python() [0x54cea2]
12            0x5daa90 _PyEval_EvalFrameDefault + 19632
13            0x54cea2 /usr/bin/python() [0x54cea2]
14            0x6f7a8c /usr/bin/python() [0x6f7a8c]
15            0x6b862c /usr/bin/python() [0x6b862c]
16      0x768c0d4bbaa4 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x768c0d4bbaa4]
17      0x768c0d548a64 __clone + 68
Exception in thread Thread-3 (_event_loop_wrapper):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
[01/15/2026-23:35:09] [TRT-LLM] [E] Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 365, in _event_loop_wrapper
    self.event_loop()
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py", line 1605, in _executor_loop_overlap
    self.resource_manager.prepare_resources(scheduled_batch)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 1417, in prepare_resources
    resource_manager.prepare_resources(scheduled_batch)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 487, in prepare_resources
    self.impl.add_token(req.py_request_id)
RuntimeError: [TensorRT-LLM][ERROR] Assertion failed: isLeaf() (../tensorrt_llm/batch_manager/kvCacheManager.cpp:465)

The engine configuration is

trtllm-serve serve /data/weights/hf-microsoft--phi-4/001 --tp_size=1 --backend=pytorch --host=0.0.0.0 --port=8000 --max_seq_len=16384 --max_num_tokens=4096 --max_batch_size=128 --extra_llm_api_options=extra.yml --enable_chunked_prefill

print_iter_log: true
enable_iter_perf_stats: true
kv_cache_config:
  free_gpu_memory_fraction: 0.1
scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION

on 1xB200 with nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc7