[None][fix] Fix _waiting_requests to use compute tokens with KV cache reuse

Cherry-pick from PR NVIDIA#12521. _waiting_requests() was using full input
sequence length (get_tokens(0)) which always exceeded the batch_wait
threshold when KV cache reuse is enabled. Now subtracts
estimated_reusable_tokens to get actual compute tokens.

Signed-off-by: Lance Liao <laliao@login-preos02.a51.clusters.nvidia.com>
Made-with: Cursor

Author: Lance Liao

tensorrt_llm/_torch/pyexecutor/py_executor.py

@@ -2730,8 +2730,11 @@ def _waiting_requests(self, context_requests: list[LlmRequest],
         - The number of waiting iterations is smaller than `self.batch_wait_timeout_iters`.
         """
 
-        num_scheduled_ctx_tokens = sum(
-            len(ctx_req.get_tokens(0)) for ctx_req in context_requests)
+        num_scheduled_ctx_tokens = 0
+        for ctx_req in context_requests:
+            req_tokens = len(ctx_req.get_tokens(0))
+            reusable = ctx_req.estimated_reusable_tokens if ctx_req.is_first_context_chunk else 0
+            num_scheduled_ctx_tokens += max(1, req_tokens - reusable)
         num_scheduled_gen_tokens = sum(1 + gen_req.num_draft_tokens
                                        for gen_req in generation_requests)
         num_scheduled_tokens = num_scheduled_ctx_tokens + num_scheduled_gen_tokens