[https://nvbugs/5569754][chore] Adjust max batch size to prevent OOM (#8876)

JunyiXu-nv · web-flow · commit c329f5f78bb5 · 2025-11-04T18:34:26.000+01:00
Signed-off-by: Junyi Xu &lt;219237550+JunyiXu-nv@users.noreply.github.com&gt;
diff --git a/examples/llm-api/llm_mgmn_llm_distributed.sh b/examples/llm-api/llm_mgmn_llm_distributed.sh
@@ -51,5 +51,6 @@ srun -l \
         trtllm-llmapi-launch python3 $script \
             --model_dir $LOCAL_MODEL \
             --prompt 'Hello, how are you?' \
-            --tp_size 2
+            --tp_size 2 \
+            --max_batch_size 256
     "

Original file line number	Diff line number	Diff line change
`@@ -51,5 +51,6 @@ srun -l \`
`51`	`51`	`trtllm-llmapi-launch python3 $script \`
`52`	`52`	`--model_dir $LOCAL_MODEL \`
`53`	`53`	`--prompt 'Hello, how are you?' \`
`54`		`- --tp_size 2`
	`54`	`+ --tp_size 2 \`
	`55`	`+ --max_batch_size 256`
`55`	`56`	`"`