Vllm batch upgrade#715

dmchoiboi wants to merge 6 commits intomainscaleapi/llm-engine:mainfrom

dmchoi/vllm_batch_upgradescaleapi/llm-engine:dmchoi/vllm_batch_upgradeCopy head branch name to clipboard