Question on Running Local Models

[Reapor-Yurnero]

Thanks for this awesome work first of all!

I have a question on how to run this benchmark on local LLMs ideally served by VLLM potentially.

I notice that in the paper there was result for llama3.1-70b but the script to execute that was not provided in the docs. And it seems in the run script all the inference is handled by litellm which mostly looks at vendor-hosted LLMs endpoints. However, in the model_util folder I do find implementations for VLLM served models but seems only used by the auto_error_identification script. What would be the expected way if I would like to run this benchmark on a locally served model? Thank you very much in advance!

[HqWei]

the same problem

[kkkeepgoing]

the same problem +1, have you solved it?

[fatday]

Same problem, I have no idea how to integrate my own local model that served by vllm for this benchmark