How to pass additional sampling parameters to the vllm server ?

[pinak-p]

I would like to pass additional sampling parameters when invoking guidellm. Is it possible to do so ?

[sjmonson]

#78 was recently merged which adds support for passing arguments to the backend when guidellm is used as a library.

[markurtz]

@pinak-p let us know if #78 fixes the issues you were seeing. Otherwise can you provide an example for your scenario?

[pinak-p]

Thank you for adding that. I'll take a look and get back.

[pinak-p]

@markurtz @sjmonson - How does one pass the argument from the cli ? I'm looking at passing my own parameters like topk or ignore_eos with the request.

[markurtz]

@markurtz @sjmonson - How does one pass the argument from the cli ? I'm looking at passing my own parameters like topk or ignore_eos with the request.

Ah, we'll need to expand out a bit more for that use case. Currently the --backend-args are only passed into the constructor for the backend, not to each request.

[pinak-p]

Got it. I think it will be a useful feature to have for benchmarking specific scenarios. Also, using ignore_eos ensures the desired number of tokens are generated always.