Support for Evaluating Self-Deployed Models

[hccngu]

I would like to know if your platform supports evaluating models that are self-deployed (i.e., models hosted on our own infrastructure rather than using pre-built or cloud-hosted models).

If so, could you please provide documentation or guidance on how to integrate and evaluate such models?

Thank you!

[zjwu0522]

hi @hccngu , you can use vllm or sglang to deploy your model as litellm supports them. Please refer to https://docs.litellm.ai/docs/providers/vllm for further information.