Can litelllm work with model serving with Triton inference server

[YunchaoYang]

Hi All,

just wondering that is there a way for LittLM to infer with Triton Inference Server using the KServe protocol？

[krrishdholakia]

Hey @YunchaoYang i think i've seen this request before - do you have docs for the api? and is there an easy way to test it

[ishaan-jaff]

@YunchaoYang we support Triton Embeddings: https://docs.litellm.ai/docs/providers/triton-inference-server

Is this what you needed ?

[k0286]

Is there any plan to support completions api with TIS?

[ishaan-jaff]

hi @k0286 - Planning on adding support for it this week by Friday. Can we hop on a call to once this is live ? It would help to debug any issues you run into

• my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
• my linkedin if you prefer DMs: https://www.linkedin.com/in/reffajnaahsi/