triton with vllm chat completions

[geraldstanje]

hi,

how can i do the following with triton and vllm backend? i only found a generate endpoint - why i cannot find /chat/completions? and how to set max_num_tokens?

import requests
import json

# Your local vLLM server (OpenAI-compatible)
API_URL = "http://localhost:8000/v1/chat/completions"

# Chat-style messages
messages = [
    {"role": "system", "content": "You are a friendly assistant that writes in haiku form."},
    {"role": "user", "content": "Write a haiku about coding on a Mac."}
]

# Request payload
payload = {
    "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    "messages": messages,
    "max_tokens": 50,
    "temperature": 0.7
}

# Send request
response = requests.post(API_URL, headers={"Content-Type": "application/json"}, data=json.dumps(payload))

# Print nicely formatted output
if response.ok:
    data = response.json()
    print("Assistant:", data["choices"][0]["message"]["content"])
else:
    print("Error:", response.text)

cc @pskiran1 @dinhanhx @amit-timalsina @ahakanbaba

[amit-timalsina]

You have to use openai compatible triton front-end. But note that it is in beta.

https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/client_guide/openai_readme.html

I have already done this so let me know if you need further help.

[geraldstanje]

@amit-timalsina if i dont use the openai compatible triton front-end - how can i send system and user prompt to the triton vllm backend?

[amit-timalsina]

This should help.

https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Quick_Deploy/vLLM/README.html

Triton is a wrapper of multiple inference engines like vllm, tensortllm, python, etc. You have to setup model repository provide it to triton and you can use the generate endpoint to call it by providing model id.

[geraldstanje]

@amit-timalsina sure - will something like this work?

import requests
import json

API_URL = "http://localhost:8000/v2/models/my_model/generate"

# Convert chat messages into a single formatted prompt
def format_chat(messages):
    formatted = ""
    for m in messages:
        formatted += f"{m['role'].capitalize()}: {m['content']}\n"
    formatted += "Assistant:"
    return formatted

messages = [
    {"role": "system", "content": "You are a friendly assistant that writes in haiku form."},
    {"role": "user", "content": "Write a haiku about coding on a Mac."}
]

payload = {
    "text_input": format_chat(messages),
    "parameters": {
        "max_num_tokens": 50,
        "temperature": 0.7
    }
}

response = requests.post(API_URL, headers={"Content-Type": "application/json"}, data=json.dumps(payload))

if response.ok:
    data = response.json()
    print("Assistant:", data["text_output"])
else:
    print("Error:", response.text)

[amit-timalsina]

Triton server is running, right?

Then it should work.

[geraldstanje]

yes triton server works fine - for max prompt there is parameter "max_tokens"

@amit-timalsina have you noticed that vllm backend sometimes doesnt return an expected output? i see some inconsistancy - also inference runs half of the expected time only 700ms instead 2000ms...