What's the future plan for torchchat serving

[jenniew]

I see current torchchat serving provides basic serving function. I'm wondering what the future plan for serving. What's the target of torchchat serve? Will it provide more optimized and high performance serving features(like Continuous batching, prefix-caching, chunked prefill, etc.)

[Jack-Khuu]

I plan on sharing a directional RFC in the coming week (or next). As a sneak peek, we plan to grow torchchat via 2 vectors:

• Optimizing the core inference logic
• Leveraging PyTorch libraries (torchtune, torchao, ExecuTorch, etc.)

I believe your list will fall into the former

Will it provide more optimized and high performance serving features(like Continuous batching, prefix-caching, chunked prefill, etc.)

We plan to add more optimized serving features through a combination of Meta contributions and OSS (both OSS initiated and us providing RFC's for the community to take on).

Batching is a popular request, I'd like to have (should be pretty straight forward).
Parallel Prefill is on by default for non-export inference

What specific features catch your eye?

[jenniew]

Thank you so your response. Could you also provide some benchmarks so we can easy test performance on our platform?

[Jack-Khuu]

Totally, I'll spin up a script