Concurrent Requests on a single runtime name

[psm-2]

Would it be possible to create multiple runtimes for a single model (ex. Qwen3-30b), and if the first runtime is generating, make the API route to the second runtime?

[neilmehta24]

Batched inference, which would enable this feature, isn't supported yet but it's certainly on our roadmap

[psm-2]

Batched inference, which would enable this feature, isn't supported yet but it's certainly on our roadmap

That would be incredible!

[vincentamato]

@neilmehta24 Batch generation was recently added to mlx-lm in #443. Are you still looking to support it in the MLX engine and in the LM Studio application?

[psm-2]

Yes, that would be incredible!

[neilmehta24]

@neilmehta24 Batch generation was recently added to mlx-lm in #443. Are you still looking to support it in the MLX engine and in the LM Studio application?

We've been following these changes closely and it's certainly on our roadmap. Stay tuned!

[GirevoyT]

@neilmehta24
Any news?

UPD 29.01.2026 The question is no longer relevant. (Unfortunately, it's only available in GGUF format at the moment.) :)