Does openllm start support model sharding at inference?

[gbmarc1]

I see that the accelerate library is in the dependencies. However, I cannot see any argument that enables model sharing for inference. I may be missing something tho.

[aarnphm]

We currrently shard the model via device_map.

[aarnphm]

But we are working on an internal sharding feature right now, so stay tuned!