How to do distributed training with for example NCCL

[npuichigo]

candle backend seems to have suuport for distributed training. so how to use that in burn

[antimora]

We only take advantage of Tensor operations of Candle. @nathanielsimard and @louisfd have recently added some ground work for distributed computing but it's not fully complete I believe.

[sharifhsn]

Any updates on the distributed computing story in burn?

[nathanielsimard]

We recently merged the backend Router (#2353). This is the foundation to support distributed training with any backend, or even mixed backends. Essentially, with lazy execution we transform tensor function calls into data and send those to backend runners to be executed. Right now everything is in-memory, but we plan to also support HTTP connections, or any network protocol really.

[eugenehp]

Exciting! May I suggest to look into OpenMPI and NGCC for distributed execution. Also if you must do HTTP, maybe protobuf and gRPC could be a good alternative to minimize serialization/deserialization, and optimize latency.

[nathanielsimard]

I implemented it using raw websockets (https://github.com/tracel-ai/burn/tree/main/crates/burn-remote) with binary messages. We control the serialization, using mpk for now, but we could switch to bincode or the most performant one (no json).