Add more environment variables for NCCL #146
Conversation
|
preview available: https://docs.tds.cscs.ch/146 |
1 similar comment
|
preview available: https://docs.tds.cscs.ch/146 |
docs/software/communication/nccl.md
Outdated
| export NCCL_CROSS_NIC=1 | ||
| export NCCL_NET_FORCE_FLUSH=1 | ||
| export NCCL_NET_GDR_LEVEL=PHB # (2)! | ||
| export NCCL_SOCKET_IFNAME=hsn |
There was a problem hiding this comment.
NCCL_CROSS_NIC and NCCL_SOCKET_IFNAME are set in the CE hook, and seem safe to recommend for most users (latter anyway seems to be just a sanity setting to avoid using the wrong network).
Good catch. I wonder if we can avoid copy-pasting and manually having to make sure they're synchronized. I like that the pytorch submission script is standalone. Do you think it would be bad if we just link to the nccl page from there? I'd imagine many users will miss copying the nccl variables in that case... From a quick search this also seems to exist: https://squidfunk.github.io/mkdocs-material/reference/code-blocks/#embedding-external-files. That might allow defining these in one place and including in many. That said, that might be overkill at the moment so I might just copy them over for now. Any comments on which vars we can actually safely recommend and which we might want to wait with still? Or if we have to add a warning about some variables only being good/useful with nccl 2.26 and libfabric 1.22? |
|
preview available: https://docs.tds.cscs.ch/146 |
msimberg
force-pushed
the
more-nccl-env-vars
branch
from
6314979 to
ef3019e
Compare
|
The added environment variables are now in ef3019e. |
|
preview available: https://docs.tds.cscs.ch/146 |
|
preview available: https://docs.tds.cscs.ch/146 |
msimberg
force-pushed
the
more-nccl-env-vars
branch
from
959ec3a to
ef4ec10
Compare
|
preview available: https://docs.tds.cscs.ch/146 |
|
I'm closing this for now. These additional variables are not understood well enough to warrant adding them without explanation to the docs. |
2 participants
Draft. Not clear if all are needed.
Do we need to start separating recommended environment variables by nccl, libfabric, etc. version, or is it sufficient if we recommend the best practices for the latest versions?