[Bug] Nan grad norm in MXFP8 with param AG overlap

[erhoo82]

Currently, param_ag_overlap is disabled.
Need to resolve and enable the overlap

[erhoo82]

@malay-nagda tried M-LM using the same config, and confirmed that it also faces NaN grad issue.

[erhoo82]

@malay-nagda Can you share what config to use to reproduce the issue?

[malay-nagda]

• Llama3 70B GB200: TP2-PP4-CP1-VP5-MBS1-GBS128
• DeepSeek v3 on GB200: TP1-PP4-CP1-EP64-VP4-MBS1-GBS2048

these lines also need to be commented to reproduce the issue- https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/scripts/performance/configs/llama/llama3_llm_pretrain.py#L142-L145

There are other configs that has this issue (basically wherever the above linked codelines are present in the performance scripts)

[erhoo82]

This is resolved in the RC1 container.