How to train Qwen3-480B EAGLE draft model on multiple nodes?

[xiaomin-D]

I want to train the qwen3_coder_480b_a35b_fp8_eagle3_online config, but I'm not sure how to run it in a multi-node environment.

Environment

• 2 nodes, each with 8x H200 GPUs (16 GPUs total)
• InfiniBand interconnect between nodes
• NCCL RDMA configured

Questions

For the qwen3_coder_480b_a35b_fp8_eagle3_online config:

1. How do I launch training on multiple nodes?
2. Which parameters need to be modified in the config file?
3. Do I need to execute commands separately on each node?

Could you provide an example command or configuration? Thanks!

[jiapingW]

I tried train qwen3-coder-480b-instruct online with B200 X 8. When the max_length is 2048, the single GPU will cost 160G VRAM at most.

[jiapingW]

The command is below: the hf backend will use less 5G VRAM than custom.

torchrun \
    --standalone \
    --nproc_per_node $NUM_GPUS \
    $ROOT_DIR/scripts/train_eagle3.py \
    --target-model-path Qwen/Qwen3-Coder-480B-A35B-Instruct \
    --draft-model-config $ROOT_DIR/configs/qwen3-coder-480B-A35B-instruct-eagle3.json \
    --train-data-path $ROOT_DIR/cache/dataset/sharegpt_train.jsonl \
    --build-dataset-num-proc $BUILD_DATASET_NUM_PROC \
    --output-dir $ROOT_DIR/outputs/qwen3-coder-480B-A35B-instruct-eagle3-sharegpt \
    --num-epochs 10 \
    --batch-size 1 \
    --learning-rate 1e-4 \
    --max-length 2048 \
    --chat-template qwen \
    --cache-dir $ROOT_DIR/cache \
    --embedding-key model.embed_tokens.weight \
    --tp-size $TP_SIZE \
    --target-model-backend hf \
    --report-to tensorboard

[jiapingW]

I suggest use tp=16 to train qwen3-coder-480b-instruct with H200 if you want to train online.

[pengyao96]

I have also encountered scenarios where a single machine cannot meet the training requirements. Could you tell me how to train the model with multiple nodes and provide some related scripts?

[xiaomin-D]

I have also encountered scenarios where a single machine cannot meet the training requirements. Could you tell me how to train the model with multiple nodes and provide some related scripts?

no,Ididn't find the way to train in multi nodes,
I trained FP8 version in single node.

[jiapingW]

See here. I trained on a GPU cluster.