DeepSpeed on 3Detr

[yxteo2]

I used deepspeed to initialize the model:

model, optimizer, _, lr_scheduler = deepspeed.initialize(
        model=model,
        optimizer=optimizer,
        args=args,
        lr_scheduler=None,
        model_parameters=model.parameters(),
    )

And I had set fp16 to false so that it doesn't use half tensor.

"fp16": {
        "enabled":false
    },

My training code:

`from criterion import build_criterion
# Initialize the best validation metrics (you can load from checkpoint if needed)
best_val_metrics = {
    'best_accuracy': 0.0,
    'best_loss': float('inf')
}
local_rank=0
criterion = build_criterion(args, dataset_config)
criterion = criterion.cuda(local_rank)

do_train(
    args,
    model,
    model,
    optimizer,
    criterion,
    dataset_config,  # Now passing the dataset config
    dataloaders,
    best_val_metrics
)`

And I got the attribute error in stage 1 and 2.py file since it says self.ipg_index not found

File ~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:955, in DeepSpeedZeroOptimizer.reduce_independent_p_g_buckets_and_remove_grads(self, param, i)
    [952](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:952)     self.extra_large_param_to_reduce = param
    [953](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:953) else:
    [954](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:954)     # keeping the gradients contiguous to prevent memory fragmentation, and avoid flattening
--> [955](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:955)     new_grad_tensor = self.ipg_buffer[self.ipg_index].narrow(0, self.elements_in_ipg_bucket, param.numel())
    [956](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:956)     new_grad_tensor.copy_(grad_reduc.view(-1))
    [957](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/c/YuXuan/3detr-plyjson/~/anaconda3/envs/wslpointnet/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:957)     grad_reduc.data = new_grad_tensor.data.view_as(grad_reduc)

AttributeError: 'DeepSpeedZeroOptimizer' object has no attribute 'ipg_index'

[yxteo2]

Hi all it is solved.

Link:
#1218