[QUESTION]About HybridDeviceOptimizer CPU Offloading Behavior

[xiaoxiwangfj]

In the HybridDeviceOptimizer's CPU offloading implementation(https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/optimizer/cpu_offloading/hybrid_optimizer.py ), which components (gradients, optimizer states and FP32 master parameters)? are both the optimizer states and FP32 master parameters offloaded to the CPU?

From the code, it appears that optimizer updates are performed on both CPU and GPU. Could you clarify why CPU-side updates are necessary?
# Step the sub-optimizers.
if self.gpu_optimizer:
self.gpu_optimizer.step(closure)

    for cpu_optimizer in self.cpu_optimizers:
        d2h_event = self._cpu_optimizer_map_data_event.pop(cpu_optimizer, None)
        if d2h_event is not None:
            d2h_event.synchronize()
        cpu_optimizer.step(closure)

[xiaoxiwangfj]

both the optimizer states and FP32 master parameters offloaded to the CPU