[dev] fix git history for dev pull main 260122 (NVIDIA#3094)

Signed-off-by: Robin Zhang <[email protected]>
Signed-off-by: oliver könig <[email protected]>
Signed-off-by: Charlie Truong <[email protected]>
Signed-off-by: Maanu Grover <[email protected]>
Signed-off-by: Jennifer Chen <[email protected]>
Signed-off-by: Antoni-Joan Solergibert <[email protected]>
Signed-off-by: Lifu Zhang <[email protected]>
Signed-off-by: Keshav Santhanam <[email protected]>
Signed-off-by: Youngeun Kwon <[email protected]>
Signed-off-by: Hongbin Liu <[email protected]>
Signed-off-by: Pingtian Li <[email protected]>
Signed-off-by: John St. John <[email protected]>
Signed-off-by: John St John <[email protected]>
Signed-off-by: kunlunl <[email protected]>
Signed-off-by: jianbinc <[email protected]>
Signed-off-by: Deepak Narayanan <[email protected]>
Signed-off-by: dimapihtar <[email protected]>
Signed-off-by: Zhongbo Zhu <[email protected]>
Signed-off-by: Boxiang Wang <[email protected]>
Signed-off-by: Deyu Fu <[email protected]>
Signed-off-by: Hao Wu <[email protected]>
Signed-off-by: Asha Anoosheh <[email protected]>
Signed-off-by: Li Tao <[email protected]>
Signed-off-by: lit <[email protected]>
Signed-off-by: Hongbin Liu <[email protected]>
Signed-off-by: root <[email protected]>
Signed-off-by: tailaim <[email protected]>
Signed-off-by: Parth Mannan <[email protected]>
Signed-off-by: Cory Ye <[email protected]>
Signed-off-by: Jimmy Zhang <[email protected]>
Signed-off-by: Jieming Zhang <[email protected]>
Signed-off-by: Dong Hyuk Chang <[email protected]>
Co-authored-by: Philip Petrakian <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Siddharth Singh <[email protected]>
Co-authored-by: Teodor-Dumitru Ene <[email protected]>
Co-authored-by: Robin Zhang <[email protected]>
Co-authored-by: Jared Casper <[email protected]>
Co-authored-by: oliver könig <[email protected]>
Co-authored-by: Lawrence McAfee <[email protected]>
Co-authored-by: Santosh Bhavani <[email protected]>
Co-authored-by: Charlie Truong <[email protected]>
Co-authored-by: Maanu Grover <[email protected]>
Co-authored-by: Teodor-Dumitru Ene <[email protected]>
Co-authored-by: wdykas <[email protected]>
Co-authored-by: William Dykas <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: HaochenYuan <[email protected]>
Co-authored-by: Philip Petrakian <[email protected]>
Co-authored-by: Jenny Chen <[email protected]>
Co-authored-by: Antoni-Joan Solergibert <[email protected]>
Co-authored-by: Deepak Narayanan <[email protected]>
Co-authored-by: Lifu Zhang <[email protected]>
Co-authored-by: Lifu Zhang <[email protected]>
Co-authored-by: Shanmugam Ramasamy <[email protected]>
Co-authored-by: Shanmugam Ramasamy <[email protected]>
Co-authored-by: Keshav Santhanam <[email protected]>
Co-authored-by: Kan Zhu <[email protected]>
Co-authored-by: helen ngo <[email protected]>
Co-authored-by: Youngeun Kwon <[email protected]>
Co-authored-by: Nick Schank <[email protected]>
Co-authored-by: Eric Harper <[email protected]>
Co-authored-by: Nick Schank <[email protected]>
Co-authored-by: wineandchord <[email protected]>
Co-authored-by: Xin Yao <[email protected]>
Co-authored-by: Chenhan D. Yu <[email protected]>
Co-authored-by: Hongbin Liu <[email protected]>
Co-authored-by: Pingtian Li <[email protected]>
Co-authored-by: John St. John <[email protected]>
Co-authored-by: kwyss-nvidia <[email protected]>
Co-authored-by: ankurv-nvidia <[email protected]>
Co-authored-by: Deepak Narayanan <[email protected]>
Co-authored-by: Jon Barker <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Yuzhong Wang <[email protected]>
Co-authored-by: Kunlun Li <[email protected]>
Co-authored-by: jianbinc <[email protected]>
Co-authored-by: Cory Ye <[email protected]>
Co-authored-by: shanmugamr1992 <[email protected]>
Co-authored-by: yobi byte <[email protected]>
Co-authored-by: Chen Cui <[email protected]>
Co-authored-by: Yu Yao <[email protected]>
Co-authored-by: Mcore Bot <[email protected]>
Co-authored-by: Dmytro Pykhtar <[email protected]>
Co-authored-by: dimapihtar <[email protected]>
Co-authored-by: Zhongbo Zhu <[email protected]>
Co-authored-by: Zijie Yan <[email protected]>
Co-authored-by: Hao Wu <[email protected]>
Co-authored-by: Boxiang Wang <[email protected]>
Co-authored-by: mikail <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Asha Anoosheh <[email protected]>
Co-authored-by: Hexin Wang <[email protected]>
Co-authored-by: Russell Hewett <[email protected]>
Co-authored-by: Li Tao <[email protected]>
Co-authored-by: shifangx <[email protected]>
Co-authored-by: Deepak Joshi <[email protected]>
Co-authored-by: Hongbin Liu <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: John Kamalu <[email protected]>
Co-authored-by: Brandon Norick <[email protected]>
Co-authored-by: Pingtian Li <[email protected]>
Co-authored-by: Duncan Riach <[email protected]>
Co-authored-by: xuwchen <[email protected]>
Co-authored-by: John St. John <[email protected]>
Co-authored-by: Parth Mannan <[email protected]>
Co-authored-by: tailaim <[email protected]>
Co-authored-by: kunlunl <[email protected]>
Co-authored-by: Jimmy Zhang <[email protected]>
Co-authored-by: Yashaswi Karnati <[email protected]>
Co-authored-by: Dong Hyuk Chang <[email protected]>

Author: 90 people

megatron/training/checkpointing.py

@@ -534,14 +534,6 @@ def save_checkpoint(iteration, model, optimizer, opt_param_scheduler, num_floati
         if not optimizer.is_stub_optimizer:
             optimizer.save_state_dict_to_file(optim_checkpoint_name)
 
-    # LayerWiseDistributedOptimizer save optimizer state to file on different ranks
-    if getattr(args, "optimizer", "adam").startswith("dist_") and args.ckpt_format == 'torch':
-        dp_rank = mpu.get_data_parallel_rank()
-        optim_checkpoint_name = os.path.join(os.path.dirname(checkpoint_name), f"layer_wise_optimizer_{dp_rank}.pt")
-        ensure_directory_exists(optim_checkpoint_name)
-        if not optimizer.is_stub_optimizer:
-            optimizer.save_state_dict_to_file(optim_checkpoint_name)
-
     async_save_request = None
     if args.async_save:
         if ckpt_type == CheckpointType.LEGACY: