Nested for loop in gpu_work.py

[Jushan-Randy-Chen]

Hi,

Thank you for sharing this interesting work!

I have a few questions regarding the implementation in gpu_work.py, and I'd appreciate it if you're happy to help!

At line 73 you have for step in range(num_step):

I am not sure why this additional for loop is needed. It looks like we can get rid of this since we already are iterating over the epochs in the outer-most for loop.

In addition,

temp_model = copy.deepcopy(worker_list[0].model)
            for name, param in temp_model.named_parameters():
                for worker in worker_list[1:]:
                    param.data += worker.model.state_dict()[name].data
                param.data /= size

            if total_step % 50 == 0:
                test_all(temp_model, temp_train_loader, temp_test_loader,
                         criterion, None, total_step, tb, device, n_swap=n_swap)
            if total_step == early_stop:
                break

Currently, you're creating a temp_model = copy.deepcopy(worker_list[0].model) at every iteration, but test_all is only called every 50 iteartions. I wonder if we can move this under if total_step % 50 == 0:?

Thanks!

[Raiden-Zhu]

Hi Jushan,

Thank you for your interest and insightful questions!

First question: Our data logging is primarily based on steps rather than epochs. This design choice was made to align with our theoretical results, which are derived based on steps. This is something that needed clarification, and in our new work, we have already revised this part to improve clarity and efficiency.

Second question: Good point! We don’t actually need to create temp_model = copy.deepcopy(worker_list[0].model) at every iteration.

We’ll be releasing the GitHub repository for our new work soon—stay tuned! Thanks again for your valuable feedback!

Best,
Tongtian

[Jushan-Randy-Chen]

Thank you for the clarifications!

[Jushan-Randy-Chen]

Hi Raiden,

I have a quick follow up question~

I ran gpu_work.py with python gpu_work.py --size 64 --batch_size 64 --learning_rate 0.4 --model_name "ResNet18" --dataset_name "CIFAR10" --milestones 2400 4800 --early_stop 6000 --num_epoch 6000 --gpu True to run D-SGD on the CIFAR 10 dataset with ResNet 18 (only tested the fully connected topology). However, the result I get is different from the one in the paper.

Here is my plot for test_loss - training_loss:

But in the paper, there is a spike at the very beginning:

I wonder what's causing the differences? Thanks!

[Raiden-Zhu]

Hi Jushan,

Thank you for your question! That’s a good point!

The code used in the paper might have had some slight adjustments (the plotting code and the repository code could be slightly different). Do you think it’s possible that the logging interval for test_loss - training_loss in the repository is too large, causing the spike to go unnoticed?

I reran the experiment with our latest paper’s code and still observed the initial spike. To be honest, I don’t have a perfect explanation for that spike. Your results seem to align better with our theory!

[Jushan-Randy-Chen]

Hi Raiden,

I see! I used the default interval of 50 steps, so the train_loss and test_loss are logged to Tensorboard every 50 time steps:
``if total_step % 50 == 0:
test_all(temp_model, temp_train_loader, temp_test_loader,
criterion, None, total_step, tb, device, n_swap=n_swap)```

I reran the experiment with our latest paper’s code and still observed the initial spike.

I wonder if there is any discrepancy between the code here in the repository and your code on your local machine?