Train locally after last global model was shared

[LSnyd]

Hi there,

I am running a federated learning scenario with MONAI and NVIDIA with ok-ish performance.
Is it possible to train another round of training locally after each client receives the last global model?
I am hoping that this could boost my performance a bit.

Best,
Lisa

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help share some comments here?
And @LSnyd , could you please help share more information about your program? Are you using NVIDIA NVFlare project?

Thanks.

[yiheng-wang-nv]

Hi @LSnyd , did you use NVFLARE? Can train another round be replaced by increasing the number of training epochs?

[yiheng-wang-nv]

In NVFLARE example, the place to change it is:
https://github.com/Project-MONAI/tutorials/blob/master/federated_learning/nvflare/nvflare_example_docker/spleen_example/custom/monai_trainer.py#L38

[LSnyd]

Hi @yiheng-wang-nv,
yes, I am using NVFlar and MONAI.
I am fine with adding more local epochs. However, I would like to run one more round of local training on the client side after the last global model was shared to overcome the data heterogeneity. Is that possible?

[yiheng-wang-nv]

Hi @LSnyd , could I ask what is the difference between run one more round of local training and increase the number of training epochs of a round? Could you provide a more detailed example? Thanks!

[LSnyd]

Hi @yiheng-wang-nv ,

first of all thanks so much for your help. I will try to explain it a bit more detailed.

My current assumption is the following:
I am configuring the local and global epochs. For each global epoch, the model gets sent to the clients and is trained the number of local epochs that I defined. After all the global epochs are finished, we receive a best global model (located in the MMAR_server folder of the run). I am using this global model to test the performance for each client. (I couldn't find any stored models on the client side)

Now, I was wondering if it is possible to take this global model and train it one more time for 5 epochs or so on the client side and use this model to test the performance. That way the model may overcome data heterogeneity and may fit better to the local data.
So finally, the one more round of local training is meant to be placed after the best global model was already reached. Do you know what I mean?

[yiheng-wang-nv]

Hi @LSnyd , sorry for late reply.

I think it's feasible to implement take this global model and train it one more time for 5 epochs or so on the client side and use this model to test the performance. For example, in this line you received the global model, a direct way is to write a training loop after the line.

In addition, recently NVIDIA FLARE 2.0 just released, and I submitted a PR to update the example (Project-MONAI/tutorials#460). I think the new version may be more easier to use, and it's more flexible to define different tasks during a workflow.