More details of the challenge

[hui-po-wang]

Hi,

I have some questions regarding the details of the challenge.

• What functions can we modify and submit exactly? I knew you already provided few examples in the jupyter notebook. Are they all we can modify? How about other parameters, such as rounds_to_train?

• In the official document, one evaluation metric is sent/received bytes, but I didn't see how it's implemented in the jupyter notebook. Would you provide the details of how you measure it and let us measure it on our side as well?

• Due to some reasons, our cluster has a wall time limit of 24h. How do I do checkpointing on your framework?

Originally posted by @a514514772 in #3 (reply in thread)

[msheller]

These are great questions! I'll let @psfoley or @alexey-gruzdev take first crack at the first two questions, but the 3rd is a gap. We don't have robust checkpointing of the simulation state, and the model checkpointing from OpenFL isn't exposed via the FeTS challenge interface. This is important, and I will raise it with the team today.

[hui-po-wang]

Thanks! looking forward to the update.

[psfoley]

Thanks @a514514772 for the great questions!

• Regarding your first question, the jupyter notebook should be submitted and any other dependent files needed to reproduce your results. We have expanded the documentation in the Jupyter notebook to make it more clear which functions can modified (aggregation_function, choose_training_collaborators, training_hyper_parameters_for_round, and validation_functions` are intended to be changed, but you can also increase the number of rounds to improve your results. Cross-posting the description from the notebook below:

# change any of these you wish to your custom functions. You may leave defaults if you wish.
aggregation_function = weighted_average_aggregation
choose_training_collaborators = all_collaborators_train
training_hyper_parameters_for_round = constant_hyper_parameters
validation_functions = [('sensitivity', sensitivity), ('specificity', specificity)]

# Final scoring will be on partitioning_1, partitioning_2, and a hidden partitioning
# We encourage you to experiment with other partitionings
institution_split_csv_filename = 'partitioning_1.csv'

# change this to point to the parent directory of the data
brats_training_data_parent_dir = '/raid/datasets/FeTS21/MICCAI_FeTS2021_TrainingData'

# increase this if you need a longer history for your algorithms
# decrease this if you need to reduce system RAM consumption
db_store_rounds = 5

# this is passed to PyTorch, so set it accordingly for your system
device = 'cuda'

# you'll want to increase this most likely. You can set it as high as you like, 
# however, the experiment will exit once the simulated time exceeds one week. 
rounds_to_train = 5

• For your second point, the evaluation of sent/received bytes has been reformulated as Time to Convergence. The purpose of this is to reward implementations that converge faster. For a detailed description of the metric's implementation, see our updated README.md

• Regarding your third question, there is now an open PR for the checkpointing capability that should be merged soon.