Per-round evaluation on held out central server validation data possible?

[Linardos]

My understanding is that metrics are calculated locally twice: once after the aggregate model is redistributed and once after its trained locally (before the new aggregation step). I was wondering whether it would fall within the rules of this contest to have a separate unseen center in the central server to calculate some utility metrics on newly trained local models before the aggregation step, so as to evaluate on an objective set that was unseen by during their training. And if yes, is this functionality already available in the notebook?

[sarthakpati]

Hi @Linardos,

Thank you for your interest in the FeTS challenge!

You can have an unseen center to calculate utility metrics BUT this mechanism will not be used during ranking. For ranking, the organizers will re-train the model using ONLY the functions defined in the notebook (which each participant will edit). In case of large discrepancy between the performance of the model reported by participant and the one we train, we will personally contact the participant for clarification (this will be reflected in the challenge manuscript).

Cheers,
Sarthak on behalf of the FeTS Challenge Organizing Committee