Questions related to db_iterator and reports for Task-1

[GorkemP]

Dear organizers,

I have serveral questions related to Task-1:

1. Training dataset is first split into 80%-20% for train-val, then train part is distributed to the collaborators according to their partitioning info, is that correct? Or each institution has its own train-val split?

2. Regarding the tags of db_iterator, are (model,), (aggregated,) , and (trained,) refers to the same thing?

3. When I printout the content of records of db_iterator, I see several reoccuring items, for example:

INFO     round                                                          0                                                          FeTS_fedavg.py:122
         nparray        [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, ...                                                                            
         tensor_name                                ds_3.in_0.running_var                                                                            
         tags                                                    (model,)                                                                            
         Name: 281, dtype: object                                           

and

 INFO     round                                                          0                                                          FeTS_fedavg.py:122
          nparray        [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, ...                                                                            
          tensor_name                                ds_3.in_0.running_var                                                                            
          tags                                                    (model,)                                                                            
          Name: 82, dtype: object

Only their Name is different, othan than that, they seem to refer to the same thing, am I correct or do I miss something?

4. At round-0 when we see a tag of (metric, validate_local, 7), it refers to the validation set result of locally trained model on collaborator 7. Then, what does (metric, validate_local) refers to?

5. I think (metric, validate_agg) and (metric, validate_agg, COLLABORATOR_ID) for the same metric will always be equal right?

6. After the collaborators finish their local tunings, aggregator.py file logs information related to aggregation. It mainly reports performance metrics for two different models: aggregated_model_validation and locally_tuned_model_validation. I assume performance metric results related to aggregated_model_validation refers to the validation score after the aggregation. What about locally_tuned_model_validation? Here is a detailed report:

In addition, is the loss under train task metrics... refers to the mean of collaborator losses?

Answers to these questions will help us understand more about the underlaying mechanism.

Thank you in advance.

[eceisik]

Dear Organizers,
Is there any progress on these questions?
Thank you in advance

[sarthakpati]

Hi,

Thank you for your interest in the FeTS Challenge!

I can help out with a few responses:

1. Training dataset is first split into 80%-20% for train-val, then train part is distributed to the collaborators according to their partitioning info, is that correct?

Yes, that is correct.

From the documentation:

The tags include unstructured information that can be used to create complex data flows. For example, model layer weights will have the same tensor_name and round before and after training, so a tag of trained is used to denote that the numpy array corresponds to the layer of a locally trained model. This field is also used to capture metric information. For example, aggregated_model_validation assigns tags of metric and validate_agg to reflect that the metric reported corresponds to the validation score of the latest aggregated model, whereas the tags of metric and validate_local are used for metrics produced through validation after training on a collaborator's local data.

4. At round-0 when we see a tag of (metric, validate_local, 7), it refers to the validation set result of locally trained model on collaborator 7. Then, what does (metric, validate_local) refers to?

The tags of 'metric' and 'validate_local' are used for metrics produced through validation after training on a collaborator's local data.

5. I think (metric, validate_agg) and (metric, validate_agg, COLLABORATOR_ID) for the same metric will always be equal right?

Yes, I believe so. Pinging @brandon-edwards, @psfoley and @alexey-gruzdev for clarification.

6. What about locally_tuned_model_validation?

Once the aggregated model is received on each collaborator's site, it is tuned on that sites data to improve performance. This results in the creation of the locally_tuned_model, and its validation score is recorded under locally_tuned_model_validation.

Hope this helps! And please open a separate discussion thread for any question/answer that wasn't clear, so that we can keep the Q&A separate and not crowd this. 😄

Cheers,
Sarthak

[GorkemP]

Thank you for the answers,

Best.