Multiple datasets

[JPXKQX]

A few months ago, we started with a working group to support multiple datasets in Anemoi. During this time, we have gathered the user requirements, drafted some architectural designs and have now started the implementation.

Below, we have divided the work into smaller units, which are documented under Milestone 1.

This discussion aims to bring the implementation discussion into the open space to give visibility to the community.

[JPXKQX]

Discussion around Issue: DataHandler I

This is the first issue of the Multiple Datasets milestone, which focuses on implementing the proposed design to train a global forecast model with only 1 dataset.

Please use this thread to

• Volunteer to work on it.
• Update on the status of the implementations.
• Share concerns about the implementation

[JPXKQX]

Current status

Thanks to the work by @havardhhaugen, @floriankrb, and all the people involved in the discussion, we now have initial implementations for each major part of the pipeline in refactor/multiple-datasets:

• Dataset handling (batch is a dict)
• Model integration (dict in, dict, out)
• Training step

Next steps

The next step is to integrate these three components into the training pipeline.
First, we need to define how information is shared between the dataset/data handlers and the model. Currently, this was handled with the data_indices object, but there are some open questions around this.

[JPXKQX]

The diagram of tasks has been updated.

The first issue has now been closed.