Skip to content

fix: multiple datasets utility#256

Closed
JPXKQX wants to merge 1 commit intomainfrom
feat/multiple-datasets
Closed

fix: multiple datasets utility#256
JPXKQX wants to merge 1 commit intomainfrom
feat/multiple-datasets

Conversation

@JPXKQX
Copy link
Member

@JPXKQX JPXKQX commented Jan 15, 2026

Description

What problem does this change solve?

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@VeraChristina
Copy link
Collaborator

VeraChristina commented Jan 15, 2026

Thanks for moving this! Just two points:

  • Do we also need get_dataset_data_config? For the import Ana found in models/src/anemoi/models/interface/init.py
  • Is there a better place than utils given that it's somewhat specific to how the training config files are set up? (I understand that it needs to be in a dependency of models, so it can't go in training itself, and perhaps the best solution for now is utils, just wondering) cc @gmertes
    edit: how about putting it into models? that's also not ideal, but it's used in models and training, and so it would remain in core?

@JPXKQX
Copy link
Member Author

JPXKQX commented Jan 15, 2026

  • The get_dataset_data_config is deleted now.
  • I could see this function being used for inference, too, to support the same config structure. What are the plans?

@VeraChristina
Copy link
Collaborator

Looks like it's not needed in inference as the approach to handling the different config layouts is via two different subclasses of the Metadata class (which handles this more explicitly, storing relevant metadata entries in cached properties). So I'd have a preference to put it into models

@JPXKQX JPXKQX closed this Jan 15, 2026
@github-project-automation github-project-automation bot moved this from To be triaged to Done in Anemoi-dev Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants