Module support for multiple ptd files

[lucylq]

Stack from ghstack (oldest at bottom):

• -> Module support for multiple ptd files #14158

Support multiple PTD files in Module. This change updates the following private variables in Module:

std::string data_path --> std::unordered_set<std::string> data_files_
std::unique_ptr<DataLoader> data_map_loader --> std::vectror<std::unique_ptr<DataLoader>> data_map_loaders_
std::unique_ptr<NamedDataMap> data_map --> std::vector<std::unique_ptr<NamedDataMap> named_data_maps_

And introduces a new private variable. When we have multiple NamedDataMaps, they need to be merged into one, for use in method, etc. This is not implemented yet.

std::unique_ptr<NamedDataMap> merged_data_map_

The process of using a PTD file is:

std::string file --> wrapped in DataLoader --> wrapped in NamedDataMap.

At each stage we can have multiple.

This diff also introduces a new Module constructor that takes in std::unordered_set<std::string> named_data_map_paths_

Differential Revision: D82059808

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14158

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e6ed284 with merge base f7c009e ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D82059808

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D82059808

[kimishpatel]

Review automatically exported from Phabricator review in Meta.

[facebook-github-bot]

@lucylq has exported this pull request. If you are a Meta employee, you can view the originating Diff in D82059808.

[lucylq]

@pytorchbot cherry-pick --onto release/1.0 -c examples

[pytorch-bot]

❌ 🤖 pytorchbot command failed:

@pytorchbot cherry-pick: error: argument -c/--classification: invalid choice: 'examples' (choose from 'regression', 'critical', 'fixnewfeature', 'docs', 'release')

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Try @pytorchbot --help for more info.