feat: introduce Multi-Storage Client (MSC) as a storage provider

[shunjiad]

Summary

Adds NVIDIA Multi-Storage Client (MSC) as an optional storage provider so DLIO can run data generation, training I/O, and PyTorch checkpointing against MSC-resolved backends (for example object storage such as S3, GCS, Azure, etc), consistent with existing local_fs, parallel_fs, s3, and aistore modes.

MSC is integrated for the indexed binary dataset format and PyTorch checkpointing.

Configuration examples

1. MSC client configuration

MSC is configured in its own YAML file; see the Configuration Reference. A minimal S3 profile looks like this:

profiles:
  experiments:
    storage_provider:
      type: s3
      options:
        base_path: my-bucket
        region_name: us-east-1

Point MSC at your config file with the MSC_CONFIG environment variable (for example in the shell before running DLIO):

export MSC_CONFIG=/path/to/msc_config.yaml

2. DLIO workload profile (Hydra YAML)

In the workload, set storage_type to msc and set storage_root to an MSC URI. The segment immediately after msc:// is the profile name (a key under profiles: in your MSC config, such as experiments in the S3 example above); the following path is the prefix for objects under that profile. DLIO passes storage_root to multistorageclient.resolve_storage_client(), so dataset and checkpoint paths combine with this prefix and are served through MSC.

storage:
  storage_type: msc
  storage_root: msc://experiments/megatron-deepspeed/

For this snippet, experiments must be a profile defined in your MSC configuration file; megatron-deepspeed/ is the artifact prefix under that profile.