feat: DataProcessor v1#381

Merged

willmj merged 17 commits intofoundation-model-stack:mainfrom

dushyantbehl:dataloader-v2-impl

Dec 3, 2024

Commits on Dec 3, 2024

Add initial implementation of dataloader v1
dushyantbehl
committed
tests: reformat mock.patch to inside unit tests

willmj
authored and
dushyantbehl
committed
Add data config argument to data preprocessor
dushyantbehl
committed
fix: Changes to support current implementation

Abhishek-TAMU
authored and
dushyantbehl
committed
Ensure data handling is done within process dataargs
dushyantbehl
committed
Remove accelerator in favor of torch distributed check for multi node
dushyantbehl
committed
Refactor data util tests as data handler tests.
dushyantbehl
committed
fix: add __init__.py to add tuning.data to python package

willmj
authored and
dushyantbehl
committed
fix: multi GPU prepare training dataset

willmj
authored and
dushyantbehl
committed
fix: lint

willmj
authored and
dushyantbehl
committed
fix: Add TODO

willmj
authored and
dushyantbehl
committed
test: add test for process_dataset_configs in HFBasedDataPreProcessor

willmj
authored and
dushyantbehl
committed
add: test cases for framework

Abhishek-TAMU
authored and
dushyantbehl
committed
fix: update function name get_dataprocessor->get_datapreprocessor

willmj
authored and
dushyantbehl
committed
Rename loader to processor
dushyantbehl
committed
data folders should be together
dushyantbehl
committed
Add code comments and make code path clearer.
dushyantbehl
committed