[RFC] on-the-fly packing#2819

Open

felipemello1 wants to merge 57 commits intometa-pytorch:impl-step-based-ckptfrom

felipemello1:online_packing

Commits on Jun 4, 2025

rfc
Felipe Mello
committed
Revert "rfc"
Felipe Mello
committed

Commits on Jun 6, 2025

Merge branch 'main' of https://github.com/pytorch/torchtune
Felipe Mello
committed

Commits on Jun 11, 2025

Merge branch 'main' of https://github.com/pytorch/torchtune
Felipe Mello
committed
add packed functions
felipemello1
committed
enable on full recipe
felipemello1
committed
fix imports + formatting
Felipe Mello
committed
add max_steps_per_epoch requirement
felipemello1
committed
address blockers
Felipe Mello
committed
Merge branch 'main' into online_packing
Felipe Mello
committed
Merge branch 'main' into online_packing
Felipe Mello
committed

Commits on Jun 25, 2025

Commits on Jun 26, 2025

it works
Felipe Mello
committed
remove code
Felipe Mello
committed
Merge branch 'iterable_dataset_final' into online_packing
Felipe Mello
committed
adjust pack to have metrics
Felipe Mello
committed
remove comment
Felipe Mello
committed

Commits on Jul 3, 2025

fix nested interleave
felipemello1
committed
changes to TuneIterableDataset
felipemello1
committed
add IterableDataset back
Felipe Mello
committed

Commits on Jul 6, 2025

Commits on Jul 7, 2025

error if duplicated metric name
Felipe Mello
committed
improve docs
Felipe Mello
committed
Merge branch 'iterable_dataset_final' into online_packing
felipemello1
committed
rename from strategy to packer
felipemello1
committed
tensors instead of lists
felipemello1
committed
tests
felipemello1
committed
docs
felipemello1
committed
tests + lint pass
Felipe Mello
committed
test collate + dataloader
felipemello1
committed
clean up
Felipe Mello
committed

Commits on Jul 8, 2025

improve packed testing
Felipe Mello
committed