Training stability loss spike recovery#558

Open

Sualeh77 wants to merge 19 commits intostagingfrom

training_stability-loss_spike_recovery

Commits on Feb 28, 2026

added triton dep for linux
firekind
committed

Commits on Mar 2, 2026

updated ci
firekind
committed

Commits on Mar 4, 2026

feat: Implement background prefetching from S3 for BinIdx dataloader (#557)

hemanth346
and
hemanth-reddy-k
authored

Commits on Mar 5, 2026

Commits on Mar 6, 2026

Included grad norm in loss spiking action mechanism. Also added embedding norm tracking for the task : Embedding norms tracked separately (embeddings can diverge)
sualehqureshi-tomtom
committed
Updated loss spike action mechanism, considering the distributive training made the auto action taking as prior choice than user intervention
sualehqureshi-tomtom
committed
Added reset logic to reset the sliding window of Loss spike tracking
sualehqureshi-tomtom
committed
Added Read Me file to explain the approach, and list down the changes made
sualehqureshi-tomtom
committed
Added local_spike_recovery_test.py file to test loss spike detection and action triggerring mechanism with wikitext data and smll gpt model
sualehqureshi-tomtom
committed