Skip to content

Comments

Log base CE loss (exclude aux) as train/loss#2726

Open
pc0618 wants to merge 4 commits intomainfrom
fix/log-base-loss
Open

Log base CE loss (exclude aux) as train/loss#2726
pc0618 wants to merge 4 commits intomainfrom
fix/log-base-loss

Conversation

@pc0618
Copy link
Contributor

@pc0618 pc0618 commented Feb 10, 2026

Changes logging so W&B train/loss reflects base next-token CE (total minus aux), while still optimizing and logging total loss and aux separately.

  • Add compute_next_token_loss_terms() to expose (base_loss, aux_loss)
  • Train loops log train/base_loss, train/aux_loss, train/total_loss
  • Trainer uses train/base_loss for StepInfo.loss so the default hook logs base CE as train/loss

Tests: uv run --with pytest pytest lib/levanter/tests/test_lm_model_loss.py -q

@dlwh
Copy link
Member

dlwh commented Feb 10, 2026

this is two distinct changes, can you split

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants