Skip to content

Add t0 scripts #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed

Add t0 scripts #50

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
94adbdd
Add t0 scripts
Muennighoff Jul 4, 2022
3b8cc31
Add T0 specific args
Muennighoff Jul 4, 2022
058e60b
Remove warmup
Muennighoff Jul 4, 2022
d93e169
Update train/t0/tr11f-6B3-ml-t0.slurm
Muennighoff Jul 4, 2022
78791e2
t0 -> tr13
Muennighoff Jul 4, 2022
094a273
t0 -> tr13
Muennighoff Jul 4, 2022
2fc43b0
Use weighted-split-path
Muennighoff Jul 5, 2022
41c1ef4
Add 350M script & adjust HPs
Muennighoff Jul 9, 2022
877158b
Adjust tr13f-350M
Muennighoff Jul 9, 2022
5674c69
Add weight decay based on FLAN
Muennighoff Jul 10, 2022
582ee32
Remove finetune & add checkpoint-activations
Muennighoff Jul 11, 2022
d5b1622
Remove finetune
Muennighoff Jul 11, 2022
56729d3
Change BS for more throughput & increase LR
Muennighoff Jul 14, 2022
318dfc7
Update train/tr13-t0/t0_test.slurm
Muennighoff Jul 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions train/t0/tr11f-6B3-ml-t0.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,6 @@ SEQ_LEN=2048
SAVE_INTERVAL=500

TRAIN_SAMPLES=6_400_000 # 13e9 / 2048
LR_WARMUP_SAMPLES=640_000 # 10% - TODO: T0 paper says nothing about warmup

# T0 paper:
# "...we use a learning rate of 1e-3..."
Expand All @@ -80,7 +79,6 @@ OPTIMIZER_ARGS=" \
--adam-eps 1e-8 \
--lr 1e-3 \
--lr-decay-style constant \
--lr-warmup-samples $LR_WARMUP_SAMPLES \
Copy link
Member

@thomasw21 thomasw21 Jul 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used Adafactor ... so technically I don't know what parameters matter (typically we used a decay argument, which I don't know how it translates to Adam optimizer)

--clip-grad 1.0 \
--weight-decay 1e-1 \
"
Expand Down