You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tutorials/pretrain.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -81,7 +81,8 @@ litgpt pretrain pythia-14m \
81
81
--tokenizer_dir EleutherAI/pythia-14m \
82
82
--data TextFiles \
83
83
--data.train_data_path custom_pretraining_data \
84
-
--train.lr_warmup_steps=200
84
+
--train.lr_warmup_steps=200 \
85
+
--optimizer AdamW \
85
86
--optimizer.lr 0.005
86
87
```
87
88
@@ -121,7 +122,7 @@ The following subsections illustrate three typical scenarioes:
121
122
For instance, let's assume we download a Pythia model:
122
123
123
124
```bash
124
-
litgpt download EleutherAI/pythia-14m
125
+
litgpt download EleutherAI/pythia-160m
125
126
```
126
127
127
128
Next, assume we have a custom dataset stored in text files similar to the *Pretrain on custom data* above. We can further pretrain the Pythia model via the `--initial_checkpoint_dir` setting as follows:
0 commit comments