Skip to content

Commit 074a6ef

Browse files
authored
Update link to vocab size increase (rasbt#526)
* Update link to vocab size increase * Update ch05/10_llm-training-speed/README.md * Update ch05/10_llm-training-speed/README.md
1 parent 908dd2f commit 074a6ef

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

ch05/10_llm-training-speed/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,7 @@ After:
169169
 
170170
### 9. Using a nicer vocab size value
171171
172-
- This is a tip suggested to me by my former colleague Carlos Moccholi, who mentioned that this tip comes from Andrej Karpathy (I suspect it's from the [nanoGPT](https://github.com/karpathy/nanoGPT/blob/93a43d9a5c22450bbf06e78da2cb6eeef084b717/model.py#L111) repository)
172+
- Here, we slightly increase the vocabulary size from 50,257 to 50,304, which is the nearest multiple of 64. This tip was suggested to me by my former colleague Carlos Mocholi, who mentioned that it originally came from Andrej Karpathy (likely from [this post](https://x.com/karpathy/status/1621578354024677377)). Karpathy's recommendation is based on [NVIDIA's guidelines on tensor shapes](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#tensor-core-shape), where batch sizes and linear layer dimensions are commonly chosen as multiples of certain values.
173173
174174
Before:
175175
- `Step tok/sec: 112046`
@@ -204,4 +204,4 @@ Before (single GPU):
204204
205205
After (4 GPUs):
206206
- `Step tok/sec: 419259`
207-
- `Reserved memory: 22.7969 GB`
207+
- `Reserved memory: 22.7969 GB`

0 commit comments

Comments
 (0)