Skip to content

Commit f4c5aaf

Browse files
authored
[example] llama3 (#5631)
* release llama3 * [release] llama3 * [release] llama3 * [release] llama3 * [release] llama3
1 parent 4de4e31 commit f4c5aaf

File tree

11 files changed

+26
-4
lines changed

11 files changed

+26
-4
lines changed

README.md

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@
5252
<li>
5353
<a href="#Parallel-Training-Demo">Parallel Training Demo</a>
5454
<ul>
55-
<li><a href="#LLaMA2">LLaMA 1/2</a></li>
55+
<li><a href="#LLaMA3">LLaMA 1/2/3 </a></li>
5656
<li><a href="#MoE">MoE</a></li>
5757
<li><a href="#GPT-3">GPT-3</a></li>
5858
<li><a href="#GPT-2">GPT-2</a></li>
@@ -270,13 +270,21 @@ Acceleration of [AlphaFold Protein Structure](https://alphafold.ebi.ac.uk/)
270270
<p align="right">(<a href="#top">back to top</a>)</p>
271271

272272
## Parallel Training Demo
273+
### LLaMA3
274+
<p align="center">
275+
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/examples/images/LLaMA3-70B-H100.png" width=600/>
276+
</p>
277+
278+
- 70 billion parameter LLaMA3 model training accelerated by 18%
279+
[[code]](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/llama)
280+
273281
### LLaMA2
274282
<p align="center">
275283
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/llama2_pretraining.png" width=600/>
276284
</p>
277285

278286
- 70 billion parameter LLaMA2 model training accelerated by 195%
279-
[[code]](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/llama2)
287+
[[code]](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/llama)
280288
[[blog]](https://www.hpc-ai.tech/blog/70b-llama2-training)
281289

282290
### LLaMA1
@@ -285,7 +293,7 @@ Acceleration of [AlphaFold Protein Structure](https://alphafold.ebi.ac.uk/)
285293
</p>
286294

287295
- 65-billion-parameter large model pretraining accelerated by 38%
288-
[[code]](https://github.com/hpcaitech/ColossalAI/tree/example/llama/examples/language/llama)
296+
[[code]](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/llama)
289297
[[blog]](https://www.hpc-ai.tech/blog/large-model-pretraining)
290298

291299
### MoE

docs/README-zh-Hans.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@
5151
<li>
5252
<a href="#并行训练样例展示">并行训练样例展示</a>
5353
<ul>
54-
<li><a href="#LLaMA2">LLaMA 1/2</a></li>
54+
<li><a href="#LLaMA3">LLaMA 1/2/3</a></li>
5555
<li><a href="#MoE">MoE</a></li>
5656
<li><a href="#GPT-3">GPT-3</a></li>
5757
<li><a href="#GPT-2">GPT-2</a></li>
@@ -261,6 +261,14 @@ Colossal-AI 为您提供了一系列并行组件。我们的目标是让您的
261261
<p align="right">(<a href="#top">返回顶端</a>)</p>
262262

263263
## 并行训练样例展示
264+
### LLaMA3
265+
<p align="center">
266+
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/examples/images/LLaMA3-70B-H100.png" width=600/>
267+
</p>
268+
269+
- 700亿参数LLaMA3训练加速18%
270+
[[code]](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/llama)
271+
264272
### LLaMA2
265273
<p align="center">
266274
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/llama2_pretraining.png" width=600/>

examples/language/llama2/README.md renamed to examples/language/llama/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
11
# Pretraining LLaMA-1/2/3: best practices for building LLaMA-1/2/3-like base models
2+
### LLaMA3
3+
<p align="center">
4+
<img src="https://raw.githubusercontent.com/hpcaitech/public_assets/main/examples/images/LLaMA3-70B-H100.png" width=600/>
5+
</p>
6+
7+
- 70 billion parameter LLaMA3 model training accelerated by 18%
28

39
### LLaMA2
410
<p align="center">
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)