Skip to content

Add basic GH200 hardware description #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 22, 2025
Merged

Add basic GH200 hardware description #118

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d00de38
node-images
bcumming May 22, 2025
1c1b720
add mi300 node diagram
bcumming May 22, 2025
7b2844c
add under-construction admonition; document custom admonitions in con…
bcumming May 22, 2025
a150c2e
Update docs/alps/hardware.md
bcumming May 22, 2025
446e6cd
Update docs/alps/hardware.md
bcumming May 22, 2025
386392f
Update docs/alps/hardware.md
bcumming May 22, 2025
d4a13e1
pr comments
bcumming May 22, 2025
8ed4257
merge
bcumming May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions docs/alps/hardware.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,20 +40,36 @@ Alps was installed in phases, starting with the installation of 1024 AMD Rome du

There are currently four node types in Alps, with another becoming available in 2025:

| type | blades | nodes | CPU sockets | GPU devices |
| ---- | ------:| -----:| -----------:| -----------:|
| NVIDIA GH200 | 1344 | 2688 | 10,752 | 10,752 |
| AMD Rome | 256 | 1024 | 2,048 | -- |
| NVIDIA A100 | 72 | 144 | 144 | 576 |
| AMD MI250x | 12 | 24 | 24 | 96 |
| AMD MI300A | 64 | 128 | 512 | 512 |
| type | abbreviation | blades | nodes | CPU sockets | GPU devices |
| ---- | ------- | ------:| -----:| -----------:| -----------:|
| NVIDIA GH200 | gh200 | 1344 | 2688 | 10,752 | 10,752 |
| AMD Rome | zen2 | 256 | 1024 | 2,048 | -- |
| NVIDIA A100 | a100 | 72 | 144 | 144 | 576 |
| AMD MI250x | mi200 | 12 | 24 | 24 | 96 |
| AMD MI300A | mi300 | 64 | 128 | 512 | 512 |

[](){#ref-alps-gh200-node}
### NVIDIA GH200 GPU Nodes

!!! todo
There are 24 cabinets, in 4 rows with 6 cabinets per row:

* 8 chassis per cabinet
* 7 blades per chassis
* a chassis can contain up to 8 blades, however Alps' gh200 chassis are underpopulated so that we can increase the amount of power delivered to each node.
* 2 nodes per blade

Each node contains four Grace-Hopper modules and four corresponding network interface cards (NICS) per blade, as illustrated below:

Blanca Peak
![](../images/alps/gh200-schematic.svg)

??? info "node xnames"
There are two boards per blade with one node per board.
This is different to the `zen2` CPU-only nodes (used for example in Eiger) that had two nodes per board for a total of four nodes per blade.
As such, there are no `n1` nodes in the xname list, e.g.:
```
x1100c0s6b0n0
x1100c0s6b1n0
```

[](){#ref-alps-zen2-node}
### AMD Rome CPU Nodes
Expand All @@ -79,6 +95,8 @@ Bard Peak
[](){#ref-alps-mi300-node}
### AMD MI300A GPU Nodes

![](../images/alps/mi300-schematic.svg)

!!! todo

Parry Peak
Loading