How to Think About GPUs | How To Scale Your Model

[jacobaustin123]

Here's some discussion about the GPU section

[FurkanGozukara]

Who can implement what is written here probably making 5m a year tell me if I am right or wrong 😁

What a great resource for free

[tcsmaster]

You're wrong

[FurkanGozukara]

@tcsmaster wow sad i would expect such payment

[Davidnet]

This resource, in conjunction with the comprehensive insights from David A. Patterson and John L. Hennessy's book on computer architecture, has been invaluable. It has significantly enhanced my understanding of TPUs and machine learning systems, enabling me to perform rapid back-of-the-envelope calculations for system design, which I thought you could only learn privately from ML Labs. thanks!

This has made me a much better differential programmer

[KonstantinWilleke]

Thank you for sharing this 🙏

[chang-joon-won]

for me beginner. thanks so much. i bookmark before reading. ^^

[manishucsd]

Can you check if there is anything like B100 as in the first picture? I think there is B200, GB200, and GB300?

[tas-cyber]

How does all of this compare against AMDs MI300x? Possible to update post to include?

[joshbav]

This was an excellent writeup. Thank you.

[LeonEricsson]

Question on the derivations in Reductions when array is sharded over a separate axis.

For the case $Y < D$, we get

$$ T_{\text{total}} =\frac{1}{Y}\max\left(\frac{B D}{W_{\text{gpu}}},,\frac{B N}{W_{\text{node}}}\right), $$

which means time decreases proportional to the $Y$ which doesn't seem to line up with the written conclusion.

On the other hand, when $Y > D$ we get

$$ T_{\text{total}}=\max\left(\frac{B}{W_{\text{gpu}}},\frac{B N}{Y W_{\text{node}}}\right) $$

with a crossover point at $\hat Y = 0.88N$. This implies that when $Y < \hat Y = 0.88N$, our scale-out term dominates and $T_{\text{total}} \propto 1/Y$, and when our sharding Y is across more than 88% of our GPUs N, then the intra-node term caps the runtime at $B/W_{gpu}$. So that seems to match the conclusion written but up to a certain point.

Also, separate from the above, shouldn't the scale-out term use the number of nodes rather than the number of GPUs?

$$T_\text{comms in scale-out network} = \frac{\text{bytes} \cdot M}{Y \cdot W_\text{node egress}}$$

[jacobaustin123]

Hi Leon, thanks for pointing this out. I think the formula below this is correct, but I think this given equation is wrong. Let me know if this looks like it's correct.

[LeonEricsson]

Looks good I think.

(sanity check) For $Y < D$ we get

$$ T_{\text{scale-out network}} > T_{\text{node}} \leftrightarrow \frac{B}{W_{\text{node egress}}} > \frac{B}{W_{\text{gpu egress}}} * \frac{1}{Y} \leftrightarrow Y > \frac{W_{\text{node egress}}}{W_{\text{gpu egress}}} = 0.88 $$

which is always true so the scale-out term dominates and sharding does not reduce time.

For $Y > D$

$$ T_{\text{scale-out network}} > T_{\text{node}} \leftrightarrow \frac{B}{W_{\text{node egress}}} * \frac{D}{Y} > \frac{B}{W_{\text{gpu egress}}} * \frac{1}{D} \leftrightarrow Y < \frac{W_{\text{gpu egress}}}{W_{\text{node egress}}} * D^2 = 72 $$

which means sharding is efficient up to 72 GPUs (9 nodes), where runtime decreases ∝ 1/Y. Beyond that, intra-node costs dominate and further sharding provides no benefit.

[jacobaustin123]

That sounds correct to me. I admit despite having written this section that I don't have much practical experience with this, so it may be off by some amount. In particular I think it's kind of hard to achieve full InfiniBand bandwidth, so the node egress bandwidth may be lower in practice.

[tiger83777]

The Tcomm with NVlinkSharp should be B/W instead of N/W. There seems to be a trivial typo there.

[jeffreywpli]

Thanks for the great write-up! Have some questions about quiz 5 that it would be great if you could please help clarify:

With ZeRO-3 and 8-way TP, we’ll be doing 512-way ZeRO-3. This won’t have any issue with memory because we’re sharding everything aggressively. We’ll have a per-GPU batch size of 4e6 / 4096 = 976. This is quite low, even below our pure DP limit, and this is twice that limit because we have to move our weights. So no.

1. Why is the per-GPU batch size here 4e6 / 4096 instead of 4e6 / 512?

2. Also, can you elaborate on why the pure DP limit is different from ZeRo-3? I thought that pure-DP and FSDP shared the same total communication cost.

For both the 70B and 314B though, because we have 16 and 64-way model sharding respectively, we get 2x and 8x better throughput at the spine level, which means we should be compute-bound at roughly 1k and 300 tokens / step respectively.

Why is it that only the amount of pipeline parallelism affects roofline and not the 8-way TP? Does it have to do with the fact that the TP is only within each node?