A speedrun on consumer grade cards?

[fzyzcjy]

Hi thanks for the great repo! I would appreciate it if there can be a speed run on consumer cards e.g. RTX4090. Since it is 125M params, the RTX4090's 24GB memory should fit in the classical way, and thus it is trainable.

[KellerJordan]

a suggestion, to reduce memory you could run with a lower sequence length

[fzyzcjy]

I think so, thanks :) Just wondering whether there will be a speedrun like the current great one but focus on RTX4090's time, because many more people have consumer grade cards than H100s.

[naoro]

I think a Google colab speed runs would be also awesome -
That would greatly commodotize research and experimentation

[fzyzcjy]

That looks interesting! (But I guess it may be too hard to have it run in acceptable times...)

[alexjc]

Realistically single-card speed-run would need a smaller model too, otherwise it's too slow to experiment.

Thinking:

n_layer = 8
n_embd = 512

The sequence length during training has been a variable factor in the last speed run; for evaluation it's fine if it's the whole document clamped at maximum 1024 window size?

[fzyzcjy]

It seems H100 has 2000 TFLOPS for bf16 tensor core, while 4090 is about 330 TFLOPS. Thus 8xH100 5minute = 1x4090 4hour, which looks not bad!

The major problem looks like the memory is only 24GB... So we may not be able to do some optimizations.

[alexjc]

4h feels quite high for a speed run. Too hard to test ideas, no?

Working on some memory optimizations now, should help a lot...

[fzyzcjy]

Surely faster would be great! But if impossible, then 4h is better than nothing :(

[naoro]

It seems H100 has 2000 TFLOPS for bf16 tensor core, while 4090 is about 330 TFLOPS. Thus 8xH100 5minute = 1x4090 4hour, which looks not bad!

The major problem looks like the memory is only 24GB... So we may not be able to do some optimizations.

A100 has got 40GB, cost is about 10usd for ~12 hours with about the same tflops as 4090.
So about 3.3usd per run. Not bad, I'd think.

[fzyzcjy]

If we optimize for cost, then 4090 is much cheaper than A100 per hour while having same tflops. Thus as long as we manage to fit in 24GB, then maybe we can further scale down the cost.

[KellerJordan]

A note: The current cost per run on an 8xH100 is about $1.90 (since it's about $3/hr for SXM H100s)

Personally, when I don't feel like spending that much, I go back to speedrunning CIFAR-10. But I understand that might not be so interesting to everyone

[fzyzcjy]

Looks like 4090 is about $0.3/hr, thus 4hr = $1.2, which looks a bit cheaper. And moreover, some people have 4090s bought in their house (e.g. many people in r/LocalLlama, me, etc), while it seems less people buy A100/H100 in their house, and the bought cost is much cheaper than a cloud.

[lapp0]

I'm also interested in this variant.

Considering the long runtime, perhaps it makes sense to compete to minimize validation loss within a 1 hour run?

[KellerJordan]

I would guess that halving the sequence length (and going to batch size 16) will allow fitting the run into 24G memory, without impacting performance very much. or quartering it, if that doesn't still fit

[lapp0]

I achieved < 3.28 in a little under two hours with a few tweaks.

@KellerJordan are you interested in hosting a 1x4090 variant of the competition in this repo? If so, I'll submit a PR for 4090/train_gpt2_4090.py and 4090/run_4090.sh and update the readme.