Any support for A100

[FengzhuoZhang]

Since there is no FP8 in A100, do you have any plan for supporting the training on A100?

[trianxy]

For me, it ran on an A100 after setting fp8 to False + reducing train_seq_len and val_seq_len to avoid out of memory errors.

If you go from 8 to 1 GPU you may also want to implement gradient accumulation if you want it to converge faster.

[BradKML]

@trianxy what more can be done for RTX 40-series GPU for hobbyist acceleration + AI scientist 2 / AlphaEvolve ?