Use just 1 lambda for input on layer 0

[kroggen]

This line here appears strange:

modded-nanogpt/train_gpt.py

Line 1329 in e22a34b

x = (resid_lambdas[0] + x0_lambdas[0]) * x + bigram_lambdas[0] * x0_bigram

Does it really need 2 lambdas for input x on layer 0?

What about using just 1 lambda?

x = x0_lambdas[0] * x + bigram_lambdas[0] * x0_bigram 

[ClassicLarry]

It is strange, but when I tested a while back it performed better this way. It also performed better than 2*resid_lambdas[0]. There are some differences in the effective lr and Adam behavior. We could probably drop it if we tuned the lr and betas for resid_lambda[0] by itself, but I felt singular-lambda tuning was overkill.