test_jit_general.py::test_litgpt_variants[cuda-codellama2-like] fails #2569
Description
Repro (requires pip install pytest-repeat)
pytest thunder/tests/test_jit_general.py -k litgpt_variants[cuda-codellama2-like --count 10
Failure
for param1, param2 in zip(reference.parameters(), model.parameters()):
assert param1 is not param2
assert param1.grad is not None
> torch.testing.assert_close(param1.grad, param2.grad, rtol=1e-2, atol=1e-2)
E AssertionError: Tensor-likes are not close!
E
E Mismatched elements: 3 / 12288 (0.0%)
E Greatest absolute difference: 0.016535282135009766 at index (137, 17) (up to 0.01 allowed)
E Greatest relative difference: 0.13542601466178894 at index (137, 10) (up to 0.01 allowed)Image - pjnl-20251001