@JohannesGaessler The OPT_STEP_ADAMW test produced a NaN in the CPU reference test:

[OPT_STEP_ADAMW] NaN at index 0 (CUDA0=0.051476 CPU=-nan) ggml_backend_cuda_graph_compute: disabling CUDA graphs due to GPU architecture

https://github.com/ggml-org/ci/blob/05d107abc73d309ef1374fe13092815cb7bfa254/llama.cpp/19/d012e78a699c4df8c7f1ac325db4632a710d66/ggml-4-x86-cuda-v100/stdall#L9496

The changes to the CPU implementation for the AdamW optimizer weren't synced. This PR still has the previous implementation where the optimizer parameters were stored differently and as a result an integer is interpreted as a float, leading to garbage results.

Ah sorry. My mistake, I forgot to update the sync scripts.