Use float32 for LoRA weights to avoid the risk of underflow and overflow.

[james77777778]

As reported in keras-team/keras-hub#2629

We should use high precision (float32) for LoRA weights to stabilize the finetuning.

References:

• https://huggingface.co/docs/peft/developer_guides/troubleshooting (autocast_adapter_dtype)
• Adapters saved in float16 are loaded in float32 huggingface/peft#2421
• https://github.com/huggingface/peft/blob/c75485a2144a15a526e0290835a7439daefc3925/src/peft/tuners/lora/layer.py#L966-L968 (huggingface/peft impl)

[gemini-code-assist]

Code Review

This pull request updates the LoRA implementation across several layers—including Convolutional, Dense, EinsumDense, and Embedding—to ensure that LoRA weights are initialized as float32 to prevent numerical instability. It also introduces explicit casting to the appropriate variable or compute dtypes during kernel composition and forward passes. A critical issue was identified in the EinsumDense layer where a trailing comma incorrectly converts the LoRA update into a tuple, which will cause a TypeError during tensor operations.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.27%. Comparing base (ebb7e78) to head (28a8fe9).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #22559      +/-   ##
==========================================
+ Coverage   77.33%   83.27%   +5.94%     
==========================================
  Files         596      596              
  Lines       67828    67835       +7     
  Branches    10562    10562              
==========================================
+ Hits        52452    56487    +4035     
+ Misses      12612     8605    -4007     
+ Partials     2764     2743      -21     

Flag	Coverage Δ
keras	83.09% <100.00%> (+5.89%)	⬆️
keras-jax	59.82% <100.00%> (+<0.01%)	⬆️
keras-numpy	54.43% <46.15%> (-0.01%)	⬇️
keras-openvino	51.70% <46.15%> (-0.01%)	⬇️
keras-tensorflow	61.14% <100.00%> (?)
keras-torch	60.00% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.