From 25c9ad414f64eae56743c6d72a3194b105060e7b Mon Sep 17 00:00:00 2001 From: bhimrazy Date: Thu, 16 Oct 2025 19:33:05 +0545 Subject: [PATCH 1/2] docs: add warnings about Float16 limitations and introduce BFloat16 for better stability --- docs/source-pytorch/common/precision_basic.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/source-pytorch/common/precision_basic.rst b/docs/source-pytorch/common/precision_basic.rst index 1134524b51247..f9acf2ba25dfe 100644 --- a/docs/source-pytorch/common/precision_basic.rst +++ b/docs/source-pytorch/common/precision_basic.rst @@ -39,6 +39,14 @@ However, this setting can sometimes lead to unstable training. Trainer(precision="16-true") +.. warning:: + + Float16 cannot represent values smaller than ~6e-5. Values like Adam's default ``eps=1e-8`` become zero, which can cause + NaN during training. Increase ``eps`` to 1e-4 or higher, and avoid extremely small values in your model weights and data. + +.. note:: + + BFloat16 (``"bf16-mixed"`` or ``"bf16-true"``) has better numerical stability with a wider dynamic range. ---- From ae81a3ff7d5c9a81334c0f5b419f783e019578fd Mon Sep 17 00:00:00 2001 From: jirka Date: Tue, 21 Oct 2025 12:25:32 +0200 Subject: [PATCH 2/2] Empty-Commit