Skip to content

Commit b7a213b

Browse files
authored
Merge branch 'master' into auto-fix-rules
2 parents 0c6bd41 + 56204d8 commit b7a213b

File tree

2 files changed

+9
-1
lines changed

2 files changed

+9
-1
lines changed

docs/source-pytorch/common/precision_basic.rst

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,14 @@ However, this setting can sometimes lead to unstable training.
3939
4040
Trainer(precision="16-true")
4141
42+
.. warning::
43+
44+
Float16 cannot represent values smaller than ~6e-5. Values like Adam's default ``eps=1e-8`` become zero, which can cause
45+
NaN during training. Increase ``eps`` to 1e-4 or higher, and avoid extremely small values in your model weights and data.
46+
47+
.. note::
48+
49+
BFloat16 (``"bf16-mixed"`` or ``"bf16-true"``) has better numerical stability with a wider dynamic range.
4250

4351
----
4452

requirements/pytorch/test.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ numpy >1.20.0, <1.27.0
1313
onnx >1.12.0, <1.20.0
1414
onnxruntime >=1.12.0, <1.24.0
1515
onnxscript >= 0.1.0, < 0.5.0
16-
psutil <7.1.1 # for `DeviceStatsMonitor`
16+
psutil <7.1.2 # for `DeviceStatsMonitor`
1717
pandas >2.0, <2.4.0 # needed in benchmarks
1818
fastapi # for `ServableModuleValidator` # not setting version as re-defined in App
1919
uvicorn # for `ServableModuleValidator` # not setting version as re-defined in App

0 commit comments

Comments
 (0)