-
Notifications
You must be signed in to change notification settings - Fork 7
Floating Point Precision Issues
LenkaNovak edited this page Apr 26, 2023
·
8 revisions
We aim to use Float64 and Flat32 precision in CliMA's ESM. Here is a summary of some challenges / things to be aware of:
- Float32 (single precision): a 32-bit float can represent up to 7 decimal numbers
- 1 sign bit, 8 exponent bits, 23 mantissa/fraction bits] $$ (-1)^{sign} \times M \times 2^{e-bias} $$
- Float64 (double precision):
julia> eps(zero(Float64))
5.0e-324
julia> eps(one(Float64))
2.220446049250313e-16
julia> eps(zero(Float32))
1.0f-45
julia> eps(one(Float32))
1.1920929f-7
see Julia docs for more info.