Skip to content

Floating Point Precision Issues

LenkaNovak edited this page Apr 26, 2023 · 8 revisions

We aim to use Float64 and Flat32 precision in CliMA's ESM. Here is a summary of some challenges / things to be aware of:

Basics (IEEE 754 standard)

  • Float32 (single precision): a 32-bit float can represent up to 7 decimal numbers
    • 1 sign bit, 8 exponent bits, 23 mantissa/fraction bits] $$ (-1)^{sign} \times M \times 2^{e-bias} $$
  • Float64 (double precision):

eps() in Julia

julia> eps(zero(Float64))
5.0e-324

julia> eps(one(Float64))
2.220446049250313e-16

julia> eps(zero(Float32))
1.0f-45

julia> eps(one(Float32))
1.1920929f-7

see Julia docs for more info.

Clone this wiki locally