IEEE 754 - float binary notation and JSON

[The-Futurist]

In the IEEE 754 spec, they explain that implementations should support the conversion to/from binary (hex) float notation. One of the justifications for that is the faithful transmission of binary floating point numbers.

In this assignment:

double number = 1.0000000000000002220446049250313080847263336181640625;

The actual float binary value that that corresponds to is 0x1.0000000000001p+0 - as you can see this is shorter and clearer insofar as the actual underlying 64-bit data looks like, but the only way to accurately express that in decimal is the long literal above.

Now when serializing such a value to JSON, the MS docs seem to say that this can be done using "r" or "R" format specifiers, but if we format that above number to a string with "r" this is what we get:

1.0000000000000002

and this is what would be in any serialized JSON data, but it's not the same value.

So would it not be a good idea to define a new format specifier (or update the options for JSON serialization of float/double) that causes the value to be optionally converted to a hex float string rather that a decimal float string?

That way we can be confident that values transmitted are a 100% faithful representation of the 64-bit datum rather a decimal expansion that has truncation/rounding complications?

Here's the bit about this from the IEEE spec:

[tannergooding]

and this is what would be in any serialized JSON data, but it's not the same value.

It is the same value. R produces the "shortest roundtrippable string". That is, the shortest string that when parsed by an IEEE 754 compliant parser will produce the bitwise identical result. This is never more than 17 digits for double (and 9 digits for float), but may be less in many cases.

#1630 is what tracks adding support for parsing/formatting using the IEEE 754 hexadecimal based format. The term binary here notably refers to the raw underlying bits comprised of the sign, exponent, and significand.

JSON, by spec, notably does not support the hexadecimal based character sequence format for its numbers, so this wouldn't be usable there by default even if it were added. You'd have to explicitly define it as a string and then have some sort of schema indicating it should parse as a float/double.

[The-Futurist]

and this is what would be in any serialized JSON data, but it's not the same value.

It is the same value. R produces the "shortest roundtrippable string". That is, the shortest string that when parsed by an IEEE 754 compliant parser will produce the bitwise identical result. This is never more than 17 digits for double (and 9 digits for float), but may be less in many cases.

I assigned 1.0000000000000002220446049250313080847263336181640625 to a double then converted it to a string with format "r" and get 1.0000000000000002 which as you can see is not the same value. If I format it as "G999" I get the full value back.

#1630 is what tracks adding support for parsing/formatting using the IEEE 754 hexadecimal based format. The term binary here notably refers to the raw underlying bits comprised of the sign, exponent, and significand.

JSON, by spec, notably does not support the hexadecimal based character sequence format for its numbers, so this wouldn't be usable there by default even if it were added. You'd have to explicitly define it as a string and then have some sort of schema indicating it should parse as a float/double.

[tannergooding]

I assigned 1.0000000000000002220446049250313080847263336181640625 to a double then converted it to a string with format "r" and get 1.0000000000000002 which as you can see is not the same value. If I format it as "G999" I get the full value back.

This is not how you compare equality of floating-point values. As you can see from the below, these parse to the same value and have the same bitwise representation.

---

This is for the same reason you cannot do something like 1.0 vs 1 vs 1.000 vs 10e-1 and so on. They all represent the same value, even though the string representation differs. As such, you must parse the string to a double and check the double value represented is the same.

Almost nothing serializes the "full" underlying decimal value because this can be up to 767 significant digits and you need no more than 17 to guarantee roundtripping. In many cases you need fewer and preserving bandwidth is important.

[The-Futurist]

Thanks for clarifying, much appreciated. So the double 1.0000000000000002 approximates to binary 0x1.0000000000001p+0 but that binary when expanded to decimal and allowing for many more places, actually corresponds to 1.0000000000000002220446049250313080847263336181640625.

This makes sense now, I see that "round tripping" ensures that the same underlying 64 bit number is preserved despite the decimal only approximating the value.

It's a shame that JSON is such a limited format, be neat if it could understand 0x1.0000000000001p+0 as readily as it understands 1.00034e+3.

[Clockwork-Muse]

It's a shame that JSON is such a limited format, be neat if it could understand 0x1.0000000000001p+0 as readily as it understands 1.00034e+3.

Why should it? Strictly speaking JSON doesn't assign a "type" to number fields, so may not be backed by anything that would reasonably understand or represent such a value.

[The-Futurist]

It's a shame that JSON is such a limited format, be neat if it could understand 0x1.0000000000001p+0 as readily as it understands 1.00034e+3.

Why should it? Strictly speaking JSON doesn't assign a "type" to number fields, so may not be backed by anything that would reasonably understand or represent such a value.

This 0x1.0000000000001p+0 form, is defined in the IEEE standard and is readily recognized as a floating point binary number, given that float and double are floating point binary types, give me a good reason why we'd support 1.234e+3 and not 1.3F5Dp+2 ? Same goes for simple hex and octal notations.

The whole round-trip stuff becomes trivial because float and double are a binary format and easily represented using the float binary notation, instead we create an approx decima expansion and have to ensure it contains enough digits to fully map to the binary.

It's dumb to take binary then convert it to variable length decimal string only to then convert it back to binary!

Json "understands" scientific (decimal) notation and could also have understood binary float notation if anyone had thought about that. I mean a computer information serialization and data exchange mechanism that only understand decimal notation? it's dumb.

What about fixed binary too? stuff like: 1011010011.1011 how can that be sent in Json? as 723.6875 so its dumb, Json really could have supported these common industry standard notations, but it's dumb.

[Clockwork-Muse]

We might support in in C#, in things like double.Parse. It's not supported in JSON, which (formally) has very limited support for representations; note that neither hex nor octal values are supported either.

And I appear to have not made my point clear: When you're dealing with a JSON number field, the type it's understood by on the computer language side may not be float, double, or any binary floating point defined by an IEEE standard. JSON imposes no such requirements at all. It could be an int, decimal, a BCD type (often found on legacy systems) or anything else, none of which have any understanding of this representation.

It's dumb to take binary then convert it to variable length decimal string only to then convert it back to binary!

At least part of the point of JSON is to be human readable, so the current representations are the obvious solution. If that's not a consideration, you'd just use the original binary representation, not encode it to text at all (eg, something like protobuf), because encoding it to any textual representation takes more effort than leaving it in the raw binary form.

What about fixed binary too? stuff like: 1011010011.1011 how can that be sent in Json? as 723.6875 so its dumb, Json really could have supported these common industry standard notations, but it's dumb.

You'd need a way to differentiate those two values, because otherwise the first looks like a valid base-10 representation like the second. But why would you bother? The base-10 form is almost always going to be a shorter string than the base-2 form, for what should be obvious reasons.