Hexadecimal-encoded constants do not respect `number` configuration setting [was 64-bit numerical calculation error]

[gagmeng]

hex(0x0AF076A6E6E12400 - 0x0112A88517556970)
"0x9ddce21cf8bba80"
The correct answer must be 0x9ddce21cf8bba90.

[gwhitney]

By default, mathjs uses the JavaScript number type (IEEE 64-bit floating point) to represent numerical constants, even hexadecimal-notated integers. This type has only 53 bits of precision for the mantissa, and hence can only exactly represent integers in roughly the range ±2^53. The integers in your example are outside that range, leading to loss of precision and hence an approximate result.

You should be able to switch to JavaScript bigint (arbitrary-precision integer) representation to get a precise answer via:

const math = create(all, {number: 'bigint'})
const res = math.evaluate('hex(0x0AF076A6E6E12400 - 0x0112A88517556970)')

However, there is a bug in that hexadecimal-encoded numeric constants do not appear to respect the number attribute of the configuration. So changing the title of this issue to reflect that bug. I can submit a PR for this bug once my upcoming PR, also related to bigints, is merged.

[josdejong]

Thanks for the clear explanation Glen. I did look into why it doesn't work for bigint, and created a PR that fixes the issue: #3348. Can you maybe have brief look at the PR?

Instead of bigint, using BigNumber will work too since it has a higher precision:

math.config({ number: "BigNumber"})

const res = math.evaluate('hex(0x0AF076A6E6E12400 - 0x0112A88517556970)') 
// 0x9ddce21cf8bba90 (correct)

[josdejong]

@gwhitney I only now see that you've also been working on a fix in #3345, I still have to review it but at first sight it looks like the two PR's are complementary.

[gwhitney]

Yes, I deliberately left this point alone in #3345 so that they could be handled orthogonally. I will review #3348 now.