Precision or encoding error storing JS numbers in MySQL DOUBLE

[wesgarland]

This bug is unique to the mysql2 npm package - it does not happen with the mysql npm package.

I have a db schema which declares a column as a DOUBLE; mysql docs say this is an eight byte floating point number, presumably a 64-bit IEEE 754 quantity. JS uses, by definition, IEEE 754-2019 numbers.

When I store certain values in my database and select them back out, I sometimes get different numbers. Once such example is 0.45494253843119004, which gets stored correctly, but selected back out as 0.4549425384311901.

My platform is mysql2@2.3.3, node v12.22.9 on Linux wes-linux-kds 5.4.0-94-generic #106~18.04.1-Ubuntu SMP Fri Jan 7 07:23:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux.

We have an ORM in the way, so I haven't written a minimal test case yet. If the team wants to follow up on this bug, I can produce one. If you want help fixing the issue, point me in the right direction...I am a C and JS programmer with some experience with the mysql and v8 APIs.

This is a complete record of statements executed in our internal test case:

SET NAMES utf8mb4;
SET autocommit=1;
select @@wait_timeout; 
START TRANSACTION;
INSERT INTO jobs (status,lastUpdate,maxSlicesPerTask,duplicationLevel) VALUES (?,?,?,?); [ 'map-test-1646070071709', 1646070071710, 1, 0.45494253843119004 ]
SELECT LAST_INSERT_ID() as id, FOUND_ROWS() as found, ROW_COUNT() as count 
SELECT SQL_CALC_FOUND_ROWS id, address, uuid, status, startTime, endTime, lastUpdate, owner, paymentAccount, lastSliceNumber, nextSliceNumber, maxSlicesPerTask, absoluteSlicePayment, mvMultSlicePayment, duplicationLevel, duplicationType, flags, dataStorageType, dataStorageDetails, resultStorageType, resultStorageDetails, resultStorageParams, requirements, description, link, name FROM jobs WHERE status LIKE 'map-test-%' AND id>=14659 FOR UPDATE 

The callback invoked from mysql2/lib/commands/query.js:86:15 received the following arguments -- the fields object has the wrong value in it:

console.log(results)
[
  {
    id: '14660',
    address: null,
    uuid: null,
    status: 'map-test-1646071286462',
    startTime: null,
    endTime: null,
    lastUpdate: '1646071286463',
    owner: null,
    paymentAccount: null,
    lastSliceNumber: null,
    nextSliceNumber: null,
    maxSlicesPerTask: 1,
    absoluteSlicePayment: null,
    mvMultSlicePayment: null,
    duplicationLevel: 0.4549425384311901,
    duplicationType: null,
    flags: null,
    dataStorageType: null,
    dataStorageDetails: null,
    resultStorageType: null,
    resultStorageDetails: null,
    resultStorageParams: null,
    requirements: null,
    description: null,
    link: null,
    name: null
  }
]
undefined
console.log(fields)
[
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 10,
    _schemaLength: 24,
    _schemaStart: 14,
    _tableLength: 4,
    _tableStart: 39,
    _orgTableLength: 4,
    _orgTableStart: 44,
    _orgNameLength: 2,
    _orgNameStart: 52,
    characterSet: 63,
    encoding: 'binary',
    name: 'id',
    columnLength: 20,
    columnType: 8,
    flags: 16899,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 72,
    _schemaLength: 24,
    _schemaStart: 76,
    _tableLength: 4,
    _tableStart: 101,
    _orgTableLength: 4,
    _orgTableStart: 106,
    _orgNameLength: 7,
    _orgNameStart: 119,
    characterSet: 45,
    encoding: 'utf8',
    name: 'address',
    columnLength: 160,
    columnType: 254,
    flags: 16516,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 144,
    _schemaLength: 24,
    _schemaStart: 148,
    _tableLength: 4,
    _tableStart: 173,
    _orgTableLength: 4,
    _orgTableStart: 178,
    _orgNameLength: 4,
    _orgNameStart: 188,
    characterSet: 45,
    encoding: 'utf8',
    name: 'uuid',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 210,
    _schemaLength: 24,
    _schemaStart: 214,
    _tableLength: 4,
    _tableStart: 239,
    _orgTableLength: 4,
    _orgTableStart: 244,
    _orgNameLength: 6,
    _orgNameStart: 256,
    characterSet: 45,
    encoding: 'utf8',
    name: 'status',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 280,
    _schemaLength: 24,
    _schemaStart: 284,
    _tableLength: 4,
    _tableStart: 309,
    _orgTableLength: 4,
    _orgTableStart: 314,
    _orgNameLength: 9,
    _orgNameStart: 329,
    characterSet: 63,
    encoding: 'binary',
    name: 'startTime',
    columnLength: 20,
    columnType: 8,
    flags: 0,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 356,
    _schemaLength: 24,
    _schemaStart: 360,
    _tableLength: 4,
    _tableStart: 385,
    _orgTableLength: 4,
    _orgTableStart: 390,
    _orgNameLength: 7,
    _orgNameStart: 403,
    characterSet: 63,
    encoding: 'binary',
    name: 'endTime',
    columnLength: 20,
    columnType: 8,
    flags: 0,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 428,
    _schemaLength: 24,
    _schemaStart: 432,
    _tableLength: 4,
    _tableStart: 457,
    _orgTableLength: 4,
    _orgTableStart: 462,
    _orgNameLength: 10,
    _orgNameStart: 478,
    characterSet: 63,
    encoding: 'binary',
    name: 'lastUpdate',
    columnLength: 20,
    columnType: 8,
    flags: 4097,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 506,
    _schemaLength: 24,
    _schemaStart: 510,
    _tableLength: 4,
    _tableStart: 535,
    _orgTableLength: 4,
    _orgTableStart: 540,
    _orgNameLength: 5,
    _orgNameStart: 551,
    characterSet: 45,
    encoding: 'utf8',
    name: 'owner',
    columnLength: 160,
    columnType: 254,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 574,
    _schemaLength: 24,
    _schemaStart: 578,
    _tableLength: 4,
    _tableStart: 603,
    _orgTableLength: 4,
    _orgTableStart: 608,
    _orgNameLength: 14,
    _orgNameStart: 628,
    characterSet: 45,
    encoding: 'utf8',
    name: 'paymentAccount',
    columnLength: 160,
    columnType: 254,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 660,
    _schemaLength: 24,
    _schemaStart: 664,
    _tableLength: 4,
    _tableStart: 689,
    _orgTableLength: 4,
    _orgTableStart: 694,
    _orgNameLength: 15,
    _orgNameStart: 715,
    characterSet: 63,
    encoding: 'binary',
    name: 'lastSliceNumber',
    columnLength: 20,
    columnType: 8,
    flags: 0,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 748,
    _schemaLength: 24,
    _schemaStart: 752,
    _tableLength: 4,
    _tableStart: 777,
    _orgTableLength: 4,
    _orgTableStart: 782,
    _orgNameLength: 15,
    _orgNameStart: 803,
    characterSet: 63,
    encoding: 'binary',
    name: 'nextSliceNumber',
    columnLength: 20,
    columnType: 8,
    flags: 0,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 836,
    _schemaLength: 24,
    _schemaStart: 840,
    _tableLength: 4,
    _tableStart: 865,
    _orgTableLength: 4,
    _orgTableStart: 870,
    _orgNameLength: 16,
    _orgNameStart: 892,
    characterSet: 63,
    encoding: 'binary',
    name: 'maxSlicesPerTask',
    columnLength: 11,
    columnType: 3,
    flags: 0,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 926,
    _schemaLength: 24,
    _schemaStart: 930,
    _tableLength: 4,
    _tableStart: 955,
    _orgTableLength: 4,
    _orgTableStart: 960,
    _orgNameLength: 20,
    _orgNameStart: 986,
    characterSet: 63,
    encoding: 'binary',
    name: 'absoluteSlicePayment',
    columnLength: 33,
    columnType: 246,
    flags: 0,
    decimals: 18
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1024,
    _schemaLength: 24,
    _schemaStart: 1028,
    _tableLength: 4,
    _tableStart: 1053,
    _orgTableLength: 4,
    _orgTableStart: 1058,
    _orgNameLength: 18,
    _orgNameStart: 1082,
    characterSet: 63,
    encoding: 'binary',
    name: 'mvMultSlicePayment',
    columnLength: 33,
    columnType: 246,
    flags: 0,
    decimals: 18
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1118,
    _schemaLength: 24,
    _schemaStart: 1122,
    _tableLength: 4,
    _tableStart: 1147,
    _orgTableLength: 4,
    _orgTableStart: 1152,
    _orgNameLength: 16,
    _orgNameStart: 1174,
    characterSet: 63,
    encoding: 'binary',
    name: 'duplicationLevel',
    columnLength: 22,
    columnType: 5,
    flags: 0,
    decimals: 31
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1208,
    _schemaLength: 24,
    _schemaStart: 1212,
    _tableLength: 4,
    _tableStart: 1237,
    _orgTableLength: 4,
    _orgTableStart: 1242,
    _orgNameLength: 15,
    _orgNameStart: 1263,
    characterSet: 45,
    encoding: 'utf8',
    name: 'duplicationType',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1296,
    _schemaLength: 24,
    _schemaStart: 1300,
    _tableLength: 4,
    _tableStart: 1325,
    _orgTableLength: 4,
    _orgTableStart: 1330,
    _orgNameLength: 5,
    _orgNameStart: 1341,
    characterSet: 63,
    encoding: 'binary',
    name: 'flags',
    columnLength: 10,
    columnType: 3,
    flags: 32,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1364,
    _schemaLength: 24,
    _schemaStart: 1368,
    _tableLength: 4,
    _tableStart: 1393,
    _orgTableLength: 4,
    _orgTableStart: 1398,
    _orgNameLength: 15,
    _orgNameStart: 1419,
    characterSet: 45,
    encoding: 'utf8',
    name: 'dataStorageType',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1452,
    _schemaLength: 24,
    _schemaStart: 1456,
    _tableLength: 4,
    _tableStart: 1481,
    _orgTableLength: 4,
    _orgTableStart: 1486,
    _orgNameLength: 18,
    _orgNameStart: 1510,
    characterSet: 45,
    encoding: 'utf8',
    name: 'dataStorageDetails',
    columnLength: 262140,
    columnType: 252,
    flags: 144,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1546,
    _schemaLength: 24,
    _schemaStart: 1550,
    _tableLength: 4,
    _tableStart: 1575,
    _orgTableLength: 4,
    _orgTableStart: 1580,
    _orgNameLength: 17,
    _orgNameStart: 1603,
    characterSet: 45,
    encoding: 'utf8',
    name: 'resultStorageType',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1638,
    _schemaLength: 24,
    _schemaStart: 1642,
    _tableLength: 4,
    _tableStart: 1667,
    _orgTableLength: 4,
    _orgTableStart: 1672,
    _orgNameLength: 20,
    _orgNameStart: 1698,
    characterSet: 45,
    encoding: 'utf8',
    name: 'resultStorageDetails',
    columnLength: 262140,
    columnType: 252,
    flags: 144,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1736,
    _schemaLength: 24,
    _schemaStart: 1740,
    _tableLength: 4,
    _tableStart: 1765,
    _orgTableLength: 4,
    _orgTableStart: 1770,
    _orgNameLength: 19,
    _orgNameStart: 1795,
    characterSet: 45,
    encoding: 'utf8',
    name: 'resultStorageParams',
    columnLength: 4294967295,
    columnType: 252,
    flags: 144,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1832,
    _schemaLength: 24,
    _schemaStart: 1836,
    _tableLength: 4,
    _tableStart: 1861,
    _orgTableLength: 4,
    _orgTableStart: 1866,
    _orgNameLength: 12,
    _orgNameStart: 1884,
    characterSet: 45,
    encoding: 'utf8',
    name: 'requirements',
    columnLength: 4294967295,
    columnType: 252,
    flags: 144,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1914,
    _schemaLength: 24,
    _schemaStart: 1918,
    _tableLength: 4,
    _tableStart: 1943,
    _orgTableLength: 4,
    _orgTableStart: 1948,
    _orgNameLength: 11,
    _orgNameStart: 1965,
    characterSet: 45,
    encoding: 'utf8',
    name: 'description',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 1994,
    _schemaLength: 24,
    _schemaStart: 1998,
    _tableLength: 4,
    _tableStart: 2023,
    _orgTableLength: 4,
    _orgTableStart: 2028,
    _orgNameLength: 4,
    _orgNameStart: 2038,
    characterSet: 45,
    encoding: 'utf8',
    name: 'link',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  },
  ColumnDefinition {
    _buf: <Buffer 01 00 00 01 1a 3a 00 00 02 03 64 65 66 18 64 63 70 5f 73 63 68 65 64 5f 64 62 5f 31 30 39 30 30 5f 32 37 38 39 32 04 6a 6f 62 73 04 6a 6f 62 73 02 69 ... 2179 more bytes>,
    _clientEncoding: 'utf8',
    _catalogLength: 3,
    _catalogStart: 2060,
    _schemaLength: 24,
    _schemaStart: 2064,
    _tableLength: 4,
    _tableStart: 2089,
    _orgTableLength: 4,
    _orgTableStart: 2094,
    _orgNameLength: 4,
    _orgNameStart: 2104,
    characterSet: 45,
    encoding: 'utf8',
    name: 'name',
    columnLength: 1020,
    columnType: 253,
    flags: 128,
    decimals: 0
  }
]

Schema --

+----------------------+------------------+------+-----+---------+----------------+
| Field                | Type             | Null | Key | Default | Extra          |
+----------------------+------------------+------+-----+---------+----------------+
| id                   | bigint(20)       | NO   | PRI | NULL    | auto_increment |
| address              | char(40)         | YES  | UNI | NULL    |                |
| uuid                 | varchar(255)     | YES  |     | NULL    |                |
| status               | varchar(255)     | YES  |     | NULL    |                |
| startTime            | bigint(20)       | YES  |     | NULL    |                |
| endTime              | bigint(20)       | YES  |     | NULL    |                |
| lastUpdate           | bigint(20)       | NO   |     | NULL    |                |
| owner                | char(40)         | YES  |     | NULL    |                |
| paymentAccount       | char(40)         | YES  |     | NULL    |                |
| lastSliceNumber      | bigint(20)       | YES  |     | NULL    |                |
| nextSliceNumber      | bigint(20)       | YES  |     | NULL    |                |
| maxSlicesPerTask     | int(11)          | YES  |     | NULL    |                |
| absoluteSlicePayment | decimal(31,18)   | YES  |     | NULL    |                |
| mvMultSlicePayment   | decimal(31,18)   | YES  |     | NULL    |                |
| duplicationLevel     | double           | YES  |     | NULL    |                |
| duplicationType      | varchar(255)     | YES  |     | NULL    |                |
| flags                | int(10) unsigned | YES  |     | NULL    |                |
| dataStorageType      | varchar(255)     | YES  |     | NULL    |                |
| dataStorageDetails   | text             | YES  |     | NULL    |                |
| resultStorageType    | varchar(255)     | YES  |     | NULL    |                |
| resultStorageDetails | text             | YES  |     | NULL    |                |
| resultStorageParams  | longtext         | YES  |     | NULL    |                |
| requirements         | longtext         | YES  |     | NULL    |                |
| description          | varchar(255)     | YES  |     | NULL    |                |
| link                 | varchar(255)     | YES  |     | NULL    |                |
| name                 | varchar(255)     | YES  |     | NULL    |                |
+----------------------+------------------+------+-----+---------+----------------+
26 rows in set (0.00 sec)

[wesgarland]

More investigation:
0.45494253843119004 is how Number.toFixed() generally displays 0.4549425384311900355527313877246342599391937255859375, which is encoded in 64-bit IEEE 754 as 0x3FDD1DC74F07C17C

0.4549425384311901 is how Number.toFixed() generally displays 0.454942538431190091063882618982461281120777130126953125, which is encoded in 64-bit IEEE 754 as 0x3FDD1DC74F07C17D

These are adjacent numbers on the IEEE 754 number line, i.e. one bit difference.

Both numbers represented in 32-bit IEEE 754 as 0x3EE8EE3A

Conclusion: the mysql2 code is probably passing MySQL DOUBLE to JS via a 32-bit float somewhere along the way.

[sidorares]

thanks for the report @wesgarland

non-prepared statement results come back as plain text, I suspect the reading code is here

node-mysql2/lib/packets/packet.js

Line 648 in 943b8b2

parseFloat(len) {

Can you compare results to what returned with .execute('select ... ?

[wesgarland]

Good hypothesis: the correct value IS returned from .execute().

[wesgarland]

And the correct arrives in parseFloat. So there must be a bug in parseFloat...

> this.buffer.slice(this.offset, this.offset + len).map((c) => String.fromCharCode(c)).join('')
'0045494253843119004'

[wesgarland]

Okay, I've given parseFloat some thought. It's not clear to me if the current strategy in there can ever be correct, especially if we look at some of the corner cases in IEEE 754 like roundToEven, although maybe you could get away with it if you assume that MySQL only generates numbers that are exactly representable in JS?

I think this is correct. On my machine, it is 4.8x slower than the existing implementation:

const _parseFloat = global.parseFloat;
function parseFloat(len)
{
  const ret = _parseFloat(String.fromCharCode.apply(null, this.buffer.slice(this.offset, this.offset + len)));
  this.offset += len;
  return ret;
}

Did I miss anything? Do you have unit tests for this?

[wesgarland]

Faster - I forgot that buffer has a native toString -

const _parseFloat = global.parseFloat;
function parseFloat(len)
{
  const ret = _parseFloat(this.buffer.slice(this.offset, this.offset + len).toString('ascii'));
  this.offset += len;
  return ret;
}

Now 3.75x slower than the incorrect code, which is 200ms over 1E6 iterations on my test machine. About 35% of that seems to be the call to buffer.slice.

[wesgarland]

PR incoming, just going to test it on a few million random numbers first.

Do you expect Infinity and -Infinity to work? They throw,

(node:7105) UnhandledPromiseRejectionWarning: Error: Unknown column 'Infinity' in 'field list'
    at Packet.asError (/home/wes/git/node-mysql2/lib/packets/packet.js:701:17)

0, Math.E, Math.PI and NaN passing fine. Not quite sure how to test -0.

[sidorares]

thanks for the PR @wesgarland !

Yes, the reason for manual parsing is performance, the slow part used to be buffer -> string conversion ( native parseFloat was and is faster than bespoke implementation ).

I'd really like to see before/after performance numbers - can you share you benchmark script? Ideally we want to have large number of rows and some number of float fields in the schema, then selecting that multiple times. Maybe even better to remove server and network from performance timing use fake server and pre-computed result set buffer, that way its mostly parsing that is benchmarked

[wesgarland]

@sidorares Here is a gist showing my benchmark code: https://gist.githubusercontent.com/wesgarland/06cb328984e38a1450618194b3d03983/raw/ab66131039ab0c2d18eaf5914cbda96c488a95d5/parse-float.js

I actually tried several approaches before settling on the buffer.toString() code. My first reimplementation started at around 1,400ms. A couple of versions got to around 31,000ms (Buffer.forEach etc). This version run in about 275ms on that machine (work linux box node 12), with the old code running around 78ms. I am home now, testing on macOS Catalina. Times are in ms, for 1,000,000 calls:

Node Version	Old	New
12.22.10	63	261
14.19.10	71	345
16.14.0	69	344
17.6.0	70	402

(What the heck is going on with NodeJS getting slower?! I am using versions from nvm fwiw).

The key issue with the old code is that it sacrifices precision by doing math on float64 quantities in float64. The IEEE 754 number line is not continuous, the "finite numbers" in the ES spec are absolutely not the same as our real numbers, and they are represented in binary, not base 10, internally. So a "place shift" by /10 or *10 is lossy - it lands on the nearest IEEE 754 number, not the exact value as though we had bit-shifted in binary.

A faster way to do this would be to use the strtod() C library call; it would be able to look directly at the buffer's backing store and perform the conversion without copying, since strtod takes both a start and end pointer. This is the Achilles heel here; in order to use JS engine's native parseFloat(), we have to copy the Buffer in order to pin the length. Of course, that has the ugly property of throwing is into native-code land with the attendant node-gyp and cross-platform pain.

So, I'm clocking 1,000,000 parse operations in 300-400ms; this patch would add about 1ms of program time for every 3000 doubles parsed as part of a prepared statement. It's slower than the old code, but still very fast.

The old code has errors in about 10-15% of numbers generated by Math.random(). It's a big problem from our POV.

[sidorares]

so current packet.parseFloat ( "old" column ) is still faster? What are your machine specs? I'll try to compare on mbpro intel ( I can test on MacBookPro15,1 - 2.6 GHz Core i7 (I7-9750H) and MacBookPro16,1 - apparently same cpu but faster ram )

also - is this again real server or fake js? is server running on the same computer (and also directly vs docker)?

A faster way to do this would be to use the strtod() C library call

you mean using some sort of native module for that? Probably not worth it because 1) cost of crossing js -> native boundary is probably higher than performance gains 2) pain of supporting native module

maybe worth exploring wasm version of the strtod code?

[sidorares]

but in general while performance is a feature, IMO secure code > correct code > fast code, if no good other solution happy to just use buffer.toString() + global.parseFloat

[sidorares]

Do you expect Infinity and -Infinity to work?

idk. How are they sent from the server via text protocol? Literal string "Infinity" ?

[wesgarland]

Yes, the old code is still the fastest. It's hard to beat from a memory access POV, I suspect the JIT is able to keep the intermediate value in a single register, possibly not loading any memory from even L2 cache during that loop. The input string is likely in L1.

The machine I ran those benchmarks on is a 2019 iMac; 3.6 GHz Quad-Core Intel Core i3 with 2400 MHz DDR4 RAM.

It's plausible that a WASM solution would be faster, but my gut tells me it won't be. I'm pretty sure that as soon as we have to get memory out of the Node Buffer into something non-native code can understand, we drop off the fast(est) path... I also explored text_encoder+DataView options, these were 50% slower than my best...suggesting that getting the memory out of a Buffer and into something we can deal with on a generic JS basis costs about 150ms in this benchmark.

[wesgarland]

To try and understand the perf bounds of this problem, I wrote a C program to measure strtod() on under similar circumstances. 275ms, built with clang and -O2: https://gist.github.com/wesgarland/682ed83db6a3f60b6fcb543a1878ae07