metadata failure with long arrays

[wtfrank]

import { Codec } from '../src/utf15';
const codec = new Codec({ depth: 8, array: 1});
let data = [];
for (let x = 0; x < 32768; ++x)
    data.push(1);
let enc_str = codec.encode(data);

 RangeError: [utf15][RangeError]: x out of bounds: 32768
  at assert (src/utf15.js:39:30)
  at num_to_code_point (src/utf15.js:44:9)
  at Codec.encode_array (src/utf15.js:107:41)
  at Codec.encode (src/utf15.js:221:36)

It happens when an array of data has a length greater than can be expressed by a 15-bit number.

    if(!depth_is_array) // Save array length as meta
        res += String.fromCodePoint(num_to_code_point(values.length));

[wtfrank]

What seems to be happening is that if "meta" is not defined, then instead of storing meta data at the start of the stream, the array length is stored in the first position. But this is vulnerable to overflow as I have discovered.

A possible solution might be to make the "meta" style mandatory.

[Mototroller]

Yes, thanks, good point, it's really an issue due to metadata format (max array length is 2^15). I see two possible solutions:

1. Array length metadata field can be easily extended to 2 code units (30 bits). Obvious drawbacks: just another hardcoded limit, memory overhead for small arrays.

2. Array length metadata field can be made variadic by "stop-bit" encoding: like sizes in range [0, 2^14) can be encoded as one code unit with format 0##..##, values in range [2^14, 2^28) can be encoded as two code units like 1##..##-0##..## and so on. Drawbacks: little bit complex, memory overhead for arrays > 2^14.

Both these solutions have no backward compatibility for huge arrays, unfortunately.

Actually just these two lines should be reworked:

• utf15/utf15.js

  Line 107 in fa7060e

  res += String.fromCodePoint(num_to_code_point(values.length));

• utf15/utf15.js

  Line 154 in fa7060e

  const length = depth_is_array ? meta.depth.length : code_point_to_num(str.codePointAt(it++));

I guess it's a good feature to allow arrays with unknown sizes to be deserialized without knowing the exact codec, so I guess it's not necessary to change codec config format.

@wtfrank, feel free to make a PR if you wish :) Otherwise I'll fix this and make PR a little bit later.

[wtfrank]

yeah I experimented with this a little and setting metadata on doesn't solve the issue, unlike my thinking in my earlier comment. I came to the conclusion that making the array length field variable like your solution 2) was the way forward, so I'll probably implement a fix along those lines at some point, unless you beat me to it.