@pretendonetwork/binary-parser

@pretendonetwork/binary-parser is a type-safe binary/struct parser and encoder with an API inspired by https://npmjs.com/binary-parser, bit-level operations, and Zod-like type inference.

Why?

The existing binary-parser library has arguably the best developer experience for reading binary data in terms of API, however it has a number of issues:

1. Lacks official encoding support. A pull request to add this does exist, however it has sat stagnant for years with no sign of being merged
   • This fork also has it's own issues, like not being able to encode bit fields larger than 32 bits
2. When used for very tightly packed bit fields that span multiple bytes, or data that swaps endianness at the bit-level, both the fork and upstream binary-parser libraries fail to decode the data correctly
   • This is an issue with most struct/binary parsing libraries
3. Despite being written in TypeScript, binary-parser is not very type safe when it comes to the outputs of the parsing
4. binary-parser relies on generating the parsing code at runtime, which can make it very difficult at times to debug issues as errors do not reference the library itself. binary-parser does provide a getCode method to get the generated code at runtime, but this feels like more of a hack than anything and doesn't help with matching errors to the library itself
5. Lacks certain quality of life features like boolean bit fields, bit skipping, etc.
6. Trying to fix these issues in binary-parser itself would prove difficult (resulting in a similar workload to just writing a library from scratch, as some of these issues are related to core design decisions by binary-parser), especially since there's no guarntee that changes will be merged (see the first point)

Because of these reasons, we have opted to write our own binary-parsing library.

Browser compatibility

@pretendonetwork/binary-parser aims to be compatible with both browsers and server-side JavaScript runtimes. Core features do not, and should not, rely on anything besides native JavaScript. Platform specific features, such as Parser().buffer() which returns a NodeJS Buffer, are allowed so long as they do not break other platforms and it is understood that said features cannot be reliably used on all platforms.

API

The API is intended to, eventually, be a superset of the original binary-parser API (as closely as possible) in an attempt to make a drop-in replacement, keeping the same API while adding in our own extra features. Currently not everything is implemented, however, and some features (such as getCode) will likely never be available.

Parser

Creating a parser can be done in one of two different ways, either with the new keyword or with the start() static method. Once a parser is created, a series of "commands" may be attached to it to define the structure and parsing/encoding behavior. See control commands and field commands for more information.

import { Parser } from '@pretendonetwork/binary-parser';

const parser1 = new Parser();
const parser2 = Parser.start();

Type inference

Once a parser has been created, type inference may be done in the exact same way as Zod, using b.infer. Additionally, b.output is also provided. This is an alias of b.infer. The result of b.infer matches the type returned by the parsers parse/safeParse methods, as well as the input type of the parsers encode/safeEncode methods, resulting in strong type safety.

import * as b from '@pretendonetwork/binary-parser';

const coreParser = new b.Parser()
	.endianness('little')
	.uint8('version')
	.booleanBit('allowCopying')
	.booleanBit('profanityFlag')
	.bit2('regionLock')
	.bit2('characterSet'); // Subset of the real FFLiMiiDataCore type

type FFLiMiiDataCore = b.infer<typeof coreParser>;
/*
type FFLiMiiDataCore = {
    version: number;
    allowCopying: boolean;
    profanityFlag: boolean;
    regionLock: number;
    characterSet: number;
}
*/

Parsing

After a parser has been created, and commands attached to it, data may be passed to the parser to be decoded. It is safe to use the same parser to parse different data in parallel. The data passed in for parsing may be either a Uint8Array, NodeJS Buffer, or instance of BitStream.

import { Parser } from '@pretendonetwork/binary-parser';

const ipHeader = new Parser()
	.bit4('version')
	.bit4('headerLength')
	.uint8('tos')
	.uint16('packetLength')
	.uint16('id')
	.bit3('offset')
	.bit13('fragOffset')
	.uint8('ttl')
	.uint8('protocol')
	.uint16('checksum')
	.array('src', {
		type: 'uint8',
		length: 4
	})
	.array('dst', {
		type: 'uint8',
		length: 4
	});

console.log(ipHeader.parse(Buffer.from([ 0x45, 0x00, 0x02, 0xc5, 0x93, 0x99, 0x00, 0x00, 0x2c, 0x06, 0xef, 0x98, 0xad, 0xc2, 0x4f, 0x6c, 0x85, 0x01, 0x86, 0xd1 ])));
/*
{
	version: 4,
	headerLength: 5,
	tos: 0,
	packetLength: 709,
	id: 37785,
	offset: 0,
	fragOffset: 0,
	ttl: 44,
	protocol: 6,
	checksum: 61336,
	src: [ 173, 194, 79, 108 ],
	dst: [ 133, 1, 134, 209 ]
}
*/

The parse method will throw any errors it encounters while parsing the provided data. Because of this, a Zod-like safeParse method is also provided.

import { Parser } from '@pretendonetwork/binary-parser';

const ipHeader = new Parser()
	.bit4('version')
	.bit4('headerLength')
	.uint8('tos')
	.uint16('packetLength')
	.uint16('id')
	.bit3('offset')
	.bit13('fragOffset')
	.uint8('ttl')
	.uint8('protocol')
	.uint16('checksum')
	.array('src', {
		type: 'uint8',
		length: 4
	})
	.array('dst', {
		type: 'uint8',
		length: 4
	});

const result = ipHeader.safeParse(Buffer.from([ 0x45, 0x00, 0x02, 0xc5, 0x93, 0x99, 0x00, 0x00, 0x2c, 0x06, 0xef, 0x98, 0xad, 0xc2, 0x4f, 0x6c, 0x85, 0x01, 0x86, 0xd1 ]))
if (result.success) {
	console.log(result.data);
	/*
	{
		version: 4,
		headerLength: 5,
		tos: 0,
		packetLength: 709,
		id: 37785,
		offset: 0,
		fragOffset: 0,
		ttl: 44,
		protocol: 6,
		checksum: 61336,
		src: [ 173, 194, 79, 108 ],
		dst: [ 133, 1, 134, 209 ]
	}
	*/
} else {
	console.log(result.error);
}

Encoding

After a parser has been created, and commands attached to it, an object of decoded values may be passed to the parser to be encode. It is safe to use the same parser to encode different data in parallel. Returns a Uint8Array of encoded data if successful.

import { Parser } from '@pretendonetwork/binary-parser';

const ipHeader = new Parser()
	.bit4('version')
	.bit4('headerLength')
	.uint8('tos')
	.uint16('packetLength')
	.uint16('id')
	.bit3('offset')
	.bit13('fragOffset')
	.uint8('ttl')
	.uint8('protocol')
	.uint16('checksum')
	.array('src', {
		type: 'uint8',
		length: 4
	})
	.array('dst', {
		type: 'uint8',
		length: 4
	});

console.log(Buffer.from(ipHeader.encode({
	version: 4,
	headerLength: 5,
	tos: 0,
	packetLength: 709,
	id: 37785,
	offset: 0,
	fragOffset: 0,
	ttl: 44,
	protocol: 6,
	checksum: 61336,
	src: [ 173, 194, 79, 108 ],
	dst: [ 133, 1, 134, 209 ]
})));
// <Buffer 45 00 02 c5 93 99 00 00 2c 06 ef 98 ad c2 4f 6c 85 01 86 d1>

The encode method will throw any errors it encounters while encoding the provided data. Because of this, a Zod-like safeEncode method is also provided.

import { Parser } from '@pretendonetwork/binary-parser';

const ipHeader = new Parser()
	.bit4('version')
	.bit4('headerLength')
	.uint8('tos')
	.uint16('packetLength')
	.uint16('id')
	.bit3('offset')
	.bit13('fragOffset')
	.uint8('ttl')
	.uint8('protocol')
	.uint16('checksum')
	.array('src', {
		type: 'uint8',
		length: 4
	})
	.array('dst', {
		type: 'uint8',
		length: 4
	});

const result = ipHeader.safeEncode({
	version: 4,
	headerLength: 5,
	tos: 0,
	packetLength: 709,
	id: 37785,
	offset: 0,
	fragOffset: 0,
	ttl: 44,
	protocol: 6,
	checksum: 61336,
	src: [ 173, 194, 79, 108 ],
	dst: [ 133, 1, 134, 209 ]
});
if (result.success) {
	console.log(Buffer.from(result.data));
	// <Buffer 45 00 02 c5 93 99 00 00 2c 06 ef 98 ad c2 4f 6c 85 01 86 d1>
} else {
	console.log(result.error);
}

size()

The parser.size() method can be used to get the size, in bytes, of the parser. Not all parsers are able to have their sizes computed, however. Such as in the case of fields which reference other fields for their lengths at runtime. In these cases, the result of parser.size() will be inaccurate as these fields will be skipped.

Control commands

"Control commands" are commands attached to a parser which change parsing behavior, but do not contribute to the structure of data.

endianness(endianness: 'big' | 'little')

Changes the endianness for all fields after this command. By default all parsers are in big-endian mode. Field commands which set an explict endianness, such as uint32le, are not affected by this change and will always use their explict endianness.

skip(length: number)

Skips length number of full bytes. When encoding, skipped bytes are left as 0x00

skipBits(length: number)

Skips length number of bits. When encoding, skipped bits are left as 0

min(value: number)

Checks the most recently parsed field to ensure its value is above, or equal to, min. If the value is smaller than min, an error is thrown.

max(value: number)

Checks the most recently parsed field to ensure its value is below, or equal to, max. If the value is larger than max, an error is thrown.

range(min: number, max: number)

Checks the most recently parsed field to ensure its value is between min and max, inclusive. If the value is not between min and max, inclusive, an error is thrown. This is effectively the same as changing min() and max() together.

validate(callback: (parsed: T) => void)

Runs callback with the intent of validating the parsed data. Can be placed anywhere in the chain. parsed will contain the currently parsed data up until this point. Useful for situations where complex value validation is required, such as comparing multiple fields or comparing a field to some global state.

Field commands

"Field commands" directly define the structure of data, relying on control commands to know how to parse the respective field.

bits(name: string, bits: number, options?: NumberFieldOptions)

Alias of bitN. Reads bits number of bits into the name field. Optionally takes in a NumberFieldOptions.

bitN(name: string, bits: number, options?: NumberFieldOptions)

Reads bits number of bits into the name field. Optionally takes in a NumberFieldOptions.

booleanBit(name: string)

Reads a single bit value, interpreted as a boolean (1 = true, 0 = false), into the name field.

bit(name: string)

Reads a single bit value into the name field.

bit1(name: string, options?: NumberFieldOptions)

Reads 1 bit into the name field. Optionally takes in a NumberFieldOptions.

bit2(name: string, options?: NumberFieldOptions)

Reads 2 bits into the name field. Optionally takes in a NumberFieldOptions.

bit3(name: string, options?: NumberFieldOptions)

Reads 3 bits into the name field. Optionally takes in a NumberFieldOptions.

bit4(name: string, options?: NumberFieldOptions)

Reads 4 bits into the name field. Optionally takes in a NumberFieldOptions.

bit5(name: string, options?: NumberFieldOptions)

Reads 5 bits into the name field. Optionally takes in a NumberFieldOptions.

bit6(name: string, options?: NumberFieldOptions)

Reads 6 bits into the name field. Optionally takes in a NumberFieldOptions.

bit7(name: string, options?: NumberFieldOptions)

Reads 7 bits into the name field. Optionally takes in a NumberFieldOptions.

bit8(name: string, options?: NumberFieldOptions)

Reads 8 bits into the name field. Differs from the uint8/int8 commands as those methods work on byte boundaries only, whereas this command can read bits across bytes. Optionally takes in a NumberFieldOptions.

bit9(name: string, options?: NumberFieldOptions)

Reads 9 bits into the name field. Optionally takes in a NumberFieldOptions.

bit10(name: string, options?: NumberFieldOptions)

Reads 10 bits into the name field. Optionally takes in a NumberFieldOptions.

bit11(name: string, options?: NumberFieldOptions)

Reads 11 bits into the name field. Optionally takes in a NumberFieldOptions.

bit12(name: string, options?: NumberFieldOptions)

Reads 12 bits into the name field. Optionally takes in a NumberFieldOptions.

bit13(name: string, options?: NumberFieldOptions)

Reads 13 bits into the name field. Optionally takes in a NumberFieldOptions.

bit14(name: string, options?: NumberFieldOptions)

Reads 14 bits into the name field. Optionally takes in a NumberFieldOptions.

bit15(name: string, options?: NumberFieldOptions)

Reads 15 bits into the name field. Optionally takes in a NumberFieldOptions.

bit16(name: string, options?: NumberFieldOptions)

Reads 16 bits into the name field. Differs from the uint16/int16 commands as those methods work on byte boundaries only, whereas this command can read bits across bytes. Optionally takes in a NumberFieldOptions.

bit17(name: string, options?: NumberFieldOptions)

Reads 17 bits into the name field. Optionally takes in a NumberFieldOptions.

bit18(name: string, options?: NumberFieldOptions)

Reads 18 bits into the name field. Optionally takes in a NumberFieldOptions.

bit19(name: string, options?: NumberFieldOptions)

Reads 19 bits into the name field. Optionally takes in a NumberFieldOptions.

bit20(name: string, options?: NumberFieldOptions)

Reads 20 bits into the name field. Optionally takes in a NumberFieldOptions.

bit21(name: string, options?: NumberFieldOptions)

Reads 21 bits into the name field. Optionally takes in a NumberFieldOptions.

bit22(name: string, options?: NumberFieldOptions)

Reads 22 bits into the name field. Optionally takes in a NumberFieldOptions.

bit23(name: string, options?: NumberFieldOptions)

Reads 23 bits into the name field. Optionally takes in a NumberFieldOptions.

bit24(name: string, options?: NumberFieldOptions)

Reads 24 bits into the name field. Optionally takes in a NumberFieldOptions.

bit25(name: string, options?: NumberFieldOptions)

Reads 25 bits into the name field. Optionally takes in a NumberFieldOptions.

bit26(name: string, options?: NumberFieldOptions)

Reads 26 bits into the name field. Optionally takes in a NumberFieldOptions.

bit27(name: string, options?: NumberFieldOptions)

Reads 27 bits into the name field. Optionally takes in a NumberFieldOptions.

bit28(name: string, options?: NumberFieldOptions)

Reads 28 bits into the name field. Optionally takes in a NumberFieldOptions.

bit29(name: string, options?: NumberFieldOptions)

Reads 29 bits into the name field. Optionally takes in a NumberFieldOptions.

bit30(name: string, options?: NumberFieldOptions)

Reads 30 bits into the name field. Optionally takes in a NumberFieldOptions.

bit31(name: string, options?: NumberFieldOptions)

Reads 31 bits into the name field. Optionally takes in a NumberFieldOptions.

bit32(name: string, options?: NumberFieldOptions)

Reads 32 bits into the name field. Differs from the uint32/int32 commands as those methods work on byte boundaries only, whereas this command can read bits across bytes. Optionally takes in a NumberFieldOptions.

uint8(name: string, options?: NumberFieldOptions)

Reads an unsigned 8-bit number into the name field. Optionally takes in a NumberFieldOptions.

int8(name: string, options?: NumberFieldOptions)

Reads a signed 8-bit number into the name field. Optionally takes in a NumberFieldOptions.

uint16(name: string, options?: NumberFieldOptions)

Reads an unsigned 16-bit number into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint16le(name: string, options?: NumberFieldOptions)

Reads an unsigned 16-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint16be(name: string, options?: NumberFieldOptions)

Reads an unsigned 16-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int16(name: string, options?: NumberFieldOptions)

Reads a signed 16-bit number into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int16le(name: string, options?: NumberFieldOptions)

Reads a signed 16-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int16be(name: string, options?: NumberFieldOptions)

Reads a signed 16-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint32(name: string, options?: NumberFieldOptions)

Reads an unsigned 32-bit number into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint32le(name: string, options?: NumberFieldOptions)

Reads an unsigned 32-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint32be(name: string, options?: NumberFieldOptions)

Reads an unsigned 32-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int32(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit number into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int32le(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

int32be(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

uint64(name: string, options?: BigIntFieldOptions)

Reads an unsigned 64-bit number into the name field using the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

uint64le(name: string, options?: BigIntFieldOptions)

Reads an unsigned 64-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

uint64be(name: string, options?: BigIntFieldOptions)

Reads an unsigned 64-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

int64(name: string, options?: BigIntFieldOptions)

Reads a signed 64-bit number into the name field using the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

int64le(name: string, options?: BigIntFieldOptions)

Reads a signed 64-bit number into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

int64be(name: string, options?: BigIntFieldOptions)

Reads a signed 64-bit number into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a BigIntFieldOptions.

float(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit float into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

floatle(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit float into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

floatbe(name: string, options?: NumberFieldOptions)

Reads a signed 32-bit float into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

double(name: string, options?: NumberFieldOptions)

Reads a signed 64-bit float ("double") into the name field using the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

doublele(name: string, options?: NumberFieldOptions)

Reads a signed 64-bit float ("double") into the name field. This value will always be read in little-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

doublebe(name: string, options?: NumberFieldOptions)

Reads a signed 64-bit float ("double") into the name field. This value will always be read in big-endian mode, regardless of the parsers current endianness mode. Optionally takes in a NumberFieldOptions.

buffer(name: string, options: BufferFieldOptions)

Reads a NodeJS Buffer into the name field. The size of the value is determined by options. Not usable in browser contexts. For browsers, use array with the type utf8.

string(name: string, options: StringFieldOptions)

Reads a string into the name field. The size and encoding of the value is determined by options.

nest(name: string, options: NestFieldOptions)

Reads a Parser instance into the name field. The type of the embedded parser is defined in options.

array(name: string, options: ArrayFieldOptions)

Reads an array of options.type into the name field. The size and type of the value is determined by options.

Field options

Certain fields may take in a series of options to change parsing behavior and/or validation.

NumberFieldOptions

Options for how to validate a field whose value is number.

type NumberFieldOptions = {
	min?: number;
	max?: number;
};

• min - The same as calling min() for this field.
• max - The same as calling max() for this field.

BigIntFieldOptions

Options for how to validate a field whose value is bigint.

type BigIntFieldOptions = {
	min?: bigint;
	max?: bigint;
};

• min - The smallest value the field can be. If the value is smaller, an error is thrown.
• max - The largest value the field can be. If the value is larger, an error is thrown.

BufferFieldOptions

Options for how to parse/encode a field whose value is Buffer.

type BufferFieldOptions = {
	length: number | string;
};

• length - The length of the Buffer. If number, the value is assumed to be full bytes. If string, the value is assumed to be the name of an already parsed field (such as a "size" field). If string and the value is either not parsed, or the target field value is not a number, an error is thrown.

StringFieldOptions

Options for how to parse/encode a field whose value is string.

type TextDecoderEncoding =
	'utf-8' | 'utf-16be' | 'utf-16le' |

	// Legacy single-byte encodings
	'ibm866' | 'iso-8859-2' | 'iso-8859-3' | 'iso-8859-4' | 'iso-8859-5' |
	'iso-8859-6' | 'iso-8859-7' | 'iso-8859-8' | 'iso-8859-8i' | 'iso-8859-10' |
	'iso-8859-13' | 'iso-8859-14' | 'iso-8859-15' | 'iso-8859-16' |
	'koi8-r' | 'koi8-u' | 'macintosh' |
	'windows-874' | 'windows-1250' | 'windows-1251' | 'windows-1252' |
	'windows-1253' | 'windows-1254' | 'windows-1255' | 'windows-1256' |
	'windows-1257' | 'windows-1258' | 'x-mac-cyrillic' |

	// Legacy multi-byte Chinese
	'gbk' | 'gb18030' | 'big5' |

	// Legacy multi-byte Japanese
	'euc-jp' | 'iso-2022-jp' | 'shift-jis' |

	// Legacy multi-byte Korean
	'euc-kr' |

	// Miscellaneous
	'x-user-defined' | 'replacement';

type StringFieldOptions = {
	length: number | string;
	encoding?: TextDecoderEncoding | BufferEncoding;
};

• length - The length of the string in bytes, regardless of encoding. If number, the value is assumed to be full bytes. If string, the value is assumed to be the name of an already parsed field (such as a "size" field). If string and the value is either not parsed, or the target field value is not a number, an error is thrown.
• encoding - If not set, defaults to utf-8. The fields string encoding. Due to limitations with TextEncoder only supporting utf-8 strings, only a subset of TextDecoderEncoding is actually implemented at this time. Supported values are hex, base64, base64url, utf-8, utf8, ascii, utf-16le, utf16le, ucs2, ucs-2, utf-16be, and latin1. If an encoding is used that is not supported, an error is thrown.

NestFieldOptions

Options for how to parse/encode a field which nests another parser.

import type { Parser } from '@/parser';

type NestFieldOptions = {
	type: Parser;
};

• type - An instance of Parser to nest.

ArrayFieldOptions

Options for how to parse/encode an array field.

import type { ParserReadMethods } from '@/field-commands/array';
import type { Parser } from '@/parser';

/*
type ParserReadMethods =
	'booleanBit' |
	'uint8' |
	'int8' |
	'uint16' |
	'uint16le' |
	'uint16be' |
	'int16' |
	'int16le' |
	'int16be' |
	'uint32' |
	'uint32le' |
	'uint32be' |
	'int32' |
	'int32le' |
	'int32be' |
	'uint64' |
	'uint64le' |
	'uint64be' |
	'int64' |
	'int64le' |
	'int64be' |
	'float' |
	'floatle' |
	'floatbe' |
	'double' |
	'doublele' |
	'doublebe';
*/

type ArrayFieldOptions<T extends ParserReadMethods | Parser<any> = ParserReadMethods | Parser<any>> = {
	type: T;
	length: number | string;
};

• type - Either an instance of Parser or a ParserReadMethods value. This type will be parsed length times.
• length - The length of the array, in elements. If number, the value is assumed to be full bytes. If string, the value is assumed to be the name of an already parsed field (such as a "size" field). If string and the value is either not parsed, or the target field value is not a number, an error is thrown.