Binary protocols

[knight10-ux]

Just read the docs, great writeup. Communicating with json instead of wrapping some binary protocol impl in some accessible interfaces to the developer seems to be criminal indeed. You mention there protobufs, but I remember spending some time pondering on real logics aeron and simple binary encoding. Soo, what's your thoughts on that and did you think about bringin some of that, speaking here about utt probably, into uws ecosystem?

[e3dio]

Everything you need to encode and decode binary with JavaScript is already in Node Buffer and browser DataView, you could add a couple helper functions which I did here https://github.com/e3dio/packBytes if you want to really pack in data small, example is 50x smaller than JSON and 10x faster to encode, also I helped a guy do his specific case here e3dio/packBytes#5 (reply in thread) if you want to see a full encode and decode in a somewhat advanced example for some GeoJson data. Would be interesting to compare the native SBE with the equivalent encode in JS and see what the performance difference is, but the JS seems to be fast enough

[e3dio]

Looks like Protocol Buffers is the most popular binary encoding system, I added those to my benchmarks:

Encoding	time (ns)	bytes
packBytes	65	1
avsc	255	4
protobuf	235	8
msgpack	1155	33
json	780	53

A custom encode is going to be faster and smaller, I might add Schemas to PackBytes to automate all the encoding and decoding to make it easy to encode the smallest possible size at the fastest speed

[ghost]

I'm repulsed by JSON and such protocols but then again I am not a web developer so I will never understand.

Yes, JSON + zlib will produce pretty much the same size as a good binary protocol, but comparing the CPU and memory overhead between JSON + compression and just simply learning to directly put your data in an efficient format from the start is MASSIVE in many cases.

A schema-based binary protocol like Protobuf is awesome and you get static types and static names, so you cannot have typo bugs (which you ALWAYS get when developing with JSON, since everything is dynamic and string-matched).

Honestly, JSON protocols are a sign of mental problems. But that's my take, and my take is not very popular.

[e3dio]

JSON is the easiest most flexible human-readable output encoding system built into JS language and is good for quick developing so you can't count it out too quickly, but yes you are leaving out optimizations and potentially very large size + speed savings depending on your data. I think if you make the encoding process easier more people would do it. My new Schema system might make it easy enough, I would use that too

[ghost]

human-readable

Exactly why it is insane. Computers aren't humans. But yeah, unpopular opinion.

[ghost]

It's like sending a picture of a hand written letter