Store to wire streaming.

[evoskuil]

I've also given some thought to wire serialization of blocks/txs from the store. While performance is excellent, it's astounding how much translation is taking place. The store spans 8 tables and two hashmap headers (header & txs) to retrieve a block by hash, and (6 tables and one header - tx) to retrieve a tx by hash independent of a block. It then deserializes from store encoding to the native chain:: object model. All objects, down to op.data stored on std::share_ptr<> instances. Then to serialize out the entire object model is walked on a system::ostream to create a single buffer. For text (hex) encoding the buffer is then translated to hex using system::base16_encode. For json the object modal is walked directly by boost::json.

For wire/text there is really no need whatsoever to translated through the normal form (object model). The wire serialization can be trivially produced directly onto the stream from store serialization. This would bypass countless allocations and copies. Scripts and witnesses for example, making up the bulk of all data, do not even require parse, they can just be written to the stream. (Though it would certainly be cleaner if no for the tragic segwit decision to serialize witnesses independently from inputs.) I'd estimate that this would cut large block deserialization time in half, and on my old machine that would put it under 20ms.

Implementation for this is just a set of store queries that operate over the stream, and swapping out the get block/tx calls that then serialize to wire with one that just returns the stream buffer. Blocks are stored with their serialized size (witnessed if enabled on the node), in the txs association table. So system::stream can be used over a preallocated contiguous buffer. But eventually this could even just stream write out to the port in chunks. It's possible to write the message heading without reading the object, since we always have the size indexed. So stream direct from store into the port with zero translation of scripts, witnesses, and point hashes, and the rest is just just organizing the pieces. I'd estimate 2-3 days of work for the whole thing to serialize blocks and txs directly from store to wire. The store queries no more than 1 day.

[evoskuil]

This of course can also be applied to non-p2p output streaming when it's not json. Furthermore, for text/hex serialization, we can provide a encoding streamer, just like our system::sha256_writer for example, that performs hex encoding on the fly, writing 2 bytes for every 1, and with a single double buffer size requirement. So with the fixed buffer can also use the fast system::ostream. That cuts allocation from 3x to 2x, and of course allows it to be chunked.

Thinking about json, that's a different problem, since boost::json operates over an object model. However the boost::json::value (intermediate json DOM) could be populated directly from store queries as well. In other words, have the query produce the model instance - instead of having the boost::json value convertor produce it from the annotated object model. This again would cut out the normal form queries. Those are really only necessary for validation. That's probably 2 days of work in the store - first day to figure out the proper boost::json pattern.

That would cut way down on memory allocations and probably cut the json output time in half (presently 75ms for large blocks on my machine, vs. 40ms for hex, and 35ms for wire encoding). So maybe json in 40ms, hex in 20ms, and 18ms for wire. Cut that in half for a 2x faster CPU, as I'm running 2.1GHz threads and these are entirely CPU-bound single threaded operations.