MultiplexingStream significantly slower than raw stream access

[AArnott]

Discussed in #503

^{Originally posted by plonkaadam July 1, 2022}
I'm trying to send a large binary file using MultiplexingStream over TcpSocket. I tried to send it directly using sockets (for now on loopback interface) and I'm able to transfer over 10GB/s.
When I try to do the same using MultiplexingStream and return Stream, PipeReader or create separate channel I could get up to only 80MB/s.

I tried to increase the buffer sizes but with no luck. Does the MultiplexingStream add so much overhead that it's not possible to get it any faster?

// after lot of tries I found out that there is a significant difference in speed between versions and I tried to use the newest one
Average speeds I got on my laptop are:
V1 - 118.11 MBps
V2 - 41.5 MBps
V3 - 48.0 MBps

Anyway - this application will run on a much slower device which during my initial tests got only up to 8MBps on V3; there's still a huge gap between multiplexed stream and raw stream speeds. Is there any way to make it faster?
Great library btw

In testing this, I measured 2-4Gbps using FullDuplexStream (basically raw pass through with no I/O overhead). That represents the ceiling of perf we might achieve (on my machine). MultiplexingStream V1 achieves 1.3Gbps, V2 931Mbps, and V3 599Mbps. So we indeed have a perf hit, and it gets worse with each protocol version. MessagePack slows down v2, and I'm not sure about V3 (window size handling, perhaps?).
What's particularly interesting is that when the transport is a named pipe instead of FullDuplexStream, mxstream slows to below 100Mbps. We should compare raw binary transfers across named pipes, for a fair comparison. But it may be that the frequency of flushes or some other Stream call has particularly high overhead when doing real I/O that involves the Windows kernel. We should look into both.

[AArnott]

I wonder if #532 being fixed will alleviate this somewhat.

[Elgot]

Is there any progress on this?

[AArnott]

No, I'm afraid it hasn't been a priority for my use cases.

Sponsorship may be an effective way to lead a contributor of this repo to prioritize the feature. As a free and open source library, all or the majority of its development is done without any monetary benefit to its authors, and all of it is made available free of charge. But sponsorship is very much appreciated and, with agreement from a particular contributor and/or the owning team, can be a great way to support the project and help get your favorite features or bugs addressed in an upcoming version.
If sponsorship to drive completion of a feature or bug fix is an interesting option to you, please let us know. You can reach out to me privately on keybase.io or gitter.im as well.

[Elgot]

Setting BackpressureSupportEnabled to false seems to improve the throughput 2-3 times, so that is perhaps the main difference between V1 and V3.

[Elgot]

Actually the real world (or "Start without debugging" in VS) performance seems much better (typically 1 GB/s vs 40 MB/s in VS). Still three times slower than FullDuplexStream, but probably good enough for most cases.

[adamplonka]

@Elgot yes I noticed the same, increasing the maximum message size to 64kB (maximum allowed by v1 of protocol) improves the performance greatly in debug mode but has no real impact to performance when running without debugger