Support for short-circuit local reads

[mawenqi]

Is there any plan to support it? thanks.

[Kimahriman]

I haven't really looked into what it would take, but I don't really have plans anytime soon to look at it. Considering networking speeds are typically faster than disk speeds these days I'm not sure how much benefit it would provide. Plus you would need to be running this on your cluster for it to take effect. Most of my personal uses come from outside the cluster in non-distributed applications.

I am curious what use case you have in mind where this would be running on something like YARN that would support locality based scheduling.

[santosh-d3vpl3x]

It could still be worth it to implement it, read throughput of nvmes can absolutely exhaust network links.

[Kimahriman]

Actually when talking about short circuit reads it's not even about exhausting a network link, since a read over the local host network would (probably?) not even touch the physical networking device. Based on some profiling I've done, a large part of the time reading is spent doing the CRC checks after getting the data, which would still need to happen with a direct local read. Would probably need to see some sort of benchmarks showing significantly faster reads avoiding the local network and going straight to disk.

The real benefit to network throughput is locality based replica selection, and the NameNode does this for clients when getting block locations, by sorting the locations by "closeness" to the requesting client (if it can resolve that client to its topology, which would mostly only happen if you are using this library with a YARN scheduled app). So preferring reads from the local node should already be happening without any work from the client.

[santosh-d3vpl3x]

That's a good point, subjected to whether physical interface or loopback interface into play.

[mawenqi]

Sorry for the late response. Our application is deployed on every DataNode, so we expect that short-circuit local reads will lead to improved performance.