Profile: is coalescing buffers or multiple writes faster?

[vimpunk]

#95 introduced a fallback for disk writes on other platforms. It currently uses a less efficient approach than the Linux positional vectored IO APIs as they are not present on other platforms.

There were two options for the fallback, quoting directly from the implementation:

                // 1. Coalesce the slices to a continuous preallocated memory buffer and
                //    write it at once.
                // 2. Iterate through the slices and issue a separate write call for each.
                //
                // The first option implies preallocating a short-living buffer
                // and destroying it shortly after. The second option
                // implies issuing a big number of writes, and possibly disk I/O operations.

We should benchmark which is actually faster at some point.

[ExceptionallyHandsome]

Any time disk I/O perf is considered, OS caching/buffering must be taken into account.

As things stand with #95 , we use a user-space buffer, and then the OS does its own buffering. I think this double-buffer makes the data transfer slower than it might have been. So yeah, careful and ample benchmarking should become our mantra eventually, but for now it can wait.

In fact, we have not two, but three options to proceed:

• User-space buffer + OS buffer (current plan on windows & mac, on linux user-space buffer is absent entirely).
• User-space buffer only. On Windows, OS buffering can be disabled by setting the FILE_FLAG_NO_BUFFERING attribute on creating/opening. Some further info (sounds like aligment-aware allocations will be needed).
• OS buffer only (the "multiple writes" strategy). This is by the way what libtorrent does.

Also, user-space buffering has its own optimizations to apply...