Support io_uring for file I/O

[mox692]

This proposal suggests adding support for io_uring in Tokio's file API. The following provides the details of this proposal.

Proposal: https://hackmd.io/@ScaowLCjTCGW9slfxHgVHw/SkRCb0zCkx

Summary

• This proposal suggests adding support for io_uring in Tokio's file API.
• Initially, the goal is to transparently replace the file API backend with io_uring from a thread pool. Advanced features such as registered fds or registered buffers will probably be addressed in separate proposal.
• The application of io_uring to network I/O is outside the scope of this proposal.
• The implementation will happen incrementally.

Current Status

• Add initial infrastructure code (Changes to the driver, add cfg, Op future, etc)

  • rt: add infrastructure code for io_uring #7320
  • rt: add check for io_uring availability at runtime #7357

• Implementing actual io_uring operations

  • fs::OpenOptions (fs: add io_uring open operation #7321)
  • fs::write ( fs: support io_uring in fs::write #7567)
  • File::open() (fs: use FileOptions inside fs::File to support uring #7617)
  comprehensive list of tokio::fs APIs

  
  

  Tier 1: Important APIs that are also used by other APIs, or frequently used

  • OpenOptions::open()
    • Internally used by File::open() and File::create()
  • File::open(), File::create()
    • Internally used by fs::read() and fs::write()
  • statx(2) (Hard to implement Metadada, see fs: support io_uring in fs::metadata (statx) #7616)
    • Required for determining buffer size in fs::read() and for the fs::metadata() API
  • fs::write()
    • Frequently used API
    • Can be supported once open is implemented
  • fs::read()
    • Frequently used API
    • Can be implemented once open and fstat are supported

  Tier 2: APIs that depend on Tier 1 APIs or are considered less frequently used

  • fs::copy()
  • Directory-related APIs (Unlike files, all directory operations seem to go through DirBuilder)
    • DirBuilder
    • create_dir()
    • create_dir_all()
    • fs::read_dir()
  • fs::set_permissions()
  • ... and other APIs

• Further improvements, such as:

  • Sharding rings in the multi-threaded runtime
  • Smarter batching logic for submission
  • Exploring the possibilities of using advanced features, such as registered buffers or registered files

[carllerche]

That sounds great to me. I think we should go ahead and move forward. It can be done incrementally.

We will need to add a new cfg feature since it will require new dependencies (at least initially). --cfg tokio_unstable_uring is probably what I would use (tokio_taskdump probably should have been tokio_unstable_taskdump). I would not do tokio_unstable_uring_fs as that implies other io_uring components will be added later via cfg flags. If we do add io_uring support for other components, it probably will not need additional features.

I am unsure, but we may also want to require runtime-level opt-in to io_uring support. For this, there would be a flag fs_mode(Mode::Uring(config)) on the runtime builder. We can worry about exposing op-level options later.

To handle the multi-threaded runtime, I would start with a single global submission queue and synchronize it. You are correct that it isn't ideal, but there isn't a mechanism yet to make one per worker. We should also make one timer per worker, which would be related. I would change the timer before worrying about sharding the io_uring submission queue.

[problame]

Came here via Discord: https://discord.com/channels/500028886025895936/500336346770964480/1361654607012692090

I'm excited to see activity on this.

I implemented the epoll-based approach similar to @mox692 dsecribes in their proposal linked at the top for our use in Neon's Pageserver: https://github.com/neondatabase/tokio-epoll-uring

We've been using it for many months across the fleet, first for buffered IO, recently direct IO for reads, now direct IO for writes.

Some feedback from skimming the RFC:

We didn't want to fork tokio, so, the addition of the io_uring FD to the tokio epoll instance is done via AsyncFd from a regular tokio task.
That is definitely suboptimal in a variety of ways, most significantly wrt hiding of "runnability".
But it avoids outside-of-runtime wakeups, which I have shown in benchmarks to be advantageous.

Wrt multi-threading, we have an io_uring instance and AsyncFd task per worker thread. Since tokio doesn't have a primitive for worker-thread-locals, we just use regular thread-locals that are only used from worker threads. (We don't use block_in_place, so, no risk of worker threads being stolen).

We opted for an owned-buffers-style API, similar to what tokio_uring does.
This avoids the need for a memory-to-memory copy into IO buffers but makes the API quite unergonomic / hard to drop-in-replace std::fs or current tokio::fs.

Also, as we are wrapping up the rollout (long & slow rollout over many past months), the "what's next" wish list is dominated by a different style of API, i.e., instead of operation().await, being able to submit a bunch of operations as we traverse our index data structure, then wait for all of them to complete at the end of traversal.

I think "high-end" users will not be satisfied with an operation().await style API.

I understand this issue here aims to remain compatible with tokio::fs, which is such a style of API.
That's going to be a big leap forward over what we have today, so I'm happy about that.

If you are open to scope-creep a bit, or at least prepare the design such that a split submit+complete style API is possible to implement from external crates, please reach out!

[one-d-wide]

Excited to see progress on this.

I poked around io_uring some time ago. And IIRC there is a bit of an incompatibility in how buffer operations are treated in io_uring vs. classic io interfaces.

AFAIK io_uring was designed to be compatible with direct memory access by hardware. This usually means that, for example, a buffer involved in a read operation must not be reused until this operation resolves, since this may lead to a race condition with an in-progress DMA request. So even if an operation was asynchronously canceled, a DMA request may still be pending.

This is in contrast to a classic io interfaces like a Read trait or a read syscall, which guarantee that once they return, the buffer is free to reuse.

As I understand, that's usually only an issue, when we allow an async request to be canceled before it resolves. For example as part of a Future, this typically necessitates an awkward api with owned buffers (like tokio-rs/tokio-uring have), or some kind of unsafe to acknowledge this possibility (like some embedded implementations have).

I guess it may be possible to implement this as part of a Drop trait of a read Future, which would cancel the request and synchronously wait until io_uring operation resolves, and hoping that it won't lock for too long. Or doing something clever inside the select block to make sure all io_uring operations are resolved before returning, since this is the place, where Futures likely to get canceled.

So I'm curious how this proposal addresses this issue. I'm also a bit rusty on this, so I might have gotten some details above wrong.

[Darksonn]

I know everyone is worried about this buffer thing but it's not a big problem. After all, using spawn_blocking to read and write like tokio::fs already does has the exact same buffer problem. When switching tokio::fs over to io_uring, we will just continue to use the same solution: copy the data into a buffer controlled by Tokio.

Now, down the road there are things such as TcpStream where this solution doesn't work. Primarily this is because it breaks behavior: it makes the stream behave like a BufWriter<TcpStream>. Probably we use IORING_OP_POLL_ADD to handle this case with identical behavior to epoll. Or we just continue to use epoll with the default tokio::net types and only use io_uring for cases such as tokio::fs.

[ADD-SP]

the goal is to transparently replace the file API backend with io_uring from a thread pool

This progress may not be "transparent", as io_uring requires higher permission than thread pool. see #7691 (comment).