Issue linking events with IOSQE_FIXED_FILE (ECANCELED & EBADF)

[james-aus]

Let me preface this by saying that this is most certainly a me problem (also I am on Linux 5.15)

Background

Because reasons, I need to use io_uring directly in Java (without using the liburing library). For the purposes of this question, I re-wrote a snippet in C++ and reproduced the exact same issue, so we can just put all of that aside, but it's worth calling out that I'm using io_uring directly, not liburing (my understanding is that Jens develops both, and this is an easier place to reach the io_uring devs than the Linux mailing list -- please redirect me if that's wrong).

What I'm trying to do

Open a file and read some bytes.

SQEs

Here are 2 SQEs that I am using:

    // (entries start off memset(0)'d

    submissionEntries[0].opcode = IORING_OP_OPENAT;
    // submissionEntries[0].flags = IOSQE_IO_LINK;
    submissionEntries[0].fd = -100; // at current directory
    submissionEntries[0].addr = (uint64_t) (void *) "test.cpp";
    submissionEntries[0].user_data = 1234;
    submissionEntries[0].file_index = 1; // Use index 0 -- per docs we add 1.

    submissionEntries[1].opcode = IORING_OP_READ;
    submissionEntries[1].flags = IOSQE_FIXED_FILE;
    submissionEntries[1].fd = 0; // Use index 0
    submissionEntries[1].addr = (uint64_t) buffer;
    submissionEntries[1].len = 32;
    submissionEntries[1].user_data = 1235;

What works

Here's a workflow that works as I expect it to.
1. Create an IORING_OP_OPENAT operation specifying a file_index
2. Call io_uring_enter to submit 1 event
3. Create an IORING_OP_READ operation using that file index (by setting fd and using IOSQE_FIXED_FILE).
4. Call io_uring_enter to submit 1 event
I get back 2 completion events, my buffer fills with data, and my heart fills with joy.

What does not work

If instead I:
1. Create an IORING_OP_OPENAT operation specifying a file_index and supplying IOSQE_IO_LINK
2. Create an IORING_OP_READ operation using that file index (by setting fd and using IOSQE_FIXED_FILE).
3. Call io_uring_enter to submit 2 events

Then I get back sadness. The OPENAT command gives me ECANCELED (?!) and IORING_OP_READ gives me EBADF.

My understanding of IOSQE_IO_LINK is that, if that event fails, the later events should give me ECANCELED, but instead it seems to be the other way around? Strangely, removing IO_LINK but submitting both entries at once doesn't give me any issues and I get data back.

My Questions

• My reading of the docs has ECANCELED only happening to other events in the chain (in this case, IORING_OP_READ), have I misunderstood the docs, or just really messed up somewhere here?
• Why does it work without IOSQE_IO_LINK present in the one submit call? Is this just racy? Do I need IOSQE_IO_LINK, or is it "implied" somehow?
• Does this reproduce on other machines? I am using WSL /w Linux 5.15, which is both very old and not a real machine.

I've attached a ~110 line stand-alone example (test.cpp) (clang++ test.cpp -o test && ./test) that can be made to fail by changing line 71.

Calling setup
Setup result 3
io_uring_register result is 0

io_uring_enter: 2
Completion event count: 2
Completion Event 0: {user_data=1234, res=0, flags=0}
Completion Event 1: {user_data=1235, res=32, flags=0}

Buffer contents:
#include <stdint.h>
#include <io

Calling setup
Setup result 3
io_uring_register result is 0

io_uring_enter: 2
Completion event count: 2
Completion Event 0: {user_data=1234, res=-125, flags=0}
Completion Event 1: {user_data=1235, res=-9, flags=0}

Buffer contents:

[AmkG]

I ran your test.cpp on my machine: Linux AmkG-laptop 6.12.10-76061203-generic #202412060638~1753385872~22.04~dc2e00d SMP PREEMPT_DYNAMIC Thu J x86_64 x86_64 x86_64 GNU/Linux and it works correctly (prints out the test.cpp file) with both IOSQE_IO_LINK and without.

I lightly modified it to assign the #define SQE0_FLAGS defined at the top of the test.cpp and here are my results:

Calling setup
Setup result 3
io_uring_register result is 0

io_uring_enter: 2
Completion event count: 2
Completion Event 0: {user_data=1234, res=0, flags=0}
Completion Event 1: {user_data=1235, res=32, flags=0}

Buffer contents:
#define SQE0_FLAGS 0
#include <s

And

Calling setup
Setup result 3
io_uring_register result is 0

io_uring_enter: 2
Completion event count: 2
Completion Event 0: {user_data=1234, res=0, flags=0}
Completion Event 1: {user_data=1235, res=32, flags=0}

Buffer contents:
#define SQE0_FLAGS IOSQE_IO_LINK

[james-aus]

Looks like a more recent kernel (in this case 6.6.87.2) fixes it -- I don't see a reason to look further given that the likely cause is some hijinks in how the wsl kernel was built or some other already fixed bug.

Thanks for checking this on your machine for me.