Can K/V key creation fail with "wrong last sequence" even if key was previously deleted?

[pcsegal]

During an experiment, I came across the following scenario:

• A key was deleted in a K/V bucket at time T.
• A bit less than 2 minutes later, a create operation was attempted, with the same key that was previously deleted. However, the create operation failed with a "wrong last sequence" error, as if the key had not been deleted at T.

After the fact, I verified that, from the key's history (using the nats kv history command), the last operation on this key was in fact DELETE, the creation timestamp of the operation matched the logged time T, and the logged timestamp of the creation attempt was about 2 minutes after the DELETE operation's timestamp.

Environment:

• NATS server version 2.9.14 running locally on localhost
• nats.go client version 1.19.0
• Ubuntu 20.04.5 on WSL2 on Windows 10

Unfortunately, I have not been able to reproduce it, which is why I didn't create an issue with the "bug" label. However, I still wanted to ask whether there is any known scenario where create could fail with "wrong last sequence" after a delete was issued.

[aricart]

If the KV is replicated it can fail, depending on the version of the client that created the KV, reads can happen from any server that has a copy of the KV, this means that it is possible for this, for your create to have failed while the key was indeed valid (not deleted) but right then a delete happened. The client performed a read (which was satisfied from a replica - and not yet updated with the delete), and used that sequence to submit an update, which when received by the leader was found to be incorrect, because the delete was now at the next sequence.

[pcsegal]

Thank you.
In this case, I have R=1.
But, in any case, I see what you mean - there could be a race condition between the Create and Delete operations.
The strange thing in my scenario is that it seems like the delete finished before the create started. And, when looking at the KV history, that delete is registered as the last operation performed on the key.
Can this still happen if the create operation started after the delete concluded?