Missing record when we try to read all records from DB

[Santhoshbesagarahalli]

We have observed issue with NATS Server Version: 2.11.4, where one of the record will be missing when we try to read all records from KV store. And this happens when other thread or application is trying to update the existing record to the same KV store.

Below is the code snapshot which we had used to reproduce the issue. i.e. we have added 100 records to KV store and trying to read all records from DB using kvStore_Keys or kvStore_Watch. Using another thread, we are trying to update the one of the record in the same DB. Some time we see one of the record (i.e. the record with the key which is getting updated) will be missing in the output of kvStore_Keys or kvStore_Watch operation.

db_read.txt

Can you please help us to root cause why one the record will be missing during get operations (i.e. kvStore_Keys or kvStore_Watch), even though record is present in the DB.

[Santhoshbesagarahalli]

We are using NATS Server Version: 2.11.4 for testing. please check this issue and comment

[MauriceVanVeen]

I'm assuming your KV/stream is replicated (i.e. has 3 replicas for example), and you expect to do a write using kv.Put and then immediately be able to observe that write with kv.Keys or kv.Watch?

Then this issue would be related: #6557
There currently are no read-after-write guarantees when using replicated streams.

[Santhoshbesagarahalli]

Hi Maurice,

Thanks for your response.

yes, we have replicated the KV (i.e 3 replicas) and we are updating the record using kv put (i.e. we have that key record already in the database).  Since we have 100 records all the time in the DB we expect Get_key and watch should return all the 100 key.

[Santhoshbesagarahalli]

Hi Maurice,

As per our current design we need data replicas on KV store. And data will be read and written by different applications simultaneously. Could you please suggest method that we can use to overcome this issue.