Possible concurrency bug (needs more info)

[dakom]

This is possibly related to #162 but also maybe not, and could be expected behavior

1. Install a load-testing tool like https://ghz.sh/
2. Run it over some time against a simple service. Example:

ghz --insecure -z 5s --call cosmos.base.tendermint.v1beta1.Service.GetLatestBlock localhost:9090

3. Observe that it reports some errors (maybe 0.1% of the time)

Other variations such as running by a large number of requests (e.g. 50,000) or switching the async mode on (--async) do not alleviate the errors

Using smaller loads does work, e.g. running a limited test over 1,000 requests is fine, also, increasing the number of connections reduces errors when it's an explicit number, but not over time. For example, this works for me:

ghz --insecure -n 50000 --connections 20 --call cosmos.base.tendermint.v1beta1.Service.GetLatestBlock localhost:9090

but this does not, even though on my machine this time cap runs roughly the same number of requests as above:

ghz --insecure -z 5s --connections 20 --call cosmos.base.tendermint.v1beta1.Service.GetLatestBlock localhost:9090

I'm not sure if this is expected behavior because we have some sort of internal rate limiting, or if I'm not running the load testing correctly, or if it's a genuine issue - but I do think it's worth looking into to make sure we don't have some genuine concurrency bug in the server

[ethanfrey]

That is very interesting, and thanks for showing how to repeat.
Note that this is piped from our grpc to tendermint rpc internally.

Could you make another such test that hit some internal query (like a bank query for the balance of some account)?

That will help distinguish between a bug in our grpc server (which would affect both) and a bug in our rpc client to cometbft (which would only affect the GetLatestBlock call, but not something internal to slay3rd)

[dakom]

same sortof results:

a few errors:

ghz --insecure -z 5s -d '{"address": "layer1qyfn9l7w78kxcerwwwc6dpad305dulhk9jks0r", "denom": "uslay"}' --call cosmos.bank.v1beta1.Query.Balance localhost:9090

a few errors:

ghz --insecure -z 5s --connections 20 -d '{"address": "layer1qyfn9l7w78kxcerwwwc6dpad305dulhk9jks0r", "denom": "uslay"}' --call cosmos.bank.v1beta1.Query.Balance localhost:9090

no errors:

ghz --insecure -n 50000 --connections 20 -d '{"address": "layer1qyfn9l7w78kxcerwwwc6dpad305dulhk9jks0r", "denom": "uslay"}' --call cosmos.bank.v1beta1.Query.Balance localhost:9090

[dakom]

Some more ideas that came up in discussion that should be done and added for more info:

1. Run against wasmd to get a baseline
2. Run with climb to make sure it's getting around the same results
3. Somehow run with signed transactions that change state, see if it's significantly worse or stays the same