[cassandra-stress] c-s should be able to handle OverloadedException

[fruch]

The error says "java.io.IOException: Operation x10 on key(s) [32374c4b364e33324e30]: Error executing: (OverloadedException): Queried host (127.0.83.1/127.0.83.1:9042) was overloaded: Too many in flight hints: 10485991".

on cases of such errors, c-s should throttle back given this error.
see https://github.com/scylladb/scylla-enterprise/issues/3306#issuecomment-1706529424

@bhalevy maybe you want to elaborate what is means ? c-s should apply a rate-limiting, where the rate limit should be taken from ?

[bhalevy]

I really don't know if we want to maintain CassandraStress and develop such features for it.
We should have our own load generator in Python that we can fully control for dtests.

[mykaul]

The code is there in the driver, but I'm unsure what the behavior is.
https://github.com/scylladb/java-driver/blob/d291df6b35f7903c0b2d935754aebcb5b35bcd81/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L40

[juliayakovlev]

It happened again with Scylla version 2025.1.2-20250422

P99 latency is very high in this time

Packages

Scylla version: 2025.1.2-20250422.502c62d91d48 with build-id 377259d8f84fbda9a22be98e3bc7319a0121ca16

Kernel Version: 6.8.0-1027-aws

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 3 nodes (i3en.2xlarge)

Scylla Nodes used in this run:

• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-7 (34.248.27.149 | 10.4.2.163) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-6 (54.229.36.110 | 10.4.0.159) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-5 (52.49.133.208 | 10.4.1.3) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-4 (34.240.85.217 | 10.4.0.63) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-3 (3.253.6.228 | 10.4.2.186) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-2 (3.250.219.83 | 10.4.0.255) (shards: 7)
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-1 (3.252.249.42 | 10.4.3.187) (shards: 7)

OS / Image: ami-082351246305cb57f (aws: undefined_region)

Test: scylla-enterprise-perf-regression-latency-650gb-with-nemesis
Test id: ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc
Test name: scylla-enterprise/perf-regression/scylla-enterprise-perf-regression-latency-650gb-with-nemesis
Test method: performance_regression_test.PerformanceRegressionTest.test_latency_write_with_nemesis
Test config file(s):

• perf-regression-latency-650gb-with-nemesis.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc

Logs:

• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-5 - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250428_202151/perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-5-ffe7cbc6.tar.zst
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-2 - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250428_202151/perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-2-ffe7cbc6.tar.zst
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-1 - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250428_202151/perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-1-ffe7cbc6.tar.zst
• perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-6 - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250428_202151/perf-latency-nemesis-ubuntu-db-node-ffe7cbc6-6-ffe7cbc6.tar.zst
• db-cluster-ffe7cbc6.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250429_072732/db-cluster-ffe7cbc6.tar.zst
• sct-runner-events-ffe7cbc6.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250429_072732/sct-runner-events-ffe7cbc6.tar.zst
• sct-ffe7cbc6.log.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250429_072732/sct-ffe7cbc6.log.tar.zst
• loader-set-ffe7cbc6.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250429_072732/loader-set-ffe7cbc6.tar.zst
• monitor-set-ffe7cbc6.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/ffe7cbc6-d4fd-4ef4-b75e-b207cd6cb1dc/20250429_072732/monitor-set-ffe7cbc6.tar.zst

Jenkins job URL
Argus

[fruch]

@juliayakovlev can you reraise it in cassandra-stress new repo, no one is looking here anymore

[juliayakovlev]

@juliayakovlev can you reraise it in cassandra-stress new repo, no one is looking here anymore

@fruch here https://github.com/scylladb/cassandra-stress?

[fruch]

@juliayakovlev can you reraise it in cassandra-stress new repo, no one is looking here anymore

@fruch here https://github.com/scylladb/cassandra-stress?

Yes