Rackawareness in 5dc setup causes significant disbalance of first region nodes load

[vponomaryov]

Argus: scylla-staging/valerii/vp-rolling-upgrade-custom-d2-w1-latency-regression#11
NOTE: Not posting logs directly as it contains sensitive schema which must not be disclosed.

Test scenario details:

• Setup: 5 DCs with 3 i3en.xlarge DB nodes in each. Total - 15 DB nodes.
• First DC is special one from load point of view - 5 loaders in first DC and 1 in each other. Total - 9 loaders.
• Number of racks - 3

Latte benchmarking tool version: 0.32.0-scylladb
Scylla rust driver version: 1.3.1

So, in this scenario was enabled rackawareness on loaders.
Having 3 nodes, each in its own rack in each DC and 5 loaders in first region I expected following load proportions: 2 2 1
But, in fact, actual proportions were closes to 1 0 0.

Screenshot from monitoring:

The nodes from the first regions are 10.11.6.89 (node-1), 10.11.5.88 (node-2) and 10.11.7.67 (node-3).
As we can see on the screenshot, their load proportions are 12300 2 2

So, in fact, only first node among 3 in first region was used for load serving x3 requests.

Later test was failing because of the failing network services on this first node, but it is not the point of this bug report.

Latte code for handling DCs and racks is here: https://github.com/scylladb/latte/blob/a6701c7a8c5713d00cc29cfc9756a3c4afa01dda/src/scripting/connect.rs#L34-L77

Then, I disabled the rackawareness in other test runs.
And it instantly removed the problem.
Example test run: scylla-staging/valerii/vp-rolling-upgrade-custom-d2-w1-latency-regression#17
Screenshot:

[wprzytula]

Thank you for reporting this.

A few questions to make sure I understand the setup correctly:

Test scenario details:

• Setup: 5 DCs with 3 i3en.xlarge DB nodes in each. Total - 15 DB nodes.

• First DC is special one from load point of view - 5 loaders in first DC and 1 in each other. Total - 9 loaders.

1. Here you mention 9 loaders. Then, in the problematic scenario you mention only 5 loaders. Can you confirm that there were only 5 loaders in the problematic scenario?

• Number of racks - 3

Latte benchmarking tool version: 0.32.0-scylladb Scylla rust driver version: 1.3.1

So, in this scenario was enabled rackawareness on loaders. Having 3 nodes, each in its own rack in each DC and 5 loaders in first region I expected following load proportions: 2 2 1 But, in fact, actual proportions were closes to 1 0 0.

2. What do you mean by loaders residing in the first region? What's region, compared to DCs and racks?

3. What configuration did the 5 loaders receive wrt the preferred DC and rack? In particular, I mean conf.rack.

4. Why would you expect 2 2 1 proportions? I have no idea what reasonable setup would result in such proportions in the scenario you described.

[vponomaryov]

Thank you for reporting this.

A few questions to make sure I understand the setup correctly:

Test scenario details:

• Setup: 5 DCs with 3 i3en.xlarge DB nodes in each. Total - 15 DB nodes.
• First DC is special one from load point of view - 5 loaders in first DC and 1 in each other. Total - 9 loaders.

1. Here you mention 9 loaders. Then, in the problematic scenario you mention only 5 loaders. Can you confirm that there were only 5 loaders in the problematic scenario?

Region-1:

• DB-node-1:rack0 - serves 99.99% of load from loaders of this region
• DB-node-2:rack1 - almost idle
• DB-node-3:rack2 - almost idle
• Loader-node-1:rack0
• Loader-node-2:rack1
• Loader-node-3:rack2
• Loader-node-4:rack0
• Loader-node-5:rack1

Region-2:

• DB-node-4:rack0
• DB-node-5:rack1
• DB-node-6:rack2
• Loader-node-6:rack0

Region-3:

• DB-node-7:rack0
• DB-node-8:rack1
• DB-node-9:rack2
• Loader-node-7:rack0

Region-4:

• DB-node-10:rack0
• DB-node-11:rack1
• DB-node-12:rack2
• Loader-node-8:rack0

Region-5:

• DB-node-13:rack0
• DB-node-14:rack1
• DB-node-15:rack2
• Loader-node-9:rack0

• Number of racks - 3

Latte benchmarking tool version: 0.32.0-scylladb Scylla rust driver version: 1.3.1
So, in this scenario was enabled rackawareness on loaders. Having 3 nodes, each in its own rack in each DC and 5 loaders in first region I expected following load proportions: 2 2 1 But, in fact, actual proportions were closes to 1 0 0.

2. What do you mean by loaders residing in the first region? What's region, compared to DCs and racks?

Since the availability zones are not point of discussion here, datacenter (DC) should be considered as identical to region.

4. What configuration did the 5 loaders receive wrt the preferred DC and rack? In particular, I mean conf.rack.

Loaders received appropriate dc and rack params as stated above.
It can be verified looking at each latte log file where following may be seen:

      Datacenter            eu-central-1                                                              
            Rack            RACK0  

Each latte command log contains loader index such ...-l1-...

6. Why would you expect 2 2 1 proportions? I have no idea what reasonable setup would result in such proportions in the scenario you described.

If I configure loaders №1 and №4 to be aware of the rack-0, loaders №2 and №5 to be aware of the rack-1 and loader №3 about rack-2 I expect exactly 2 2 1 proportions.

[Lorak-mmk]

@vponomaryov Is this test using tablets?

[Lorak-mmk]

Ah, this is Scylla 2024.1, so no tablets.

[Lorak-mmk]

Where can I find the schema, especially replication strategy?

[vponomaryov]

Where can I find the schema, especially replication strategy?

Details about custom schema are here:
https://github.com/scylladb/scylla-qa-internal/blob/master/custom_d2/workload1/latte/custom_d2_workload1.rn

Note that this repo is private and nothing from it should be posted in here - public repo.