QQs: check if schema database record needs an update on tick

[SimonUnge]

Proposed Changes

Let QQ run rabbit_quorum_queue:repair_amqqueue_nodes on tick to repair potential membership discrepancy (RA members compared to amqqueue state members)

See #11029

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

[michaelklishin]

This can introduce a lot of CPU burn in clusters with thousands of quorum queues. Has the CPU impact of this been measured?

[lukebakken]

Relevant: #7863

[SimonUnge]

This can introduce a lot of CPU burn in clusters with thousands of quorum queues. Has the CPU impact of this been measured?

Nope, but I will run some benchmark tests before final review!

[SimonUnge]

@kjnilsson Looking at the repair_amqqueue_nodes, what could perhaps be a bit costly is the ra:members(Leader) call, perhaps the members could be provided in the tick from RA itself?

(there are of course costs in actually repairing too, but I would assume that a queue actually needs repairing is unusual, and that a lot of queues needing it even more so)

[kjnilsson]

@kjnilsson Looking at the repair_amqqueue_nodes, what could perhaps be a bit costly is the ra:members(Leader) call, perhaps the members could be provided in the tick from RA itself?

(there are of course costs in actually repairing too, but I would assume that a queue actually needs repairing is unusual, and that a lot of queues needing it even more so)

The ra_leaderboard should have the current members which you can use to avoid the ra:members/1 call back into the process.

[SimonUnge]

@kjnilsson Looking at the repair_amqqueue_nodes, what could perhaps be a bit costly is the ra:members(Leader) call, perhaps the members could be provided in the tick from RA itself?
(there are of course costs in actually repairing too, but I would assume that a queue actually needs repairing is unusual, and that a lot of queues needing it even more so)

The ra_leaderboard should have the current members which you can use to avoid the ra:members/1 call back into the process.

Aha, I'll update the function to use that one instead.

[SimonUnge]

@michaelklishin
Michal ran 10k QQs on a 3 node cluster (all 10k queue leaders on the same node), and produced some flame graphs (and, taught me how to do it, for the next time! Much obliged, thank you very much @mkuratczyk!)

With the addition of rabbit_quorum_queue:repair_amqqueue_nodes, and the change to call ra_leadership:lookup_members, the graphs show that 0.53% CPU time is spent in rabbit_quorum_queue:repair_amqqueue_nodes, 0.42% spent in ra_leadership:lookup_members.

[michaelklishin]

In other words, with the more efficient Ra leaderboard-based implementation we waste about 1% of the CPU with 10K quorum queues.

[SimonUnge]

In other words, with the more efficient Ra leaderboard-based implementation we waste about 1% of the CPU with 10K quorum queues.

I wrote it a bit weird, but it would be 0.5% CPU with 10k QQs in total (the 0.42 is part of the 0.53)

[michaelklishin]

@Mergifyio backport v3.13.x

[mergify]

backport v3.13.x

✅ Backports have been created

• #11361 QQs: check if schema database record needs an update on tick (backport #11278) has been created for branch v3.13.x