disputes: implement validator disabling (Stale)

[ordian]

⚠️ FOR THE MOST UP TO DATE INFO REFER TO: Disabling Project Board and this meta overview State of Disabling ⚠️

---

Once a dispute is concluded and an offence is submitted with DisableStrategy::Always, a validator will be added to DisabledValidators list.

Implement on-chain and off-chain logic to ignore dispute votes for X sessions. Optionally, we can ignore backing and approval votes and remove from the reserved validator set on the network level.

• Gather requirements across stack
• Design unified disabling strategy (Guide Update with all considerations #1961)
• Track disabled validators Disabled Validators Runtime API #2950
• Read disabled status from the runtime and apply on node side, for ignoring votes for example in dispute-coordinator.
  • Validator disabling in statement-distribution subsystem #1591 statement-distribution: validator disabling #1841
  • Enforce validator disabling in the runtime #1592 Filter votes from disabled validators in BackedCandidates in process_inherent_data #1863
  • dispute-coordinator - Disablement: Dispute participation disabling #2225
• Make disabled state persisted for full era (to avoid repeated slashes)
• Remove I am online slash
• No chilling: Validators should get re-elected as long as they have enough stake
• Handle validators of previous era correctly in disputes
• Implement re-enabling
• Testing
  • Zombienet testing for validator disabling #1590
  • Test validator disabling on Westend #2249
• Provide tooling and instruct validators to monitor for slash events, so we can expect honest operators to react quickly to issues. (Within an era.)
• Cleanup tasks
  • Cleanup boilerplate code when DisabledValidators runtime api call is released  #1940

Possibly related paper here.

⚠️ FOR THE MOST UP TO DATE INFO REFER TO: Disabling Project Board ⚠️

Goals for new validator disabling/Definition of Done

1. Not affecting consensus - disabling can never become a security threat.
2. Handling broken validators nicely (prevent continuous spam).
3. Plays well with existing disabling in substrate
4. Makes sure to never break.
5. (Consequence of the above): We can enable slashing - safe and securely.

Timeline

As quickly as possible, definitely by the end of the year.

[eskimor]

Also disable validators who repeatedly vote against valid. Disabling means in general that we should not accept any votes/statements from that validator for some time, those include:

• backing
• approval
• explicit dispute statements

In addition, depending on how quickly we disable a validator, it might already have raised thousands of disputes (if it disputes every single candidate for a few blocks), we should also consider deleting already existing disputes (at the dispute-coordinator) in case one side of the dispute consists only and exclusively of disabled validators - so we apply disabling to already pending participations, not just new ones.

This might be tricky to get right (sounds like it could be racy). The reason we should at least think about this a bit, is that so many disputes will delay finality for a significant amount of time resulting in DoS.

Things to consider:

• How quickly do we disable?
• How much does the rate limiting in dispute-distribution help?
• Is the risk worth the complication?

[ordian]

Also disable validators who repeatedly vote against valid.

That's tracked in #785 and is purely runtime changes.

How quickly do we disable?

We can disable as soon as a dispute (reaching threshold) concludes.

This might be tricky to get right (sounds like it could be racy)

Indeed. I'd be in favor of not complicated this unnecessarily.

[eskimor]

Just had a discussion with @ordian . So what is the point of disabling in the first place? It is mostly about avoiding service degradation due to some low number of misbehaving nodes (e.g. just one). There are other mechanism in place which provide soundness guarantees even with such misbehaving nodes, but service quality might suffer for everybody (liveness).

On the flip side, with disabling, malicious actors could take advantage of bugs/subtle issues to get honest validators slashed and thus disabled. Therefore disablement if done wrong, could actually lead to security/soundness issues.

With this two requirements together, we can conclude that we don't need perfect disablement, but an effective rate limit for misbehaving nodes is enough to maintain service quality. Hence we should be able to limit the number of nodes being disabled at any point in time, to something like 10% maybe 20% ... in any case to something less than 1/3 of the nodes. If this threshold is reached, we can either by random choice or based on the amount of accumulated slashes (or both) enable some nodes again.

This way we do have the desired rate limiting characteristics, but at the same time make it unlikely that an attacker can get a significant advantage via targeted disabling.

Furthermore as this is about limiting the amount of service degradation a small number of nodes (willing to get slashed) can cause, it makes sense to only start disabling once a certain threshold in accumulated slashes is reached.

For the time being, we have no reason to believe that these requirements are any different for disabling in other parts of the system, like BABE. We should therefore double check that and if it holds true strive for a unified slashing/disabling system that is used everywhere through the stack in a consistent fashion.

[eskimor]

1. Figure out a disabling strategy that limits severeness of honest nodes getting disabled.
2. keep the network functional in all cases: have enough validators enabled for grandpa to work.
3. Expose an API to the node, for retrieval of disabled validator.
4. Don't accept statements/votes from disabled validators on node and runtime.
5. Don't accept connections from disabled validators

[tdimitrov]

I'll leave my thoughts on a strategy for validator disabling here so that we can discuss it and improve it further (unless it's a total crap 💩).

When a validator gets slashed it's disabled following these rules:

1. The validator will be disabled during the rest of the session. Or in other words - the list of disabled validators will be cleared on each session start.
2. No more than BYZANTINE_THRESHOLD validators are disabled at the same time. Otherwise we'll break the network.
3. Each validator will have an offense score indicating how bad his offense was. I think it's safe to use the slash amount for this score. When we reach BYZANTINE_THRESHOLD number of disabled validators, we can re-enable a small offender so that we can disable a bigger one.
4. If we reach a point where the total offense score is BYZANTINE_THRESHOLD * SLASH FOR SERIOUS OFFENSE we can force a new era, because we have got too much offenders in the active set.

Open questions:

• Is it enough to disable a validator for a single session? We can also pick a period based on the seriousness of the offense but I prefer to start simple.
• Is 4 from the list above an over-complication?
• Should we keep track of the offense score of a validator? For example our disabled list is almost full. We add validator A for a small offense. Then validator B makes something more severe so we remove A and add B. Then validator A does something bad again. What should be his offense score - old score + new offense score or just new offense score? The latter makes more sense to me but it will require extra runtime storage.
• Considering the previous point - should we disable validator for a really minor offenses? E.g. voting invalid for a valid candidate? This is related to disputes: punishment on repeated dispute initiations (stale) #785. The alternative is just to increase it's offense score and disable it if it keeps on causing problems.

[eskimor]

Reiterating Requirements:

1. For re-enabling slashes for approval voters, we need disablement being proportional to the slash.
2. We would like to rate limit pretty quickly to avoid validators accumulating slashes too much in case of bugs/hardware faults.
3. We need to make sure to never disable too many validators, as this would cause consensus issues. Target should be adjustable, but 10% seems like a reasonable number.

2 is conflicting with 1, as a small slash would result in barely any rate limiting. On the flip side, if a node is misbehaving it is definitely better to have it disabled and protect the network this way, than keep slashing the node over and over again for the same flaw.

Luckily there is a solution to these conflicting requirements: Having the disabling strictly proportional to the slash is only necessary once a significant number of nodes would get disabled, hence we can introduce another (lower) threshold on number of slashed nodes, if it is below that threshold we just disable all of them, regardless of the amount.

Meaning of Disabling

Disabled nodes will always be determined in the runtime, so we do have consensus. There should be an API for the node to retrieve the list of currently disabled nodes as per a given block. The effect will be that no data from a validator disabled in a block X, should ever end up in block X+1. For simplicity and performance we will ignore things like relay parents of candidates, all that is relevant is the block being built. On the node side, we do have forks, therefore we will ignore data from validators as long as a disabling block is in our view.

Runtime

• Filter out any statements (backing, dispute, approval, ..) from disabled nodes: Disabled nodes are not able to back a candidate, nor can they raise a dispute/participate in it.
• Filter out bitfields from disabled nodes.

Node

For all nodes being disabled in at least one head in our current view:

• Don't connect to disabled nodes, remove them from the reserved set + actively drop any connection attempts. (Maybe only do this for 100% slashed nodes: slash amount/offense score should be exposed)
• Don't accept any statements from disabled nodes (backing, approval, disputes)
• Honest nodes should also honor themselves being disabled and should not issue any statements/doing any validations if they have any blocks in their view, for which they are disabled. This is mostly for self protection: If they relied only on others to ignore their dispute statements, they might still get in, in a later block, where they are enabled again - causing them to get slashed again.

Affected subsystems:

• Provisioner - should filter out any data from disabled validators based on the disabled state of the block currently building upon.
• Dispute coordinator: Should early drop statements from disabled validators to avoid participation/escalation, this includes own statements if we are disabled ourselves.
• Backing should ignore statements from disabled validators for a given block, this is so we don't end up validating a candidate proposed by a malicious backer, wasting resources. This is less important than provisioner and dispute coordinator.
• Approval subsystems should ignore assignments and approvals as long as a disabling block is in view. This is also less important than the provisioner and the dispute-coordinator changes.

If we wanted to go fully minimal on nodes side changes, it should be enough to honor disabled state in the dispute coordinator. Degradation in backing performance should be harmless, approval subsystems are also robust against malicious actors and filtering in the provisioner is strictly speaking redundant as the filtering will also be performed in the runtime.

Disabling Strategy

We will keep a list of validators that have been slashed, sorted by slash amount. For determining for the current block, which validators are going to be disabled we do the following:

1. We check whether the list of currently slashed validators is less than lower threshold amount (see above), if so - all slashed validators go on the disabled list and we skip the remaining points.
2. For each slashed validator, add it to the list of disabled validators randomly with a probability equal to their slash amount: 100% slash - always on the list, 10% slash - in 10% of the time, ..
3. We check whether the list of disabled validators is less than 10% of all validators, if not we randomly remove nodes from the disabled list until we reached the threshold.

I would suggest to ignore slash amount in 3 for simplicity, because:

1. The higher the slash the higher the probability to be on the list to begin with, so we are already weighing based on slash.
2. The protocols should be robust against a few rogue validators, having nothing to lose.
3. Having so many nodes disabled is an edge case, that should never happen and if it did it is very likely due to a bug: Therefore while 100% slashed nodes have nothing to lose, it is actually quite likely that less slashed validators don't behave any better regardless.

Rule 1 protects the network from single (or a low amount) of rogue validators and also protects those validators from themselves: Instead of getting slashed over and over again, they will end up being disabled for the whole session. Giving operators time to react and fix their nodes. (See point 2 in requirements)

This means we will have two thresholds: One where, as long as we are below we always disable 100% and one where, once we are above start to randomly enable validators again.

Disabling, eras, sessions, epochs

Information about slashes should be preserved until a new validator set is elected. With a newly elected validator set, we can drop information about slashed validators and start anew with no validators disabled.

If we settle on this approach, then this would be obsoleted by the proposed threshold system.

[tdimitrov]

Two questions/comments:

For all nodes being disabled in at least one head in our current view:

Why head in current view instead of 'slashed in finalized block'? To be proactive in case of finality stall?

And the second related to disabling stragegy:

3. We check whether the list of disabled validators is less than 10% of all validators, if not we randomly remove nodes from the disabled list until we reached the threshold.

I think we should do this in two steps:

1. Randomly remove nodes which are not big offenders (100% slash).
2. If all the nodes in the list are big offenders - start removing them randomly too.

[eskimor]

I think we should do this in two steps:

1. Randomly remove nodes which are not big offenders (100% slash).

2. If **all** the nodes in the list are big offenders - start removing them randomly too.

Yes, we could do that, but I argued above that we should be able to keep it simple without any harm done.

Why head in current view instead of 'slashed in finalized block'? To be proactive in case of finality stall?

Yes. Given that attacks on disputes can trigger a finality stall, it would be really bad if attackers could avoid getting disabled by their very attack. While at the same time for honest, but malfunctioning nodes they might already accumulate a significant amount of slash before getting disabled.