add non-default config that allows InboundQuarantineCheck to ignore 'harmless' quarantine events

[pjfanning]

• relates to Clustering issues leading to all nodes being downed #578
• basic tests that now watch for the quarantine event and with an experimental change to try to suppress that quarantine event when harmless=true
• this suppression is non-default and can be enabled with the config pekko.remote.artery.propagate-harmless-quarantine-events = off

[fredfp]

we may need to modify the test harmless=true test to send a message from one cluster member to the other to cause the shutdown issue

Without an active SBR, no node will be shutdown: it is the SBR that downs itself when receiving ThisActorSystemQuarantinedEvent. Without a cluster setup in the test (and as such without SBR running), we need to watch the ThisActorSystemQuarantinedEvent event instead, which is what I did to check the bug exists (test passes if the bug exists):

"eliminate quarantined association when not used (harmless=true)" in withAssociation {
  (remoteSystem, remoteAddress, _, localArtery, localProbe) =>
    remoteSystem.eventStream.subscribe(testActor, classOf[ThisActorSystemQuarantinedEvent]) // event to watch out for, indicator of the issue

    val remoteEcho = remoteSystem.actorSelection("/user/echo").resolveOne(remainingOrDefault).futureValue

    val localAddress = RARP(system).provider.getDefaultAddress

    val localEchoRef = remoteSystem.actorSelection(RootActorPath(localAddress) / localProbe.ref.path.elements).resolveOne(remainingOrDefault).futureValue
    remoteEcho.tell("ping", localEchoRef)
    localProbe.expectMsg("ping")

    val association = localArtery.association(remoteAddress)
    val remoteUid = futureUniqueRemoteAddress(association).futureValue.uid
    localArtery.quarantine(remoteAddress, Some(remoteUid), "HarmlessTest", harmless = true)
    association.associationState.isQuarantined(remoteUid) shouldBe true

    eventually {
      remoteEcho.tell("ping", localEchoRef) // trigger sending message from remote to local, which will trigger local to wrongfully notify remote that it is quarantined
      expectMsgType[ThisActorSystemQuarantinedEvent] // this is what remote emits when it learns it is quarantined by local. This is not correct and is what (with SBR enabled) triggers killing the node.
    }
}

[pjfanning]

I added the new test case but I am aware that it needs to be moved to the cluster or cluster-tests projects and the Split Brain Resolver added. I am busy on other tasks so don't expect to get back to this for a while.

[fredfp]

What would it add to move the test to the cluster or cluster-tests projects? To me this is a bug of the remote module and is better tested here. For sure, you could test consequences of the bug in cluster, but the root cause is here. Is that what you have in mind: to also cover/test the consequences?

[pjfanning]

I've added a change to InboundQuarantineCheck based on #578 (comment). This may not be the best solution but it seems to help in this one test case.

[fredfp]

It seems good to me like that, thank you!

[pjfanning]

@raboof @mdedetrich @jrudolph what do you think about the runtime change? We could add a config to users to control if the new runtime check is enabled.

[raboof]

I finally got a chance to review this change.

The fact that 'harmless' quarantines are not propagated, but are returned in InboundQuarantineCheck, indeed looks like a bug.

I think there's a good possibility that this caused the scenario in #578 and that it'd be worth releasing this new behavior. I'd even lean towards making the new behavior the default, but I'm also OK with being conservative and first testing with the reporters of #578.

Whether this fix completely prevents situations like the one described in #578 is harder to say with confidence: there's still quite a few situations that can cause non-harmless quarantines, and it's possible there's still situations where a single misbehaving node may 'take out' its surrounding peers.

[raboof]

This class is in private[remote] context, so indeed these changes are safe.

I think you could use a wildcard:

ProblemFilters.exclude[Problem]("org.apache.pekko.remote.artery.AssociationState#QuarantinedTimestamp*")

[pjfanning]

Thanks for the review. I was running the tests a few weeks ago and I'm not sure that they work. @fredfp tested this a few weeks ago and said it helped but I don't yet think this is ready - until the new tests are improved.

[He-Pin]

A workmate told me this ,but as I am not using the cluster at work, so thanks for take care of this hard task.

I knew a team is using akka/pekko cluster with a centralized nodes server, we called it vipserver, which will decided which nodes are on the same cluster as a single source of truth instead of this gossip thing, where a wrong message spreading the cluster can take the whole cluster down.

Another reason I'm not using clustering at work is because the chaos monkey, where the sre team will schedule some randomly network partition, and I think that will always require a reboot by the application owner, I am a little lazy here.

[pjfanning]

The tests seem to be working for me today. If reviewers are amenable, we could merge this to main and 1.1.x branches and document that the config exists to enable an experimental fix for #578.

[He-Pin]

@pjfanning @raboof I think we should backport this to 1.1.0, for people who can't upgrade netty but need this.

[pjfanning]

I can live with this being backported. It is a reasonably big change for a patch release but it is disabled by default so users who upgrade to 1.1.6 won't get this change unless they update their config settings.

[pjfanning]

@He-Pin doesn't this only affect users who use Artery remoting - and you say elsewhere that you are blocked by Netty issues?

[He-Pin]

@pjfanning Yes, she is using Artery, transport, but to fix this, she has to upgrade to pekko 1.2, and then Netty 4.2 cause another problem :(

[He-Pin]

@fredfp Did you try this release, would you like to share the result?

[fredfp]

We didn't try the fix (as it's only included in a milestone releases). I'd be very happy to turn the needed flag on if this made it into 1.1.6. However only time will tell as we don't get that kind of cluster crash very often.

[He-Pin]

@fredfp Thanks, there is a backport to 1.1.6 and 1.2.0 will be released soon too.

[fredfp]

Over the last couple of days, we've successfully deployed pekko 1.2.0 with pekko.remote.artery.propagate-harmless-quarantine-events = off in 2 clusters. Both seem stable, and more stable than before i.e., no more nodes downed with DownSelfQuarantinedByRemote. A great success so far, thank you!

[He-Pin]

@fredfp Thank you for that udpate~~, great sharing.

[He-Pin]

@fredfp Are you using the pekko-management at the sametime?

[fredfp]

Yes, we rely on pekko-management to create and then join the cluster.

[He-Pin]

One of my workmates said, her cluster node is killed by k8s, not sure why :(, k8s say the management port is unreachable, and the kill the node.

[pjfanning]

One of my workmates said, her cluster node is killed by k8s, not sure why :(, k8s say the management port is unreachable, and the kill the node.

@He-Pin this code has nothing to do with kubernetes. If you have pekko-management issues, it would be better to raise issues in the pekko-management project.

[He-Pin]

@pjfanning I knew, I just asking @fredfp if he knew that issue too.