Fix flaky test TestClusterJoinAndReconnect/TestTLSConnection again (#4635)

Two commits have already merged in order to address the flakiness of this test.

However, I can still reproduce the issue using:
```sh
go test -failfast -run "TestClusterJoinAndReconnect/TestJoinLeave" -count 600 ./cluster
```
An easy way to increase the failure rate is to increase CPU load, e.g.,
```sh
yes > /dev/null &; yes > /dev/null &; yes > /dev/null &; yes > /dev/null &
```
On my machine the combination of these commands fails every time.

The underlying reason for the failure is that the test only waits for
`p2` to be ready, but this does not reflect whether `p` has updated its
memberlist. The test now retries the assertions in question using
`Eventually`.

Fixes #3287

Signed-off-by: Solomon Jacobs <[email protected]>

Don't modify cluster for TestClusterJoinAndReconnect/TestTLSConnection

Author: SoloJacobs

cluster/cluster_test.go

@@ -106,9 +106,9 @@ func testJoinLeave(t *testing.T) {
 	go p2.Settle(context.Background(), 0*time.Second)
 	require.NoError(t, p2.WaitReady(context.Background()))
 
-	require.Equal(t, 2, p.ClusterSize())
+	require.Eventually(t, func() bool { return p.ClusterSize() == 2 }, 5*time.Second, time.Second)
 	p2.Leave(0 * time.Second)
-	require.Equal(t, 1, p.ClusterSize())
+	require.Eventually(t, func() bool { return p.ClusterSize() == 1 }, 5*time.Second, time.Second)
 	require.Len(t, p.failedPeers, 1)
 	require.Equal(t, p2.Self().Address(), p.peers[p2.Self().Address()].Address())
 	require.Equal(t, p2.Name(), p.failedPeers[0].Name)