409 on simultaneous leader status update

[jkubrynski]

Hi! I can confirm the problem described in #2032 exists and occurs under the following conditions:

1. Instance A is the leader, while instances B and C are running.
2. Instance A is terminated, and instances B and C attempt to acquire leadership at exactly the same time (in my case, the difference was 0.1s).
3. Instance B successfully obtains leader status, and instance C receives a 409 response from k8s.

I’m not sure if it’s even possible to create a sample project to reliably reproduce this issue.

However, the solution can be simple - just retry on a 409 error from k8s in the Fabric8LeadershipController.update() method.

[ryanjbaxter]

We could probably add some retry functionality to the update method but it would be very useful if we at least had a stacktrace to pinpoint where the 409 error is coming from and which API call is causing it