Decoupling controller-runtime from k8s leader election libraries #3331
Description
Summary
This proposal aims to provide the option to decouple controller-runtime's leader election implementation from client-go's leaderelection and resourcelock packages.
Motivation
Currently, controller-runtime is tightly coupled to client-go's leader election implementation, which limits the usage of controller-runtime for use cases outside of Kubernetes. For instance, this will allow cross-k8s cluster leader election to be implemented for controllers.
Proposed Solution
Rough outline of the solution would look like this:
Create backward compatible interfaces for leader election
// pkg/leaderelection/interfaces.go
type LeaderElector interface {
Run(ctx context.Context, config LeaderElectionConfig) error
GetLeader() string
IsLeader() bool
}
// Needed for client-go compatibility
type ClientGoLeaderElectorWrapper struct {
elector *leaderelection.LeaderElector
}
type LeaderElectionConfig struct {
Identity string
LeaseDuration time.Duration
RenewDeadline time.Duration
RetryPeriod time.Duration
Name string
Clock clock.Clock
}
func (w *ClientGoLeaderElectorWrapper) Run(ctx context.Context, config LeaderElectionConfig) error {
// client-go's Run doesn't take config, so we ignore it
w.elector.Run(ctx)
return nil
}
func (w *ClientGoLeaderElectorWrapper) GetLeader() string {
return w.elector.GetLeader()
}
func (w *ClientGoLeaderElectorWrapper) IsLeader() bool {
return w.elector.IsLeader()
}
Add new options to the Manager class
// pkg/manager/manager.go
type Options struct {
// ... existing fields ...
CustomLeaderElection LeaderElector
// LeaderElectionClock allows providing a custom clock for leader election timing
LeaderElectionClock clock.Clock
}
Updates to initLeaderElection function in manager
func (cm *controllerManager) initLeaderElector() (LeaderElector, error) {
// If custom leader election provided, use it
if cm.customLeaderElection != nil {
return cm.customLeaderElection, nil
}
// Use existing client-go mechanism (unchanged)
clientGoElector, err := leaderelection.NewLeaderElector(/* existing config */)
if err != nil {
return nil, err
}
// Wrap to match newly defined interface
return &ClientGoLeaderElectorWrapper{elector: clientGoElector}, nil
}
Updates to Start function in manager
func (cm *controllerManager) Start(ctx context.Context) (err error) {
// ... existing code ...
var leaderElector LeaderElector
if cm.resourceLock != nil {
leaderElector, err = cm.initLeaderElector()
if err != nil {
return fmt.Errorf("failed during initialization leader election process: %w", err)
}
}
// ... existing code ...
if leaderElector != nil {
// Create config
config := LeaderElectionConfig{
Identity: cm.leaderElectionID,
LeaseDuration: cm.leaseDuration,
RenewDeadline: cm.renewDeadline,
RetryPeriod: cm.retryPeriod,
Name: cm.leaderElectionID,
Clock: cm.leaderElectionClock,
}
go func() {
err := leaderElector.Run(leaderCtx, config)
if err != nil {
cm.errChan <- err
}
close(cm.leaderElectionStopped)
}()
}
// ... existing code ...
}
Some of the options exposed in the Options struct can be deprecated and streamlined with the new capability to specify the LeaderElector and potentially simplify how the ControllerManager handles LeaderElection