Change leader election to run in kube-system namespace when cluster-scoped and namespace is unspecified

[andrewstucki]

Fixes K8S-734

Prior to this commit when namespace is unspecified, then we wind up with:

opts.managerOptions.LeaderElectionNamespace = ""

as our default value in our leader election options handed to the controller-runtime manager due to opts.namespace not being set (as we're cluster-scoped).

When that's set to an empty string, the controller-runtime attempts to automatically detect the running namespace of the container via calling:

const inClusterNamespacePath = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"

func getInClusterNamespace() (string, error) {
	// Check whether the namespace file exists.
	// If not, we are not running in cluster so can't guess the namespace.
	if _, err := os.Stat(inClusterNamespacePath); os.IsNotExist(err) {
		return "", fmt.Errorf("not running in-cluster, please specify LeaderElectionNamespace")
	} else if err != nil {
		return "", fmt.Errorf("error checking namespace file: %w", err)
	}

	// Load the namespace file and return its content
	namespace, err := os.ReadFile(inClusterNamespacePath)
	if err != nil {
		return "", fmt.Errorf("error reading namespace file: %w", err)
	}
	return string(namespace), nil
}

and using that. What this means is that we have no enforcement mechanism for making the operator unable to run multiple cluster-scoped controllers simultaneously. Moving the lease namespace to kube-system enables all cluster-scoped operators by default to attempt the same lease lock, causing only a single operator to grab the leader election lock and run its controllers at any given time.