Potential TCP manager (re)start issue

[kyzrfranz]

I don't quite know if this is actually an issue but during testing I encountered the following behavior:
With ExternalClusterReference enabled:

• create a new kcp/tcp
• delete it again
• recreate the same resources

then the ExternalClusterReferenceReconciler will try to setup the TCP controller again running into an error. Only a restart will fix this. Is this intentional?
I did check a quick hack to just ignore the error for "blah blah .. duplicate tcp controller .. blah blah" and it went about it's busines fine.

wdyt?

[prometherion]

Thanks for opening this, @kyzrfranz, happy we could have this discussion in triaging a potential issue.

May I ask you to provide further logs in order to evaluate if it's a valid bug, or not?

[kyzrfranz]

Certainly, so this is from the second run:

2024-11-13T13:46:27+01:00       INFO    reconciliation started  {"controller": "kamajicontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "KamajiControlPlane", "KamajiControlPlane": {"name":"test-cluster","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "test-cluster", "reconcileID": "d9374408-db97-46f0-99d5-392a72a34f0a"}
2024-11-13T13:46:27+01:00       INFO    secret entry is referenced      {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"cpc-kubeconfig","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "cpc-kubeconfig", "reconcileID": "0c53e024-6292-47c1-bcd5-dbbfff40e21c", "key": "cp-test-cluster/cpc-kubeconfig/kubeconfig", "count": 1}
2024-11-13T13:46:27+01:00       INFO    new configuration, loading manager      {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"cpc-kubeconfig","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "cpc-kubeconfig", "reconcileID": "0c53e024-6292-47c1-bcd5-dbbfff40e21c"}
2024-11-13T13:46:27+01:00       ERROR   unable to get remote Client     {"controller": "kamajicontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "KamajiControlPlane", "KamajiControlPlane": {"name":"test-cluster","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "test-cluster", "reconcileID": "d9374408-db97-46f0-99d5-392a72a34f0a", "error": "remote manager is not yet initialized", "errorVerbose": "remote manager is not yet initialized {...}
github.com/clastix/cluster-api-control-plane-provider-kamaji/controllers.(*KamajiControlPlaneReconciler).Reconcile
       /cluster-api-control-plane-provider-kamaji/controllers/kamajicontrolplane_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
      {...}
2024-11-13T13:46:27+01:00       ERROR   unable to create controller     {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"cpc-kubeconfig","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "cpc-kubeconfig", "reconcileID": "0c53e024-6292-47c1-bcd5-dbbfff40e21c", "controller": "PushKamajiChange", "error": "controller with name tenantcontrolplane already exists. Controller names must be unique to avoid multiple controllers reporting to the same metric"}
github.com/clastix/cluster-api-control-plane-provider-kamaji/controllers.(*ExternalClusterReferenceReconciler).Reconcile
        /cluster-api-control-plane-provider-kamaji/controllers/externalclusterreference_controller.go:104
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
    {...}
2024-11-13T13:46:27+01:00       ERROR   Reconciler error        {"controller": "secret", "controllerGroup": "", "controllerKind": "Secret", "Secret": {"name":"cpc-kubeconfig","namespace":"cp-test-cluster"}, "namespace": "cp-test-cluster", "name": "cpc-kubeconfig", "reconcileID": "0c53e024-6292-47c1-bcd5-dbbfff40e21c", "error": "controller with name tenantcontrolplane already exists. Controller names must be unique to avoid multiple controllers reporting to the same metric"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
      {...}

caused by setting up the tcp manager a second time

I did a quick hack to verify if it is indeed the issue:

err = (&PushKamajiChange{ParentClient: r.Client, Client: mgr.GetClient(), TriggerChannel: r.TriggerChannel}).SetupWithManager(mgr)
if err != nil && !strings.HasSuffix(err.Error(), "Controller names must be unique to avoid multiple controllers reporting to the same metric") {
	log.Error(err, "unable to create controller", "controller", "PushKamajiChange")

	return ctrl.Result{}, err
}

This did in fact mitigate, but not sure if it's a real solution ;)

[prometherion]

Thanks, opened #152 which will be closed by #153: definitely a bug even tho it's an edge case! Thanks for contributing to Kamaji! ❤️