Multiple ACK controller shards watch resources conflict

[jas-nik]

Describe the bug
Hi Team,

We have a use case where we deploy ack controllers shards with independent CARM and watch selectors configurations. First shard of ACK controllers is configured to use "team:role" CARM configuration, Cluster scoped (ACK_WATCH_NAMESPACE or ACK_WATCH_SELECTORS are not configured). Second shard of ack controllers using "service:account:role" CARM setup with watch selectors configured

Due to the "team:role" CARM config ACK controller shard running without watch selectors, all ACK resources on the cluster were being reconciled by this controller shard. This reconciliation conflicts with the second ACK controller shard with WATCH selectors.

We added WATCH SELECTOR to the first ACK controller shard running at cluster scope to prevent it from reconciling all resources on the cluster.

The issue now is that the second ACK controller shard which is supposed to reconcile/watch resources is ignoring them.

Restarting both the controllers or scaling down one set of the controllers did not help.

Resource: EC2 SecurityGroup

Steps to reproduce

• Deploy and configure EC2 controller in two different namespaces one with team ID carm setup and the other with service:account:role CARM setup
• Add WATCH SELECTOR to only the controller with service:account:role CARM setup
• Create a set of resources(ec2:securitygroup) that match the selector from the previous step

Expected outcome

• Both controller shards should be able to reconcile and dynamically manage resources depending on selectors configured.

Environment

• Kubernetes version - 1.32
• Using EKS (yes/no), if so version? v1.32.8-eks-e386d34
• AWS service targeted (S3, RDS, etc.) EC2 - SecurityGroup Resource

[github-actions]

Hello @jas-nik 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[jas-nik]

Is there a controller <-> resource mapping outside of the controllers' cache? Similar to how Kubernetes manages Ownership references

[jas-nik]

Issue was with the first ACK controller shard going through the adoption flow partially, adding "adopted:true" annotation to the resource as well as a finalizer. Removing annotation and finalizer fixed the issue with the 2nd controller shard.

Thanks for your help! @michaelhtm @a-hilaly and Mani!

The error was misleading stating "Resource Not Found" - Could we add this case to error handling?
Also adding an annotation to the resource to see which controller has the "Ownership" would be really useful.

[michaelhtm]

The error was misleading stating "Resource Not Found" - Could we add this case to error handling?

I agree, I believe the 2nd controller shard was patching the annotation and finalizer, and then the 1st shard patched the status with the error.
We need to investigate why the second shard was no longer able to update the status with its own errors, even when the first shard was no longer around.

Also adding an annotation to the resource to see which controller has the "Ownership" would be really useful.

That makes sense.
What identifier should we use for ownership (deployment name/namespace)?
Is this Ownership annotation only for user to know who's reconciling the resource, or should the controller consider it when reconciling a resource (ignore resources you don't own, ownership transfer logic, etc.)?

[jas-nik]

What identifier should we use for ownership (deployment name/namespace)?

That sounds good.

Is this Ownership annotation only for user to know who's reconciling the resource, or should the controller consider it when reconciling a resource (ignore resources you don't own, ownership transfer logic, etc.)?

Ignoring resources owned by a different controller can protect resources in scenarios like this by ensuring only one controller manages a resource at any given time.