Summary
Vigil's ServiceAccount lacks permission to create Events for cluster-scoped objects (Nodes). Every successful taint removal logs an RBAC error.
Observed error
E0427 18:06:03.709475 1 event_broadcaster.go:270] "Server rejected event (will not retry!)"
err="events.events.k8s.io is forbidden: User \"system:serviceaccount:vigil-system:vigil-vigil-controller\" cannot create resource \"events\" in API group \"events.k8s.io\" in the namespace \"default\""
event="...Reason:TaintRemoved,Regarding:{Node ip-10-32-25-206.us-west-2.compute.internal ...},Note:All 13 expected DaemonSet pods are Ready,..."
Root cause
Kubernetes routes Events for cluster-scoped objects (Nodes) into the default namespace by convention. Vigil's Helm chart binds events create permission only on the install namespace (vigil-system), so every legitimate TaintRemoved event Vigil tries to publish is rejected.
Impact
- Operators can't
kubectl describe node <name> and see Vigil's actions reflected in events
- Audit trail for the
nextdoor.com/initializing taint lifecycle is missing
- Log noise: error stack on every successful taint removal
- Functionality is otherwise unaffected — taint removal still works
Proposed fix
In the chart's RBAC, change the events-create permission from a namespaced RoleBinding to a ClusterRoleBinding, or grant it explicitly on the default namespace. The standard upstream pattern for controllers that act on cluster-scoped objects is a ClusterRole with events.k8s.io/events: [create, patch].
Summary
Vigil's ServiceAccount lacks permission to create Events for cluster-scoped objects (Nodes). Every successful taint removal logs an RBAC error.
Observed error
Root cause
Kubernetes routes Events for cluster-scoped objects (Nodes) into the
defaultnamespace by convention. Vigil's Helm chart bindseventscreate permission only on the install namespace (vigil-system), so every legitimateTaintRemovedevent Vigil tries to publish is rejected.Impact
kubectl describe node <name>and see Vigil's actions reflected in eventsnextdoor.com/initializingtaint lifecycle is missingProposed fix
In the chart's RBAC, change the events-create permission from a namespaced
RoleBindingto aClusterRoleBinding, or grant it explicitly on thedefaultnamespace. The standard upstream pattern for controllers that act on cluster-scoped objects is a ClusterRole withevents.k8s.io/events: [create, patch].