Predicates and error handling - don't clash?

[Danil-Grigorev]

Hi, I've got a question regarding usage of predicates. This functionality is currently behind feature gate, but it is useful for the controllers in general.

It seems though that it's only purpose is to compare against a previous resource state, scoped down to a hash of previous values.

Does this mean that an event which errors out down the processing chain after being evaluated by the predicate and gets re-added to the queue will be filtered out as well?

Also that mechanism could support configurable behavior on resource deletion, because these events are valuable and shouldn't be missed

[clux]

Does this mean that an event which errors out down the processing chain after being evaluated by the predicate and gets re-added to the queue will be filtered out as well?

Short answer; no.

Long answer; there are two ways main path ways events enter the big reconcile stream in applier and predicates only filter one of them. The predicates are applied pretty early to the input stream(s) (sometimes even on the user side with the streams interface). They filter out repeat events from the Kubernetes watch side where "no real changes happen" (as far as the predicate hash is concerned), but they do not affect internal rescheduling in the applier as these are fed back into the reconcile stream later via a channel.

Specifics

If an input stream that is predicated triggers a reconcile (i.e. the first event of this predicate hash) and this reconcile fails (or if you ask for an Action::requeue), then this still causes a RescheduleReconciliation and these requests are blended into the combined stream for reconciliation in the trystream_try_via part in applier.

If you return Ok(Action::await_change()) from a reconciler on the other hand, then kube will not reschedule the object. So if you have both predicates and no re-queuing going on (via either reconciler or error policy; both return an Action) then you may mute the object (with respect to predicate hash) entirely.

[Danil-Grigorev]

Interesting, thanks for the answer. I observed a strange behavior in a custom controller setup while using labels predicate and shared_stream setup methods, where no reconciles were invoked after controller restart on existing resources. It is as if some error handling was missing, and didn’t request another reconcile.

Removing labels predicate did help. I suspect that was due to other reasons, like original stream being polled and consumed objects earlier than a downstream subscriber stream got registered. So I felt compelled to open a discussion.

I’ll hope to followup on the issue if this is something which will appear later.