feat: add debug logging to diagnose label sync issues [minor] #51
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When investigating potential issues with node labels syncing to EC2 was no visibility into the controller's decision-making. The predicates silently filter events, and reconcile doesn't log which labels it finds vs expects. Without this, debugging requires guessing whether the predicate rejected the event or the label simply wasn't on the node. Adds V(1) debug logs throughout the sync pipeline. The predicate now logs every Create/Update event with `shouldProcess=true/false` so you can see if events are being filtered. Reconcile logs which monitored labels are missing from the node and the final set of tags being synced. Replaces the custom `--json` flag with controller-runtime's standard zap flags (`--zap-log-level`, `--zap-encoder`, etc). To see debug logs, run with `--zap-log-level=debug` or `--zap-log-level=1`. Additionally, introduce a new 4 hour resync period and update the predicate on Update events to allow a full resync to the cloud provider to catch any potentially missed label sync events. This is a trade-off between eventual consistency and the cost of the extra DescribeTags calls. > NOTE: Due to the dropping of `--json` flag and replacing with zap flags, > the minor version is being bumped to v0.1.0.
mickael-carl
reviewed
Dec 9, 2025
controller.go
Outdated
| if value, exists := node.Labels[k]; exists { | ||
| tagsToSync[k] = value | ||
| } else { | ||
| logger.V(1).Info("Monitored label not found on node", "label", k) |
There was a problem hiding this comment.
Won't that write N line, one for each label?
Member
Author
There was a problem hiding this comment.
it does, but it turns out to be quite readable due to the fixed size of the preceding text in the log line. I will try to accumulate and do one log to see how it looks instead tho. The goal here is to print much more info than the current logging to try to chase down some elusive edge cases
ex:
{"level":"debug","ts":"2025-12-09T17:28:22Z","logger":"controller","msg":"Monitored label not found on node","node":{"name":"ip-10-0-126-121.ec2.internal"},"label":"foo"}
{"level":"debug","ts":"2025-12-09T17:28:22Z","logger":"controller","msg":"Monitored label not found on node","node":{"name":"ip-10-0-126-121.ec2.internal"},"label":"foo-bar"}
{"level":"debug","ts":"2025-12-09T17:28:22Z","logger":"controller","msg":"Monitored label not found on node","node":{"name":"ip-10-0-126-121.ec2.internal"},"label":"foo-bar-baz"}
Member
Author
There was a problem hiding this comment.
changed to accumulating with one log line. slightly less readable in terminal but might be better from a log viewer (ie: loki, vlogs) view. will try it out.
mickael-carl
approved these changes
Dec 9, 2025
…gging them individually
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When investigating potential issues with node labels syncing to EC2 was no visibility into the controller's decision-making. The predicates silently filter events, and reconcile doesn't log which labels it finds vs expects. Without this, debugging requires guessing whether the predicate rejected the event or the label simply wasn't on the node.
Adds V(1) debug logs throughout the sync pipeline. The predicate now logs every Create/Update event with
shouldProcess=true/falseso you can see if events are being filtered. Reconcile logs which monitored labels are missing from the node and the final set of tags being synced.Replaces the custom
--jsonflag with controller-runtime's standard zap flags (--zap-log-level,--zap-encoder, etc). To see debug logs, run with--zap-log-level=debugor--zap-log-level=1.Additionally, introduce a new 4 hour resync period and update the predicate on Update events to allow a full resync to the cloud provider to catch any potentially missed label sync events. This is a trade-off between eventual consistency and the cost of the extra DescribeTags calls.