Skip to content

Implement log deduplication to minimize repetitive messages #525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link

@Copilot Copilot AI commented Aug 1, 2025

This PR implements a memory mechanism in the logging system to prevent consecutive duplicate log messages from accumulating, addressing the issue where repetitive operations like IP address resolution queries would spam the logs.

Problem

The DDNS Updater application generates many duplicate log messages during normal operation, particularly:

  • Repeated IP resolution success messages when the public IP hasn't changed
  • "No dynamic DNS record needs to be updated" messages during regular intervals
  • Connection error messages that occur repeatedly

These duplicate messages clutter the logs and make it difficult to identify actual changes or issues.

Solution

Added a thread-safe deduplication mechanism to the logging package (pkg/logging/logging.go) that:

  1. Tracks recent messages: Uses a LogMemory structure with mutex protection to track the last logged message and occurrence count for each log level
  2. Suppresses consecutive duplicates: Only logs the first occurrence of identical messages, silently counting subsequent duplicates
  3. Provides duplicate summaries: New ClearDuplicateMemory() function shows how many times each message was suppressed before clearing the memory
  4. Preserves critical logs: Fatal messages are never deduplicated to ensure important errors are always visible

Example

Before this change, repeated IP queries would generate:

2025/08/01 06:28:41 INF ipify succeeded: 203.0.113.42
2025/08/01 06:28:42 INF ipify succeeded: 203.0.113.42  
2025/08/01 06:28:43 INF ipify succeeded: 203.0.113.42
2025/08/01 06:28:44 INF ipify succeeded: 203.0.113.42

After this change:

2025/08/01 06:28:41 INF ipify succeeded: 203.0.113.42
... (duplicates suppressed)
2025/08/01 06:28:45 INF (message repeated 4 times): ipify succeeded: 203.0.113.42

Implementation Details

  • Modified Infof() and Errorf() functions to check for duplicates before logging
  • Added shouldLog() method that returns false for consecutive identical messages
  • Both console and file logging respect the deduplication rules
  • Thread-safe implementation using sync.Mutex
  • Minimal changes - only affects the logging package, no other code modified

Fixes #379.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@Copilot Copilot AI changed the title [WIP] Minimize logs Implement log deduplication to minimize repetitive messages Aug 1, 2025
@Copilot Copilot AI requested a review from Plaenkler August 1, 2025 06:34
Copilot finished work on behalf of Plaenkler August 1, 2025 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Minimize logs
2 participants