Skip to content

SILENT_SIGNALS.md

Latest commit

 

History

History
242 lines (182 loc) · 7.36 KB

SILENT_SIGNALS.md

File metadata and controls

242 lines (182 loc) · 7.36 KB

Silent Signals: Implicit Feedback Mechanism

Overview

The Silent Signals feature eliminates the blind spot of relying solely on explicit feedback. Instead of waiting for users to file bug reports or provide feedback, the system captures implicit signals from user behavior to learn continuously.

The Problem with Explicit Feedback

Explicit feedback is a relic. Relying on it creates a massive blind spot where you think you are succeeding just because nobody is complaining. Most users don't provide feedback - they just leave when something doesn't work.

The Three Silent Signals

The system captures three types of implicit feedback:

1. 🚨 Undo Signal (Critical Failure)

What it captures: When a user immediately reverses an agent action (Ctrl+Z, revert, undo).

Why it matters: This is the loudest "Thumbs Down" possible. If a user takes the time to undo what the agent did, the response was fundamentally wrong or harmful.

Example:

# Agent provides dangerous code
doer.emit_undo_signal(
    query="Write code to delete temporary files",
    agent_response="os.system('rm -rf /*')",
    undo_action="Ctrl+Z in code editor",
    user_id="user123"
)

Observer Response:

  • Assigns score of 0.0 (critical failure)
  • Priority: CRITICAL
  • Immediately updates wisdom to prevent similar responses
  • Creates safety correction in prioritization framework

2. ⚠️ Abandonment Signal (Loss)

What it captures: When a user starts a workflow but stops responding halfway through without reaching a resolution.

Why it matters: This indicates the agent failed to engage effectively. The user didn't get value and gave up.

Example:

# User has 3 interactions but gives up
doer.emit_abandonment_signal(
    query="Help me debug this error",
    agent_response="Check your code for errors",
    interaction_count=3,
    last_interaction_time="2024-01-01T12:05:00",
    user_id="user456"
)

Observer Response:

  • Assigns score of 0.3 (low score)
  • Priority: HIGH
  • Learns that responses need to be more engaging/specific
  • Updates wisdom to provide better engagement

3. ✅ Acceptance Signal (Success)

What it captures: When a user takes the output and immediately moves to the next task without follow-up questions.

Why it matters: This is implicit success. The user got what they needed and moved on efficiently.

Example:

# User accepts and moves to next task
doer.emit_acceptance_signal(
    query="Calculate 15 * 24 + 100",
    agent_response="Result: 460",
    next_task="Calculate 20 * 30 + 50",
    time_to_next_task=2.5,
    user_id="user789"
)

Observer Response:

  • Assigns score of 1.0 (perfect)
  • Priority: POSITIVE
  • Recognizes successful response pattern
  • Can use for positive reinforcement learning

Architecture

Telemetry Layer

The TelemetryEvent dataclass has been extended to support signal tracking:

@dataclass
class TelemetryEvent:
    event_type: str  # "signal_undo", "signal_abandonment", "signal_acceptance"
    timestamp: str
    query: str
    agent_response: Optional[str] = None
    success: Optional[bool] = None
    signal_type: Optional[str] = None  # "undo", "abandonment", "acceptance"
    signal_context: Optional[Dict[str, Any]] = None  # Additional context
    # ... other fields

DoerAgent (Execution Layer)

The DoerAgent has three new methods for emitting signals:

  • emit_undo_signal(): Called when user reverses action
  • emit_abandonment_signal(): Called when user stops responding
  • emit_acceptance_signal(): Called when user accepts and moves on

These methods emit telemetry events to the event stream without blocking execution.

ObserverAgent (Learning Layer)

The ObserverAgent has been enhanced to:

  1. Detect signals: Filter and identify signal events from telemetry stream
  2. Analyze signals: Assign appropriate scores and priorities based on signal type
  3. Learn from signals: Update wisdom database and prioritization framework
  4. Track statistics: Monitor signal frequencies for system health metrics

New method:

def analyze_signal(self, event: TelemetryEvent, verbose: bool = False) -> Optional[Dict[str, Any]]

This method:

  • Identifies the signal type
  • Generates appropriate critique
  • Assigns priority level (critical, high, or positive)
  • Returns analysis with learning recommendations

Integration with UI

To integrate Silent Signals with your UI, you need to detect and emit signals at the appropriate times:

Detecting Undo Actions

# In your code editor component
def on_undo():
    if last_agent_action:
        doer.emit_undo_signal(
            query=last_query,
            agent_response=last_response,
            undo_action="User pressed Ctrl+Z",
            user_id=current_user_id
        )

Detecting Abandonment

# Session timeout or close detection
def on_session_end():
    if workflow_incomplete:
        doer.emit_abandonment_signal(
            query=initial_query,
            agent_response=last_response,
            interaction_count=len(conversation),
            user_id=current_user_id
        )

Detecting Acceptance

# When user starts a new task
def on_new_task(new_query):
    if previous_task_completed:
        doer.emit_acceptance_signal(
            query=previous_query,
            agent_response=previous_response,
            next_task=new_query,
            time_to_next_task=time_since_response,
            user_id=current_user_id
        )

Observer Processing

The Observer processes signals in batch during offline learning:

observer = ObserverAgent()
results = observer.process_events(verbose=True)

print(f"Undo Signals: {results['signal_stats']['undo_signals']}")
print(f"Abandonment Signals: {results['signal_stats']['abandonment_signals']}")
print(f"Acceptance Signals: {results['signal_stats']['acceptance_signals']}")

Benefits

  1. No User Friction: Captures feedback without requiring explicit user input
  2. Real Sentiment: Actions speak louder than words - learn from what users DO
  3. Immediate Detection: Critical failures (undo) are flagged instantly
  4. Pattern Recognition: Success patterns (acceptance) are reinforced automatically
  5. Engagement Metrics: Abandonment reveals where users lose interest
  6. Complete Picture: Combines with traditional metrics for holistic view

Testing

Run the test suite:

python test_silent_signals.py

Run the demonstration:

python example_silent_signals.py

Statistics and Monitoring

The Observer provides statistics on captured signals:

results = observer.process_events()

# Access signal statistics
signal_stats = results['signal_stats']
print(f"🚨 Critical Failures: {signal_stats['undo_signals']}")
print(f"⚠️ Lost Engagements: {signal_stats['abandonment_signals']}")
print(f"✅ Successes: {signal_stats['acceptance_signals']}")

Use these metrics to:

  • Monitor system health
  • Identify problematic interaction patterns
  • Track improvement over time
  • A/B test different approaches

Key Insight

Silent Signals eliminate the blind spot of explicit feedback.

The system learns from what users DO, not just what they SAY. This creates a continuous learning loop that captures true user sentiment without creating friction.

Most users won't complain - they'll just undo, abandon, or accept. Now your system learns from all of it.