feat: Confidence & Uncertainty Quantification

[lsm]

Purpose

Enable NeoKai to express and track confidence in its outputs, distinguishing between "probably right" and "definitely right" actions. This is essential for AGI-level autonomy because:

• Safe decision making: Knowing when to proceed vs when to ask for help
• Appropriate escalation: Escalating uncertain decisions to users
• Trust building: Being honest about limitations
• Risk-aware execution: Adjusting behavior based on confidence levels

Without confidence quantification, NeoKai cannot make informed decisions about when to act autonomously vs when to seek human input.

---

Current State

NeoKai has:

• Binary verification: PASS/FAIL from Verifier agent
• No confidence scores on outputs
• No uncertainty detection
• No escalation thresholds based on confidence

All outputs are treated equally regardless of how certain NeoKai is about them.

---

Proposed Approach

Phase 1: Confidence Score System

1. Confidence Data Model

   interface ConfidenceScore {
     overall: number;  // 0-100
     breakdown: {
       understanding: number;    // How well understood the request
       approach: number;         // How confident in the approach
       execution: number;        // How confident in the execution
       verification: number;     // How confident it meets requirements
     };
     factors: ConfidenceFactor[];
     uncertainties: Uncertainty[];
   }
   
   interface ConfidenceFactor {
     factor: string;
     impact: 'positive' | 'negative';
     magnitude: number;  // How much this affected confidence
   }
   
   interface Uncertainty {
     description: string;
     severity: 'low' | 'medium' | 'high';
     possibleResolutions: string[];
   }

2. Confidence Sources

   // Factors that increase confidence:
   - Similar task completed successfully before
   - Clear, unambiguous requirements
   - Strong test coverage in affected areas
   - User confirmed understanding
   
   // Factors that decrease confidence:
   - Novel task type
   - Ambiguous or incomplete requirements
   - Low test coverage in affected areas
   - Previous failures on similar tasks
   - Complex dependency chains

Phase 2: Uncertainty Detection

1. Uncertainty Types

   type UncertaintyType = 
     | 'requirement_ambiguity'   // Unclear what user wants
     | 'approach_selection'      // Multiple valid approaches
     | 'knowledge_gap'           // Missing necessary information
     | 'execution_risk'          // Risk of things going wrong
     | 'verification_uncertainty' // Hard to verify correctness
     | 'context_incompleteness'; // Missing context

2. Detection Mechanisms

   interface UncertaintyDetector {
     // LLM self-assessment
     selfAssess(output: AgentOutput): Promise<UncertaintyAssessment>;
     
     // Pattern-based detection
     detectKnownPatterns(output: AgentOutput): Uncertainty[];
     
     // Verification difficulty estimation
     estimateVerificationDifficulty(output: AgentOutput): number;
   }

3. LLM Confidence Elicitation

   const confidencePrompt = `
   After completing the task, assess your confidence:
   
   1. Understanding (0-100): How well did you understand what was requested?
   2. Approach (0-100): How confident are you this approach is correct?
   3. Execution (0-100): How confident are you in the implementation?
   4. Verification (0-100): How confident are you this meets all requirements?
   
   Also list:
   - Any uncertainties or concerns
   - Assumptions you made
   - Areas that might need review
   `;

Phase 3: Escalation System

1. Escalation Thresholds

   interface EscalationConfig {
     thresholds: {
       proceed: number;        // Above this: proceed autonomously
       escalate: number;       // Below this: always ask user
       // Between: case-by-case based on risk
     };
     riskMultipliers: {
       production: 1.5;        // Require higher confidence for production
       security: 2.0;          // Much higher for security-related
       destructive: 2.0;       // Higher for destructive operations
     };
   }

2. Escalation Actions

   type EscalationAction = 
     | 'proceed'              // Autonomous execution
     | 'proceed_with_notice'  // Execute but inform user
     | 'ask_confirmation'     // Get user confirmation first
     | 'present_options'      // Let user choose approach
     | 'request_clarification'; // Ask for more information
   
   function determineAction(
     confidence: ConfidenceScore,
     risk: RiskAssessment,
     config: EscalationConfig
   ): EscalationAction;

3. User Communication

   interface ConfidenceReport {
     overallConfidence: number;
     confidenceLevel: 'high' | 'medium' | 'low';
     mainUncertainties: string[];
     assumptions: string[];
     recommendation: string;
     requiresUserAction: boolean;
   }

Phase 4: Verification Confidence

1. Verifier Enhancement

   interface EnhancedVerificationResult {
     pass: boolean;
     confidence: number;  // How confident in the pass/fail
     checkedCriteria: VerificationCriterion[];
     unverifiedCriteria: string[];  // Things we couldn't verify
     edgeCases: string[];  // Edge cases that might cause issues
   }

2. Multi-Level Verification

   const verificationLevels = {
     quick: {
       confidence: 0.7,
       description: "Fast verification, may miss edge cases"
     },
     standard: {
       confidence: 0.85,
       description: "Normal verification depth"
     },
     thorough: {
       confidence: 0.95,
       description: "Exhaustive verification, catches edge cases"
     }
   };

Phase 5: Learning from Confidence

1. Confidence Calibration

   interface ConfidenceCalibrator {
     // Track confidence vs actual outcomes
     record(confidence: number, outcome: 'success' | 'failure'): void;
     
     // Get calibration adjustments
     getCalibration(): Map<number, number>;  // stated → actual
   
     // Check if well-calibrated
     assessCalibration(): CalibrationAssessment;
   }

2. Improving Over Time

   • Learn which factors predict success
   • Adjust confidence computation based on calibration
   • Identify systematically over/under-confident areas

---

Technical Considerations

Calibration Accuracy

• Ensuring confidence scores are well-calibrated
• Avoiding systematic over/under-confidence
• Handling rare events and edge cases

User Experience

• How to present confidence without being annoying
• Balancing transparency with simplicity
• Avoiding false confidence (sounding sure when wrong)

Performance Impact

• Cost of confidence elicitation
• Additional LLM calls for uncertainty detection
• Caching strategies for confidence factors

Cultural/Communication

• Different users may prefer different certainty levels
• Some domains require higher confidence than others
• Balancing speed vs certainty

---

Success Metrics

1. Calibration Error: Difference between stated confidence and actual success rate
2. Escalation Appropriateness: % of escalations that were warranted
3. User Trust: Correlation between confidence accuracy and user trust
4. Autonomy Efficiency: Ratio of successful autonomous decisions to escalations

---

Implementation Roadmap

1. Phase 1: Basic confidence scoring in outputs
2. Phase 2: Uncertainty detection and reporting
3. Phase 3: Escalation system
4. Phase 4: Verification confidence
5. Phase 5: Calibration and learning

---

Questions for Discussion

1. What's the right default confidence threshold for autonomous action?
2. Should confidence be visible to users by default?
3. How to handle cases where NeoKai is confidently wrong?
4. Should different specialists have different confidence models?

---

Part of the AGI-Level Autonomy initiative