fuse score-configs caching

[jgwill]

Complete Score-Config Application System with Smart Auto-Refresh

Phase 1: Enhanced Score Application Commands (Core Enhancement)

1.1 Extend Current Score Commands

• Extend scores apply to support both traces and sessions:
  coaia fuse scores apply --trace-id --score-id --value
  coaia fuse scores apply --session-id --score-id --value

1.2 Add Score-Config Application Commands

• New score-configs apply subcommand with different syntax:
  coaia fuse score-configs apply "Helpfulness" --trace-id --value 4
  coaia fuse score-configs apply "Overall Quality" --session-id --value 8.5
  coaia fuse score-configs apply --config-id --trace-id --value

1.3 Value Validation System

• Validate values against config constraints:
• CATEGORICAL: Value must match one of the defined category values
• NUMERIC: Value must be within minValue/maxValue range
• BOOLEAN: Value must be true/false (or 1/0)
• Helpful error messages with valid options/ranges

Phase 2: Project-Aware Smart Caching System (Performance Enhancement)

2.1 Project-Aware Cache Implementation

• Cache directory: ~/.coaia/score-configs/
• Per-project cache files: {project_id}.json (e.g., cm6m5rrk6001vvkbq7zovttji.json)
• Auto-detect project: Use coaia fuse projects to get current project ID
• Cache isolation: Each Langfuse project has separate cache

2.2 Smart Auto-Refresh Logic (NO Manual Sync Required)

• Cache miss triggers: Config name/ID not found → fetch from API automatically
• Age-based refresh: Cache >24h old → refresh in background transparently
• Error-based refresh: API "not found" errors → purge stale cache entries
• Command-based checks: Listing commands check staleness first
• Optional manual refresh: coaia fuse score-configs refresh (not sync)

2.3 Project-Aware Cache Structure

{
"project_id": "cm6m5rrk6001vvkbq7zovttji",
"project_name": "depotoire2502",
"last_sync": "2025-01-20T10:30:00Z",
"configs": [
{
"id": "config-123",
"name": "Helpfulness",
"dataType": "CATEGORICAL",
"categories": [{"label": "Poor", "value": 1}, ...],
"cached_at": "2025-01-20T10:30:00Z"
}
]
}

Phase 3: User Experience Enhancements

3.1 Interactive Config Selection

• List available configs with filtering:
  coaia fuse score-configs available --category narrative
  coaia fuse score-configs show "Helpfulness" --requirements

3.2 Enhanced Validation Feedback

• Category validation: "Value must be one of: 1(Poor), 2(Fair), 3(Good), 4(Excellent), 5(Outstanding)"
• Range validation: "Value must be between 0.0 and 10.0"
• Type validation: "Expected boolean value (true/false)"

3.3 Smart Value Input

• Category shortcuts: Accept both numeric values and label names
• Range guidance: Show valid range in error messages
• Example usage: Show example commands with valid values

Implementation Architecture

New Functions (cofuse.py)

• get_current_project_info() - Get project ID/name from Langfuse API
• get_project_cache_path() - Generate project-specific cache path
• get_config_with_auto_refresh() - Smart cache-first retrieval with auto-refresh
• validate_score_value() - Value validation against config constraints
• apply_score_config() - Apply score using config with validation
• list_available_configs() - List configs with smart caching
• cache_score_config() - Store config in project-specific cache
• is_cache_stale() - Check if cache needs refresh

New CLI Commands (coaiacli.py)

• score-configs apply - Main application command with validation
• score-configs available - List configs with filtering and auto-refresh
• score-configs refresh - Optional manual cache refresh
• score-configs show --requirements - Display config constraints
• Enhanced scores apply - Add session support

Auto-Refresh Implementation:

def get_config_with_auto_refresh(config_name_or_id):
project_info = get_current_project_info()
cache_path = get_project_cache_path(project_info['id'])

cached = get_from_cache(cache_path, config_name_or_id)

if not cached or is_cache_stale(cached):
    # Transparent auto-refresh
    fresh_config = fetch_from_langfuse(config_name_or_id)
    cache_score_config(cache_path, fresh_config)
    return fresh_config

return cached

Command Examples with Auto-Refresh

Just works - auto-refreshes as needed, no manual sync

coaia fuse score-configs apply "Helpfulness" --trace-id abc123 --value 4

List configs - auto-refreshes stale cache transparently

coaia fuse score-configs available --category ai

Show requirements - uses cache, refreshes if needed

coaia fuse score-configs show "Helpfulness" --requirements

Apply to session with validation

coaia fuse score-configs apply "Overall Quality" --session-id def456 --value 8.5

Optional manual refresh (rare usage)

coaia fuse score-configs refresh

Enhanced session scoring (existing functionality extended)

coaia fuse scores apply --session-id def456 --name "Custom Metric" --value 7.2

Benefits

For Users:

• Zero maintenance: No manual sync commands required, ever
• Project isolation: Different projects don't interfere with each other
• Fast performance: Cache-first approach with transparent refresh
• Always fresh: Auto-refresh ensures latest configs without user intervention
• Intuitive workflow: Apply configs by name with instant validation

For System:

• Smart caching: Only refreshes when needed, reduces API calls
• Project awareness: Handles multiple Langfuse projects correctly
• Resilient: Handles configs created elsewhere automatically
• Extensible: Framework supports future enhancements