Improve Autocomplete Suggestions by Experimenting with models/prompts

[hassoncs]

Summary

Experiment with multiple OpenRouter models for autocomplete to identify the best trade-off between quality, latency, and cost, then expose a dropdown UI allowing end users to select their preferred model.

Background

Currently using Gemini Flash 2.5 for inline autocomplete with good accuracy but significant latency issues (taking at least 2 seconds) under load. We need to determine if Gemini 2.5 is actually the best choice by comparing it against potentially cheaper/faster models using a consistent prompt template.

Goals

1. Define a standard prompt template that ensures "just the code" output with minimal token overhead
2. Integrate at least four OpenRouter endpoints (Llama 3.3 8B, Mistral 3B, Mistral 7B, Qwen 2.5 Coder 7B)
3. Instrument metrics for latency and quality measurement
4. Add UI dropdown in settings for model selection
5. Produce a comparative report on timing and quality

Acceptance Criteria

• Standardized prompt helper function with tests
• All OpenRouter models configured and accessible
• Metrics logging for latency and quality
• UI dropdown in settings panel
• Reporting script for data aggregation

Tasks

• Create prompt builder helper
• Add OpenRouter models to configuration
• Implement latency logging
• Build quality test harness with golden set
• Add UI dropdown with persistence
• Create reporting script
• Document findings

Technical Notes

• Quality measurement will use exact-match scoring initially
• Users should see timing stats in the UI
• Include fallback logic for rate limits