feat: autorag search

[TheOutdoorProgrammer]

Summary

Adds hybrid AI-powered search to Spacelift documentation, combining traditional keyword search with AutoRAG-based intelligent question answering.

Motivation

Users often have specific questions when searching documentation. Traditional keyword search requires knowing the exact terms to search for, which can be frustrating when exploring unfamiliar concepts. This hybrid approach allows users to ask natural language questions (ending with "?") to get AI-generated answers alongside relevant documentation pages, while maintaining fast keyword search for traditional queries.

Changes

Architecture

• Hybrid routing: Queries ending with "?" trigger AI search; all others use native MkDocs/Lunr.js search
• Dual API integration:
  • Search API for retrieving relevant documentation chunks
  • Answer API for generating natural language responses
• Progressive enhancement: AI features layer on top of existing search without breaking fallback behavior

Frontend (overrides/partials/search.html)

• Custom search modal replacing default MkDocs search component
• Real-time query classification (keyword vs. question)
• Asynchronous AI answer generation with streaming-like UX
• Request lifecycle management (abort controllers, timeout handling, stale request detection)
• Security-first sanitization of all API responses

Styling (docs/assets/css/custom.css)

• Visual differentiation for AI search mode (colored input border)
• Styled AI answer cards with gradient backgrounds
• Loading states with CSS animations
• Syntax highlighting for code snippets in AI responses
• Dark mode support for all new UI elements

Header Integration (overrides/partials/header.html)

• Search button always visible (removed plugin check)
• Ensures custom search modal is included regardless of configuration

Security Considerations

All API responses are sanitized to prevent XSS attacks:

• Script tags and event handlers removed
• Dangerous HTML elements blocked (iframe, object, form, etc.)
• URL validation prevents javascript:, data:, and vbscript: protocols
• HTML attribute escaping prevents injection attacks

Performance

• 300ms debounce on search input
• 30-second timeout for AI API calls with automatic fallback
• Request cancellation on new user input prevents resource waste
• Lazy loading of AI answers (only fetched after search results)

User Experience

• Clear visual feedback during AI processing
• Inline disclaimer about AI accuracy
• Graceful degradation when AI services unavailable
• Keyboard shortcut hint (? suffix) in placeholder text
• Search term highlighting in results

Fallback Behavior

• AI search timeout → automatic keyword search
• AI API error → automatic keyword search
• No AI results → automatic keyword search
• JavaScript disabled → native MkDocs search still works

Testing Notes

To test AI search:

1. Open search modal
2. Type a question ending with "?" (e.g., "What is a stack?")
3. Observe AI answer generation and highlighted results
4. Try a keyword search without "?" to verify native search still works
5. Test with various query lengths and edge cases