feat(rag): enhance url loader with Jina Reader for better HTML parsing

[yueliao11]

Summary

This PR significantly improves the capabilities of the RAG system's URL loader by integrating Jina Reader. It shifts from a simple regex-based HTML stripper to a smart extraction engine that converts web pages into structured Markdown, filtering out noise like sidebars and ads.

Key Changes

1. Hybrid Loading Strategy

The loader now intelligently distinguishes between target types:

• Direct Path

  • GitHub Raw URLs (raw.githubusercontent.com) and pure text files (.py, .md, .json, etc.) are still downloaded directly to ensure code accuracy and zero latency.

• Smart Path

  • General web pages are routed through Jina Reader (https://r.jina.ai/) to extract high-quality, LLM-friendly content.

2. Fallback Mechanism

If the Jina Reader service times out or fails, the system automatically falls back to the original method (direct download + regex stripping) to ensure no disruption in service.

3. Configuration

Added support for an optional JINA_API_KEY environment variable, though it works anonymously by default.

[chatgpt-codex-connector]

The account who enabled Codex for this repo no longer has access to Codex. Please contact the admins of this repo to enable Codex again.

[yueliao11]

I pushed a small follow-up commit to refine spoon_ai/rag/qa.py after local testing. No other files are included.