Skip to content

feat(fetch-url): 支持可配置 reader 回退与 Jina 配额提升 #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DCjanus merged 6 commits into from
Mar 9, 2026
Merged

feat(fetch-url): 支持可配置 reader 回退与 Jina 配额提升 #31

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
84c4d8a
feat(fetch-url): add configurable reader fallback
DCjanus Mar 9, 2026
8a0f98f
fix(fetch-url): honor strategy-based fallbacks
DCjanus Mar 9, 2026
5572ea4
fix(fetch-url): narrow jina rate-limit fallback
DCjanus Mar 9, 2026
357f775
docs(fetch-url): clarify jina rate-limit guidance
DCjanus Mar 9, 2026
4de94dc
docs(fetch-url): simplify skill usage guide
DCjanus Mar 9, 2026
26757de
fix(fetch-url): neutralize jina fallback logs
DCjanus Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions skills/fetch-url/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,46 @@ uv run playwright install chromium
- `--timeout-ms`：Playwright 导航超时（毫秒，默认 60000）。
- `--browser-path`：指定本地 Chromium 系浏览器路径（默认自动探测）。
- `--output-format`：输出格式（默认 `markdown`），支持 `csv`、`html`、`json`、`markdown`、`raw-html`、`txt`、`xml`、`xmltei`；`raw-html` 直接输出渲染后的 HTML（不经 trafilatura）。
- `--disable-twitter-api`：关闭 Twitter/X 的 FxTwitter API 优化路径。
- `--fetch-strategy`：仅 `markdown` 可用，支持 `auto`、`agent`、`jina`、`browser`。默认 `auto`。

BREAKING CHANGE
- 已删除 `--disable-twitter-api`。
- 影响范围：原先依赖 `--disable-twitter-api` 关闭 FxTwitter 的调用方式不再可用。
- 迁移方式：改为使用 `--fetch-strategy` 控制抓取路径。只有 `--fetch-strategy auto` 会对 Twitter/X 推文链接优先使用 FxTwitter；如果想跳过 FxTwitter，改用 `agent`、`jina` 或 `browser`。

Markdown 抓取顺序：
- Twitter/X 推文链接：只有 `--fetch-strategy auto` 时才会先走 FxTwitter API。
- 其它 Markdown 请求：`--fetch-strategy auto` 时先尝试原站 `Accept: text/markdown` 协商，再尝试 [Jina Reader](https://r.jina.ai/)，最后回退到本地 Playwright 渲染并提取。
- 如需更明确控制兜底方式，可手工指定：
- `--fetch-strategy agent`：只尝试原站 Markdown 协商。
- `--fetch-strategy jina`：只尝试 Jina Reader。
- `--fetch-strategy browser`：直接走本地 Playwright。

限流或挑战页：
- 当前只会对 Jina Reader 做非常保守的明显限流页识别；只有命中少数高置信度特征时，才会继续回退到后续流程。
- 为避免误伤正常正文，这个判定刻意做得很窄；部分限流页没有被识别出来而直接输出是可接受的。
- 如果明确知道某个 reader 的结果不可用，agent 可以直接切换到更兜底的 `--fetch-strategy browser`。

Jina Reader：
- 脚本会读取环境变量 `JINA_API_KEY`；如果存在，就以 `Authorization: Bearer <token>` 方式传给 Jina Reader。
- 不设置 `JINA_API_KEY` 也能用 Jina Reader，但官方公开配额较低；当前按更保守口径可认为无 Key 时大约 `20 RPM`。
- 如果遇到 Jina Reader 限流，可提示用户配置 `JINA_API_KEY` 以提升配额；当前官方 Reader 产品页给出的普通 API Key 配额是 `500 RPM`，Premium 是 `5000 RPM`。

Twitter/X 特化（仅 `markdown`）：
- 当 URL 命中 `x.com`/`twitter.com` 推文链接且未设置 `--disable-twitter-api`，脚本会优先调用 `https://api.fxtwitter.com/2/status/{id}`。
- 当 URL 命中 `x.com`/`twitter.com` 推文链接且 `--fetch-strategy auto` 时，脚本会优先调用 [FxTwitter API](https://api.fxtwitter.com/)。
- 当 FxTwitter 返回 `thread` 数据时，Markdown 会附加 `## Thread` 小节，按顺序列出 thread 内其它推文（自动去重主推文）。
- 输出的 Markdown 首行会包含注释，明确标记内容来自 FxTwitter API，而非直接访问页面。
- 若 FxTwitter API 请求失败，命令会直接报错（不降级到网页抓取）；如需跳过该逻辑，请显式传入 `--disable-twitter-api`。
- 输出的 Markdown 会在元数据列表里标记内容来自 FxTwitter API，而非直接访问页面。
- 若 FxTwitter API 请求失败，命令会直接报错（不降级到网页抓取）；如需跳过该逻辑，请显式传入 `--fetch-strategy agent`、`jina` 或 `browser`。

Comment on lines +32 to 40
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Document that auto may send the URL to Jina Reader before local fallback.

The default strategy now changes the network/privacy boundary, but the doc only says auto is the default. Users should be told that auto may forward the target URL to Jina Reader and only falls back when the response looks like a block/challenge page; otherwise the default behavior is easy to misread as purely local fetching.

✍️ Suggested wording 
 `--fetch-strategy` 常用值：
-- `auto`：默认选择。
+- `auto`：默认选择。会先尝试无需本地浏览器的抓取路径；其中可能会把目标 URL 发送给 Jina Reader。若返回内容命中明显限流 / 验证码特征，再继续 fallback 到更兜底的方式。
 - `agent`：优先用原站 Markdown 协商。
 - `jina`：优先用 Jina Reader。
 - `browser`：直接用本地 Playwright。
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
`--fetch-strategy` 常用值：
- `auto`：默认选择。
- `agent`：优先用原站 Markdown 协商。
- `jina`：优先用 Jina Reader。
- `browser`：直接用本地 Playwright。
环境变量：
- 可设置 `JINA_API_KEY` 提升 Jina Reader 限流：`JINA_API_KEY=your-token ./scripts/fetch_url.py ...`
`--fetch-strategy` 常用值：
- `auto`：默认选择。会先尝试无需本地浏览器的抓取路径；其中可能会把目标 URL 发送给 Jina Reader。若返回内容命中明显限流 / 验证码特征，再继续 fallback 到更兜底的方式。
- `agent`：优先用原站 Markdown 协商。
- `jina`：优先用 Jina Reader。
- `browser`：直接用本地 Playwright。
环境变量：
- 可设置 `JINA_API_KEY` 提升 Jina Reader 限流：`JINA_API_KEY=your-token ./scripts/fetch_url.py ...`

示例：

```bash
./scripts/fetch_url.py https://example.com --output ./page.md --timeout-ms 60000
./scripts/fetch_url.py https://example.com --fetch-strategy jina
JINA_API_KEY=your-token ./scripts/fetch_url.py https://example.com --fetch-strategy jina
./scripts/fetch_url.py https://example.com --fetch-strategy browser
./scripts/fetch_url.py https://x.com/jack/status/20 --output-format markdown
./scripts/fetch_url.py https://x.com/jack/status/20 --output-format markdown --disable-twitter-api
./scripts/fetch_url.py https://x.com/jack/status/20 --output-format markdown --fetch-strategy browser
```

Reference：[`scripts/fetch_url.py`](scripts/fetch_url.py)
109 changes: 102 additions & 7 deletions skills/fetch-url/scripts/fetch_url.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from __future__ import annotations

import json
import os
from pathlib import Path
import re
from typing import Any, Literal
Expand All @@ -33,6 +34,7 @@
CONSOLE = Console()

OutputFormat = Literal["csv", "html", "json", "markdown", "raw-html", "txt", "xml", "xmltei"]
FetchStrategy = Literal["auto", "agent", "jina", "browser"]
TWITTER_HOSTS = {
"x.com",
"www.x.com",
Expand All @@ -42,6 +44,16 @@
"mobile.twitter.com",
}
FXTWITTER_API_ROOT = "https://api.fxtwitter.com/2/status"
JINA_READER_API_ROOT = "https://r.jina.ai/"
JINA_API_KEY_ENV = "JINA_API_KEY"
JINA_BLOCK_PAGE_SIGNALS = (
("rate limit", "jina"),
("too many requests", "jina"),
("rate limit exceeded", "jina"),
("retry after", "jina"),
("request limit reached", "jina"),
("security verification", "jina"),
)
# FxTwitter source repository: https://github.com/allnodes/FxTwitter


Expand Down Expand Up @@ -209,6 +221,64 @@ def fetch_agent_markdown(url: str, timeout_ms: int, verbose: bool) -> str | None
return None


def fetch_jina_reader_markdown(url: str, timeout_ms: int, verbose: bool) -> str | None:
"""通过 Jina Reader 获取 Markdown, 命中则直接返回。"""

reader_url = f"{JINA_READER_API_ROOT}{url}"
api_key = os.getenv(JINA_API_KEY_ENV, "").strip()
headers = {
"Accept": "text/markdown, text/plain;q=0.9, */*;q=0.1",
"User-Agent": "fetch-url/1.0 (+https://github.com/DCjanus/prompts/tree/master/skills/fetch-url)",
}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"

if verbose:
auth_mode = f"with {JINA_API_KEY_ENV}" if api_key else "without API key"
CONSOLE.print(
f"[cyan]Trying Jina Reader[/cyan] {auth_mode}",
highlight=False,
)

request = Request( # noqa: S310 - 本地 CLI 可信输入, URL 由用户主动提供
reader_url,
headers=headers,
)
try:
with urlopen(request, timeout=max(timeout_ms / 1000.0, 1.0)) as response: # noqa: S310
charset = response.headers.get_content_charset() or "utf-8"
markdown = response.read().decode(charset, errors="replace")
if not markdown.strip():
return None
if is_obvious_jina_block_page(markdown):
if verbose:
CONSOLE.print(
"[yellow]Jina Reader returned a probable rate-limit page, continue fallback[/yellow]",
highlight=False,
)
return None
if verbose:
CONSOLE.print(
f"[green]Jina Reader hit[/green] {len(markdown)} chars",
highlight=False,
)
return markdown
except (URLError, OSError) as exc:
if verbose:
CONSOLE.print(
f"[yellow]Jina Reader failed, fallback to browser render[/yellow] ({exc})",
highlight=False,
)
return None


def is_obvious_jina_block_page(content: str) -> bool:
"""识别少数非常明显的 Jina 限流或挑战页。"""

snippet = content[:4000].lower()
return any(all(part in snippet for part in signal) for signal in JINA_BLOCK_PAGE_SIGNALS)


def extract_twitter_status_id(url: str) -> str | None:
"""从 x.com/twitter.com 推文链接提取 status id。"""

Expand Down Expand Up @@ -432,22 +502,23 @@ def fetch(
"markdown",
help="Output format: csv, html, json, markdown, raw-html, txt, xml, xmltei.",
),
disable_twitter_api: bool = typer.Option(
False,
"--disable-twitter-api",
help="Disable FxTwitter API optimization for x.com/twitter.com links in markdown mode.",
fetch_strategy: FetchStrategy = typer.Option(
"auto",
help="Fetch strategy for markdown: auto, agent, jina, browser.",
),
verbose: bool = typer.Option(False, "--verbose", help="Print progress and diagnostic logs."),
) -> None:
"""通过 Playwright 渲染并用 trafilatura 提取内容。"""
parsed = urlparse(url)
if parsed.scheme not in {"http", "https"}:
raise typer.BadParameter("Only http or https URLs are supported.")
if output_format != "markdown" and fetch_strategy != "auto":
raise typer.BadParameter("Custom fetch strategy is only supported with markdown output.")

resolved_browser_path = str(browser_path) if browser_path else detect_browser_path()
try:
content: str | None = None
if output_format == "markdown" and not disable_twitter_api:
if output_format == "markdown" and fetch_strategy == "auto":
twitter_status_id = extract_twitter_status_id(url)
if twitter_status_id:
payload = fetch_fxtwitter_status(
Expand All @@ -458,14 +529,38 @@ def fetch(
if payload is None:
raise ValueError(
"FxTwitter API request failed for this Twitter/X URL. "
"Use --disable-twitter-api to skip this path."
"Use --fetch-strategy agent, jina, or browser to skip this path."
)
content = render_fxtwitter_markdown(payload, source_url=url)
if verbose:
CONSOLE.print("[green]Using FxTwitter API markdown path[/green]", highlight=False)
if output_format == "markdown":
if content is None:
content = fetch_agent_markdown(url, timeout_ms=timeout_ms, verbose=verbose)
if fetch_strategy == "auto":
content = fetch_agent_markdown(url, timeout_ms=timeout_ms, verbose=verbose)
if content is None:
content = fetch_jina_reader_markdown(
url,
timeout_ms=timeout_ms,
verbose=verbose,
)
elif fetch_strategy == "agent":
content = fetch_agent_markdown(url, timeout_ms=timeout_ms, verbose=verbose)
if content is None:
raise ValueError(
"Markdown negotiation did not return usable content. "
"Try --fetch-strategy jina or --fetch-strategy browser."
)
elif fetch_strategy == "jina":
content = fetch_jina_reader_markdown(url, timeout_ms=timeout_ms, verbose=verbose)
if content is None:
raise ValueError(
"Jina Reader did not return usable content. "
f"If this is rate limiting, configure {JINA_API_KEY_ENV} or try "
"--fetch-strategy browser."
)
elif fetch_strategy == "browser" and verbose:
CONSOLE.print("[cyan]Skipping non-browser markdown readers[/cyan]", highlight=False)
if content is None:
html = render_html(
url,
Expand Down