feat(xiaohongshu): add proxy configuration support #20952
Conversation
Allow xiaohongshu routes to bypass anti-scraping measures by routing requests through a configurable proxy. When XIAOHONGSHU_PROXY is set, all xiaohongshu HTTP requests (user, board, notes) will be routed through the proxy endpoint instead of using Puppeteer. Co-Authored-By: Claude Haiku 4.5 <[email protected]>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cc81503ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async function fetchWithProxy(url: string, cookie?: string): Promise<string> { | ||
| const proxy = config.xiaohongshu.proxy; | ||
| if (proxy) { | ||
| const proxyUrl = `${proxy}?url=${encodeURIComponent(url)}`; | ||
| logger.http(`Requesting ${url} via proxy`); | ||
| return await ofetch(proxyUrl, { parseResponse: (txt) => txt }); |
There was a problem hiding this comment.
When XIAOHONGSHU_PROXY is set, fetchWithProxy ignores the cookie parameter and sends no headers to the proxy. That means any route that relies on XIAOHONGSHU_COOKIE (e.g., getUserWithCookie and getFullNote) will silently lose authentication when a proxy is configured, often returning a login page or incomplete data that then fails JSON.parse/data extraction. If the proxy is expected to access logged-in content, it needs the cookie forwarded (e.g., as a header or query parameter).
Useful? React with 👍 / 👎.
Allow xiaohongshu routes to bypass anti-scraping measures by routing requests through a configurable proxy. When XIAOHONGSHU_PROXY is set, all xiaohongshu HTTP requests (user, board, notes) will be routed through the proxy endpoint instead of using Puppeteer. Co-authored-by: Claude Haiku 4.5 <[email protected]>
Allow xiaohongshu routes to bypass anti-scraping measures by routing requests through a configurable proxy. When XIAOHONGSHU_PROXY is set, all xiaohongshu HTTP requests (user, board, notes) will be routed through the proxy endpoint instead of using Puppeteer. Co-authored-by: Claude Haiku 4.5 <[email protected]>
Summary
Allow xiaohongshu routes to bypass anti-scraping by routing requests through a configurable proxy. When
XIAOHONGSHU_PROXYenvironment variable is set, all xiaohongshu HTTP requests will use the proxy endpoint instead of Puppeteer.Implementation
proxyconfiguration option to xiaohongshu configfetchWithProxyhelper that uses proxy when availablegetUser,getBoard,getFullNote, andgetUserWithCookieto use proxyextractInitialSsrStatehelper for board page parsingRoutes
Co-Authored-By: Claude Haiku 4.5 [email protected]