feat: AI client user-agent detection for automatic markdown serving #15060
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![semgrep-code-getsentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
Bundle ReportChanges will increase total bundle size by 9.19kB (0.04%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sentry-docs-client-array-pushAssets Changed:
view changes for bundle: sentry-docs-edge-server-array-pushAssets Changed:
view changes for bundle: sentry-docs-server-cjsAssets Changed:
|
codyde
force-pushed
the
feat/ai-client-markdown-redirect-clean
branch
from
ab697c5 to
d47c6bc
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- Detect AI/LLM tools (Claude, Cursor, Copilot, etc.) via user-agent - Automatically redirect to .md versions for better LLM consumption - Add manual override with ?format=md query parameter - Preserve existing redirect functionality 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Log all user agents for non-static requests - Log when redirects to markdown occur - Helps identify what user agents AI tools are sending
- Add /got/i pattern to detect sindresorhus/got library - Common HTTP client used by Node.js tools and automation - Ensures markdown serving for tools using got for requests
- Show 📄 MARKDOWN or 🌐 HTML indicators in logs - Determine content type before logging for better debugging - Makes it clear which requests get markdown vs HTML content
- Replace url.pathname.includes('.') with specific file extension regex
- Fixes AI client detection for .NET, Node.js, Next.js documentation
- Now only excludes actual static files (.js, .png, .css, etc.)
- Resolves Sentry bug report about .NET docs not getting markdown redirects
Fixes: Critical bug where paths like /platforms/dotnet/ were incorrectly
excluded from AI client detection due to containing dots in the path.
- Replace /\/$/ with /\/+$/ to remove ALL trailing slashes - Prevents malformed URLs like /path//.md when multiple slashes present - Defensive programming for edge cases bypassing Next.js normalization - Reported by Cursor - simple one-character regex fix Examples: - /platforms/react/// -> /platforms/react.md ✅ - /platforms/react/ -> /platforms/react.md ✅ - /platforms/react -> /platforms/react.md ✅
codyde
force-pushed
the
feat/ai-client-markdown-redirect-clean
branch
from
c52c198 to
1a242d3
Compare
rahulchhabria
approved these changes
Sep 29, 2025
- Process canonical URL redirects first before AI client detection - Ensures deprecated URLs redirect to current paths before markdown conversion - Prevents AI clients from getting outdated content via old URLs - Fixes Cursor bug report about bypassing canonical URL redirects Example flow: 1. /old-path/ → /new-path/ (canonical redirect) 2. AI client requests /new-path/ → /new-path.md (markdown) Instead of: 1. AI client requests /old-path/ → /old-path.md (404/outdated)
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- Implement hybrid strategy: Accept header + user-agent detection - Support standards-compliant text/markdown and text/x-markdown Accept headers - Maintain backward compatibility with existing user-agent detection - Enhanced logging shows detection method (Accept header, User-agent, Manual) - Inspired by OpenCode's content negotiation approach Examples: - Accept: text/markdown → markdown (standards-compliant) - User-Agent: Claude/1.0 → markdown (fallback for existing tools) - ?format=md → markdown (manual override) This makes docs accessible to more tools while following HTTP standards.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements automatic markdown serving for AI/LLM tools and development environments through HTTP Accept header content negotiation AND user-agent pattern matching. When clients request markdown content, they are automatically redirected to
.mdversions for better LLM consumption and processing.Detection Methods
🌐 HTTP Accept Header Content Negotiation (Standards-Compliant)
Follows HTTP standards for content type negotiation:
Accept: text/markdown- Standard markdown MIME typeAccept: text/x-markdown- Alternative markdown MIME typeExamples:
🤖 User-Agent Pattern Detection (Backward Compatibility)
Detects AI tools and development environments that benefit from markdown:
AI/LLM Tools
/claude/i) - Claude Desktop/Code/chatgpt/i) - ChatGPT clients/openai/i) - OpenAI tools and APIs/anthropic/i) - Anthropic toolsDevelopment Tools & IDEs
/cursor/i) - Cursor IDE/copilot/i) - GitHub Copilot/vscode/i) - VS Code extensions/intellij/i) - IntelliJ plugins/sublime/i) - Sublime Text pluginsHTTP Libraries
/got/i) - sindresorhus/got HTTP library for Node.js🎛️ Manual Override
?format=mdforces markdown for any clientHow It Works
.mdversions (302 redirect)Example Usage
Accept Header (Standards-Compliant)
User-Agent Detection (Legacy Tool Support)
Manual Override
Enhanced Logging
Shows detection method and content type:
Key Improvements
Standards Compliance
Robust Detection
.includes('.')/path///Enhanced Debugging
Benefits