Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
f3d38be
Add llms.txt middleware for generating markdown documentation summaries
cursoragent Jun 10, 2025
cf617cc
Implement LLMs.txt feature with API route for markdown content extrac…
cursoragent Jun 10, 2025
81d3771
Refactor llms.txt feature with dynamic path routing and improved cont…
cursoragent Jun 10, 2025
0b58824
Enhance LLMs.txt feature with advanced JSX processing and content ext…
cursoragent Jun 10, 2025
68d66bb
Add platform-specific code snippets to LLMs.txt feature
cursoragent Jun 10, 2025
9c1c749
Running autofix
codyde Jun 11, 2025
755dc72
Bump to next 15.2.3
codyde Jun 11, 2025
3a38ff3
Correcting linting issues
codyde Jun 11, 2025
b279e2f
Removing log statement from middleware
codyde Jun 18, 2025
233937f
Correcting linting errors
codyde Jun 18, 2025
27df3f8
Refactor match handling in resolvePlatformIncludes function
codyde Jun 18, 2025
3181a0d
[getsentry/action-github-commit] Auto commit
getsantry[bot] Jun 18, 2025
77445d3
Correcing param ordering based on app router
codyde Jun 18, 2025
0d7d5c4
Update parameter naming in GET function for consistency
codyde Jun 18, 2025
8a5474e
Revert "Correcting linting issues"
codyde Jun 18, 2025
2661a20
Moving LLM generation functionality out of middleware and into nextjs…
codyde Jun 18, 2025
4eb6d3f
Implement Markdown Export Feature: Add API route for .md exports and …
codyde Jun 18, 2025
75e1d84
Enhance Markdown Export Feature: Implement static file generation at …
codyde Jun 18, 2025
e0aa2b0
let's go
BYK Jun 19, 2025
fadcf3d
esm stuff
BYK Jun 19, 2025
af75f4c
moar esm stuff
BYK Jun 19, 2025
f7c8e6e
all back to cjs
BYK Jun 19, 2025
bf42d67
revert stuff
BYK Jun 19, 2025
ce6ae41
revert tsconfig too
BYK Jun 19, 2025
97533a9
hack tsc
BYK Jun 19, 2025
5a4f3e5
cleaner md
BYK Jun 19, 2025
305267c
remove debug thing
BYK Jun 19, 2025
f832981
fix root detection
BYK Jun 19, 2025
a822450
even more clean up
BYK Jun 19, 2025
d564bfd
Merge branch 'master' into cursor/convert-page-to-markdown-format-39de
BYK Jun 19, 2025
693c4f4
remove LLM instructions file
BYK Jun 19, 2025
4536db8
parallelize
BYK Jun 19, 2025
88a59b5
fix typo
BYK Jun 19, 2025
435606b
bump min workers
BYK Jun 19, 2025
ad400e2
add source for vercel build cpus
BYK Jun 19, 2025
33121cd
back to 2 max workers
BYK Jun 19, 2025
21830c9
Update middleware.ts
codyde Jun 20, 2025
31c3ab9
add Markdown links
BYK Jun 20, 2025
19ee67f
nofollow on md
BYK Jun 20, 2025
6d687ef
revert useless NODE_ENV check
BYK Jun 20, 2025
29eda98
revert middleware.ts changes
BYK Jun 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions LLMS_TXT_FEATURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Markdown Export Feature Documentation

This feature allows converting any page on the Sentry documentation site to a plain markdown format by simply appending `.md` to the end of any URL (without a trailing slash). The feature extracts the actual page content from the source MDX files and converts it to clean markdown, making the documentation more accessible to Large Language Models and other tools.

## Example URLs

- Markdown Export: https://docs.sentry.io/platforms/javascript/guides/react/user-feedback.md
- Markdown Export: https://docs.sentry.io/platforms/javascript/guides/nextjs/user-feedback.md
- Markdown Export: https://docs.sentry.io/platforms/javascript/guides/react/tracing.md

## How it works

- URL: /platforms/javascript/guides/react/user-feedback.md
- Rewrite: /api/md-export/platforms/javascript/guides/react/user-feedback

## Example usage

```
curl "http://localhost:3000/platforms/javascript/guides/react/user-feedback.md"
curl "http://localhost:3000/platforms/javascript/guides/nextjs/user-feedback.md"
curl "http://localhost:3000/platforms/javascript/guides/react/tracing.md"
curl "http://localhost:3000/platforms/javascript/guides/vue/user-feedback.md"
```

## ✅ **Feature Status: FULLY WORKING**

The feature successfully extracts full page content from source MDX files, resolves platform-specific code snippets, and converts JSX components to clean markdown format.

## 🚀 **Major Update: Code Snippets Included!**

The feature now properly resolves `<PlatformContent includePath="..." />` components by loading the actual platform-specific code snippets from the `platform-includes/` directory.

### Code Snippet Resolution Features
- ✅ **Platform Detection**: Automatically detects platform and guide from URL path
- ✅ **Dynamic Includes**: Loads content from `platform-includes/{section}/{platform}.{guide}.mdx`
- ✅ **Fallback Handling**: Falls back to platform-level or generic includes if specific ones don't exist
- ✅ **Code Block Preservation**: Existing markdown code blocks are preserved during JSX cleanup
- ✅ **Multiple Platforms**: Works correctly across different JavaScript frameworks (React, Next.js, Vue, etc.)

## Usage Examples

### React User Feedback with Code Snippets
```
Original: https://docs.sentry.io/platforms/javascript/guides/react/user-feedback/
LLMs.txt: https://docs.sentry.io/platforms/javascript/guides/react/user-feedback/llms.txt
```

**Now Includes**:
- **Prerequisites**: Full SDK requirements and browser compatibility
- **Installation**: Actual npm/yarn/pnpm commands for React
- **Setup**: Complete JavaScript configuration code
- **API Examples**: Actual code snippets for user feedback implementation

### Next.js User Feedback (Platform-Specific)
```
Original: https://docs.sentry.io/platforms/javascript/guides/nextjs/user-feedback/
LLMs.txt: https://docs.sentry.io/platforms/javascript/guides/nextjs/user-feedback/llms.txt
```

**Shows Next.js-Specific Content**:
```bash
npx @sentry/wizard@latest -i nextjs
```
Instead of generic npm install commands.

### React Tracing with Enhanced Content
```
Original: https://docs.sentry.io/platforms/javascript/guides/react/tracing/
LLMs.txt: https://docs.sentry.io/platforms/javascript/guides/react/tracing/llms.txt
```

**Now Includes**:
- **Enable Tracing**: Platform-specific activation instructions
- **Configure**: Detailed sampling rate configuration
- **Code Examples**: Actual JavaScript implementation code

## Content Resolution Architecture

```
URL: /platforms/javascript/guides/react/user-feedback/llms.txt
↓ (Middleware intercepts)
Rewrite: /api/llms-txt/platforms/javascript/guides/react/user-feedback
↓ (API route processes)
1. Parse path: platform='javascript', guide='react'
2. Load: docs/platforms/javascript/common/user-feedback/index.mdx
3. Detect: <PlatformContent includePath="user-feedback/install" />
4. Resolve: platform-includes/user-feedback/install/javascript.react.mdx
5. Replace: Include actual React installation code snippets
6. Output: Full documentation with real code examples
```

## Platform Include Resolution

### Detection Logic
```typescript
// From URL: /platforms/javascript/guides/react/user-feedback/
platform = 'javascript' // pathSegments[1]
guide = 'react' // pathSegments[3]
platformId = 'javascript.react' // Combined identifier
```

### File Resolution Priority
```typescript
// For <PlatformContent includePath="user-feedback/install" />
1. platform-includes/user-feedback/install/javascript.react.mdx ✓ Most specific
2. platform-includes/user-feedback/install/javascript.mdx ↓ Platform fallback
3. platform-includes/user-feedback/install/index.mdx ↓ Generic fallback
```

### Real Example Output

**Before (Missing Code)**:
```markdown
### Installation
*[Installation instructions would appear here for javascript.react]*
```

**After (With Real Code)**:
```markdown
### Installation
The User Feedback integration is **already included** with the React SDK package.

```bash {tabTitle:npm}
npm install @sentry/react --save
```

```bash {tabTitle:yarn}
yarn add @sentry/react
```

```bash {tabTitle:pnpm}
pnpm add @sentry/react
```
Loading
Loading