test(docs): add comprehensive tests and monitoring for llms.txt endpoints

[devin-ai-integration]

Add production monitoring for llms.txt endpoints with incident.io integration

This PR adds synthetic monitoring for llms.txt, llms-full.txt, and .md/.mdx endpoints used by LLM tools for documentation discovery, plus fixes a critical bug in the markdown route that was preventing proper slug parameter handling from middleware.

What was the motivation & context behind this PR?

After fixing the routing bug in PR #4675, we needed production monitoring to detect when these critical endpoints go down. During development, the monitoring script was failing with 404 errors on .mdx endpoints that worked fine in browsers, which led to discovering both a URL construction bug in the monitoring script and a critical bug in the markdown route itself.

Additionally, David raised concerns about duplicate Slack notifications (incident.io already posts to Slack) and Vercel AI identified a missing response validation in the monitoring script.

Changes Made

1. Markdown Route Bug Fix (CRITICAL)

File: packages/fern-docs/bundle/src/app/[host]/[domain]/api/fern-docs/markdown/route.ts

The route now checks for the slug search parameter (passed by middleware) before falling back to pathname extraction:

const slugParam = req.nextUrl.searchParams.get("slug");
const slug = slugParam ?? path.replace(MARKDOWN_PATTERN, "");

This is backward-compatible but critical for middleware integration with llms.txt endpoints.

2. Synthetic Monitoring Script

File: scripts/monitor/check-llms-md-endpoints.ts (504 lines)

Comprehensive monitoring script with:

• Monitors 5 sites: Fern Docs, ElevenLabs, OpenRouter, Ada, Letta
• Checks: llms.txt, llms-full.txt, and first .md/.mdx link from llms.txt
• Slack alerts (opt-in only):
  • Requires ENABLE_SLACK_ALERTS=1 to avoid duplicate notifications with incident.io
  • Suppressed in DRY_RUN and TEST_MODE to prevent noise
  • Uses SLACK_WEBHOOK_URL_DOCS_INCIDENTS env var
• incident.io integration:
  • Creates incidents when endpoints are down
  • Duplicate prevention: Queries for existing open incidents before creating new ones
  • Updates existing incidents instead of creating duplicates
  • Auto-resolves incidents when endpoints recover using CANCELED status (Monitoring→Closed transitions result in 422 errors)
  • Response validation fix: Added response.ok check before parsing second llms.txt fetch (addresses Vercel AI comment)
• TEST_MODE support: Creates real incidents but labels them with [TEST] prefix and mode: "test" field, with separate idempotency keys
• DRY_RUN mode: Skips all mutations for safe CI testing
• URL construction fix: Uses new URL(pathOrUrl, base) to prevent path duplication bug

3. GitHub Actions Workflow

File: .github/workflows/monitor-llms-md-endpoints.yml

• Runs every 5 minutes via cron
• Can be manually triggered via workflow_dispatch
• Runs on push to PR branch for testing (with DRY_RUN mode and continues-on-error)
• Uses secrets: SLACK_WEBHOOK_URL_DOCS_INCIDENTS and INCIDENT_API_KEY

4. Unit Tests

File: packages/fern-docs/bundle/src/app/[host]/[domain]/api/fern-docs/markdown/route.test.ts

5 test cases covering slug parameter handling in the markdown route.

How has this PR been tested?

• ✅ All unit tests pass
• ✅ Lint and format checks pass
• ✅ Monitoring script tested in CI with DRY_RUN mode - all endpoints return 200
• ✅ URL construction logic verified with test cases
• ✅ incident.io integration tested with real credentials in TEST_MODE:
  • Successfully created test incident with [TEST] prefix and mode: "test" field
  • Verified duplicate prevention by querying existing incidents
  • Verified response.ok validation prevents processing error responses
• ⚠️ Full production flow not tested end-to-end (creating real incidents, updating, resolving in production environment)
• ⚠️ Slack integration not tested end-to-end (no access to webhook URL locally, also now opt-in only)

Human Review Checklist

Critical items requiring verification:

1. ⚠️ Hardcoded incident status/severity IDs (lines 52-55 in monitoring script):

   • INCIDENT_STATUS_MONITORING = "01HR85VFNXWH1H6976YCEJ5XJB"
   • INCIDENT_STATUS_CANCELED = "01HR85VFNXMV8SBQ3FRPMDBCST" (used for resolution instead of CLOSED)
   • INCIDENT_STATUS_CLOSED = "01HR85VFNXJPF6TXWYTXA6NBS2" (defined but not used)
   • INCIDENT_SEVERITY_MINOR = "01HR85VFNX9NYZG6B5Z40K8Y9V"
   • These were copied from check_sites.py - please verify they're correct for this workspace

2. ⚠️ Slack alerting behavior change: Slack alerts are now opt-in only (require ENABLE_SLACK_ALERTS=1) to avoid duplicate notifications with incident.io. This means:

   • By default, NO Slack alerts will be sent
   • incident.io is expected to handle Slack notifications when incidents are created/updated
   • If incident.io Slack integration isn't configured, we'll lose Slack visibility unless ENABLE_SLACK_ALERTS=1 is set
   • Please confirm this behavior is acceptable and that incident.io Slack integration is configured

3. ⚠️ Incident resolution status: Script uses CANCELED instead of CLOSED because testing showed Monitoring→Closed resulted in 422 errors. Please confirm this is the correct terminal status for these incidents.

4. Monitored sites list (lines 41-46): Currently monitors 5 sites. Please confirm this list is complete.

5. URL construction logic (lines 88-89): Uses new URL(pathOrUrl, base) to handle both absolute and relative URLs. Verify this correctly handles all edge cases.

6. ⚠️ Workflow PR branch trigger: The workflow includes push.branches for testing. This MUST be removed before merging (as noted in PR comment fix: clamp prose width to 600px and improve list rendering #11).

7. Secret configuration: Verify SLACK_WEBHOOK_URL_DOCS_INCIDENTS and INCIDENT_API_KEY are configured in GitHub repo settings.

8. TEST_MODE vs DRY_RUN: Two different modes with different purposes:

   • DRY_RUN=1: Skips all mutations (no incidents created/updated/resolved)
   • TEST_MODE=1: Creates real incidents but labels them as tests with [TEST] prefix and mode: "test"
   • Both can be used together or separately

---

Link to Devin run: https://app.devin.ai/sessions/3d070b89262340449cde1aa04e516963
Requested by: David Konigsberg ([email protected]) (@davidkonigsberg)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Updated (UTC)
dev.ferndocs.com	Ready	Preview	Nov 5, 2025 3:20pm
fern-dashboard	Ready	Preview	Nov 5, 2025 3:20pm
fern-dashboard-dev	Ready	Preview	Nov 5, 2025 3:20pm
ferndocs.com	Ready	Preview	Nov 5, 2025 3:20pm
preview.ferndocs.com	Ready	Preview	Nov 5, 2025 3:20pm
prod-assets.ferndocs.com	Ready	Preview	Nov 5, 2025 3:20pm
prod.ferndocs.com	Ready	Preview	Nov 5, 2025 3:20pm

1 Skipped Deployment

Project	Deployment	Preview	Updated (UTC)
fern-platform	Ignored		Nov 5, 2025 3:20pm

[devin-ai-integration]

Fixed in commit 0bda778. Changed the environment variable name from SLACK_WEBHOOK_URL to SLACK_WEBHOOK_URL_DOCS_INCIDENTS to match what the monitoring script expects.

[devin-ai-integration]

Fixed in commit 4466868. Added AbortController with 10-second timeout to all fetch calls in the monitoring script:

• llms.txt fetch in checkSite (the one you flagged)
• Slack webhook fetch in sendSlackAlert
• All incident.io API fetch calls (findOpenIncident, updateIncident, createIncident, resolveIncident)

This ensures the monitoring script never hangs indefinitely if external services don't respond.