feat(agents): make the markdown docs surface complete and reachable

[crazyuploader]

Pull Request

---

📄 Summary

Agents now reach the docs through llms.txt and .md URLs. The last audit run recorded 5 of 5 agents navigating that way, without opening a single HTML page. This PR fixes what that traffic pattern exposed.

Four docs pages returned 404 in production. Each page had a multi-line <Figure /> block directly after a list item, so the closing line parsed as a lazy continuation and the page failed to compile. The markdown route tolerated it, the HTML route did not. All four sit in the sidebar:

• azure-monitoring/app-service/logging.mdx
• azure-monitoring/az-container-apps/logging.mdx
• logs-pipelines/guides/json.mdx
• logs-pipelines/guides/trace.mdx

Every /docs/ redirect source returned 404 with .md appended. Next applies redirects before middleware runs, and the proxy rewrites /docs/<slug>.md straight to the markdown API, so the redirect table never ran. That covered 365 sources, including /docs/instrumentation/python.md. We created this failure mode by telling agents to append .md, which they now do. A new resolver follows the HTML redirect chain on a miss and returns 308 to the canonical markdown URL. Redirects that leave the docs tree, such as /docs/userguide/apis/ to /api-reference/, are preserved.

The region table was invisible without a browser. RegionContext initialized its region list to an empty array and RegionTable returned a skeleton while loading, so no region value appeared anywhere in the HTML response. The markdown export said the data was "available in the rendered docs", which was not true. The built-in region list now seeds the state and the export renders a real table.

llms.txt listed no language guide. getLlmStarterLinks takes one route per matcher, so the instrumentation prefix could only ever return the overview page. Python, Node.js, Go, and Java are now named explicitly.

The markdown-versus-HTML split was not measurable. custom_prefers_markdown reflected only the Accept header, so every .md request logged as HTML. The .md suffix is the path agents actually use.

Smaller docs fixes from the same run: sample apps moved off port 8080, which the self-hosted SigNoz UI uses; the exit-139 note now covers memory pressure on any host; the /docs/introduction page states the .md convention and links its three Get Started steps.

---

✅ Change Type

Select all that apply

• ✨ Feature
• 🐛 Bug fix
• ♻️ Refactor
• 🛠️ Infra / Tooling
• 🧪 Test-only

---

🐛 Bug Context

Required if this PR fixes a bug

Root Cause

What caused the issue?
Regression, faulty assumption, edge case, refactor, etc.

Fix Strategy

How does this PR address the root cause?

---

🧪 Testing Strategy

Tests added/updated: tests/agent-markdown-stubs.test.js asserts the region table renders real rows instead of a pointer. tests/agent-discovery.test.js raises the starter cap to 34 for the four language matchers. Full suite: 291 node tests, 83 vitest, yarn lint clean, yarn build green.

Manual verification, local dev server:

Check	Result
All 905 docs routes on production	4 return 404, and they are the 4 this PR fixes. No others.
Those 4 routes locally	200
/docs/instrumentation/python.md	308 to /docs/instrumentation/opentelemetry-python.md, then 200 text/markdown
/docs/userguide/apis.md (leaves the docs tree)	308 to /api-reference
/docs/overview/whats-new/changelog-0.76.md	308 to /changelog.md
/docs/nope-not-real.md	404, 0 redirects, markdown recovery body intact
Region values in the HTML of /docs/ingestion/signoz-cloud/overview/	0 before, present after
/llms.txt	4 language links and 4 access-control links
<link rel="alternate" type="text/markdown">	Present on the catch-all and on /docs/introduction/
POST /api/v5/query_range with the payload in traces-api.mdx	status: success, 1 row, against a live workspace

Edge cases covered: redirect chains with a trailing-slash hop; redirect cycles; redirect targets outside /docs; genuine 404s; markdown responses, which get no Link header of their own.

---

⚠️ Risk & Impact Assessment

What could break? How do we recover?

• Blast radius:
• Potential regressions:
• Rollback plan:

---

📝 Changelog

Fill only if this affects users, APIs, UI, or documented behavior
Use N/A for internal or non-user-facing changes

Field	Value
Deployment Type	Cloud / OSS / Enterprise
Change Type	Feature / Bug Fix / Maintenance
Description	User-facing summary

---

📋 Checklist

• Tests added or explicitly not required
• Manually tested
• Breaking changes documented
• Backward compatibility considered

---

👀 Notes for Reviewers

Four files conflict with #4067: proxy.ts, app/(site)/api/docs-markdown/[[...slug]]/route.ts, utils/docs/agentDiscovery.ts, and tests/agent-discovery.test.js. The content is disjoint, but the regions are the same. Land #4067 first, will rebase this PR then.

Two known gaps, both tracked and not in this PR: check-stale-urls skips .json under constants/, which hides 41 redirect-source hrefs; extending it surfaces 508 issues across 29 files. And no ingestion-key-only ingestion check exists, so the headless verification in traces-api.mdx needs a service-account key.

---

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
signoz-web	Ready	Preview	Sep 3, 2026 8:27am UTC

[github-actions]

Claude finished @crazyuploader's task in 3m 50s —— View job

---

PR Review: Agent docs discovery

Reviewed with signoz-docs-pr-review (docs) and signoz-website-frontend-pr-review (code). This is a well-scoped, high-quality PR: the four 404-on-HTML fixes (lazy-continuation after <Figure>), the markdown-redirect resolver, the region-table seeding, and the llms.txt language links all address real, verifiable failure modes. Two inline notes below, both Low.

---

Code findings

Low

• utils/docs/canonicalDocsMarkdownPath.ts:104 — external redirect targets lose their host (new URL(location, origin).pathname discards the domain). Not reachable today (no /docs/* redirect in next.config.js has an external destination — verified: all 7 external destinations are non-docs sources), so it's a latent guard rather than a live bug. [inline]
• app/(site)/docs/introduction/DocsCtaSection.tsx:58,71 — clickName values (CTA Step Link / CTA Alternative Link) describe the item rather than a stable reusable action; the rubric prefers one reusable clickName with the distinction in clickText. Matches this file's existing ad-hoc convention, so non-blocking. [inline]

Verified clean

• Go import removal (semconv "…/v1.21.0") has no remaining references in the example — clean, won't break compilation.
• RegionContext seeding FALLBACK_REGIONS + RegionTable gating on isLoading && regions.length === 0: deterministic initial state means no SSR/hydration mismatch, and region rows now appear in the HTML. Test coverage updated accordingly.
• regions.ts extraction is a sound server/client split; regionTableRows reused by both RegionTable and the agent stub (no duplication).
• proxy.ts markdown-suffix logging and hasMarkdownExtension import resolve correctly.
• robots.ts AI-crawler rule is functionally identical to the wildcard (intentional, documents policy) — no behavior change, fine.

Documentation findings

No P1/P2. Notes:

• Anchors verified: #validate (python), #confirm-a-service-is-sending-spans (traces-api), #endpoint all resolve.
• traces-api.mdx curl fence (~27 lines) correctly carries defaultCollapsed; under 30 lines so no minimap required — compliant.
• OTLP guidance uses HTTP :4318 in the self-hosted callouts (matches the OTLP/HTTP-first preference). Node.js env-var refactor (drop hardcoded url/resource, rely on OTEL_EXPORTER_OTLP_ENDPOINT + OTEL_SERVICE_NAME) is spec-correct.
• No em dashes or obvious AI-writing tells in added prose.
• Port 8080 → 8089 moves are consistent with the self-hosted UI reserving 8080.

Personas & JTBD: Serves agent/script/CI consumers and OTel-instrumentation users. The new "Confirm a service is sending spans" flow and the ingestion-key-scope clarification give a clean happy path with a real validation step and correct next-step links. JTBD checks pass; the ingestion-key API claim was verified against the docs' own service-account guidance.

Onboarding label: skipped — all data/docs/** changes are MODIFIED; the only added files are code (components/Region/regions.ts, utils/docs/canonicalDocsMarkdownPath.ts).

Open questions / residual risk

• The resolver adds up to 5 sequential self HEAD fetches (cache: 'no-store') per markdown 404 on legacy slugs. Bounded and only on the miss path, but worth watching latency on the ~365 legacy sources.
• PR notes a rebase-after-feat(agents): publish the OpenAPI spec and make agent surfaces recoverable #4067 dependency on 4 overlapping files and two tracked out-of-scope gaps (check-stale-urls .json blind spot; ingestion-key-only verification). Both acknowledged, not blockers here.

---

• Gather context (diff + skills + contributing playbooks)
• Review frontend/code changes
• Review docs changes
• Post inline findings
• Post summary
  · feat/agent-docs-discovery