feat(agents): publish the OpenAPI spec and make agent surfaces recoverable

[revmag]

Pull Request

---

📄 Summary

Why does this change exist?
What problem does it solve, and why is this the right approach?

Agents reach /api-reference more than any other non-docs surface on signoz.io — 5,190 requests in the last 30 days, including claude-code/2.1.x (~360), ChatGPT-User (372), PerplexityBot (42), GPTBot (37) and OAI-SearchBot (29). It is also the one page with no markdown representation: /api-reference is on PAGE_MARKDOWN_EXCLUDED_PREFIXES, so .md and Accept: text/markdown both fall through to the interactive viewer's HTML — 840 words of visible text once tags are stripped. The page where an agent most needs structured data is the page that gives it the least.

This PR closes that, and the related discovery gaps:

Publish the OpenAPI spec. We already fetch a full OpenAPI 3.0.3 document (211 operations, 142 paths, security schemes) to render /api-reference, but only at /api/api-reference-openapi/<version> as YAML — referenced 0 times in llms.txt, llms-full.txt, docs/sitemap.md or robots.txt. It is now also served at the conventional /openapi.json and /openapi.yaml. The upstream spec ships an empty info.version, which reads as malformed to validators, so it is stamped with the release tag it was fetched from.

/api-reference.md. A dedicated route built from the spec — auth schemes, base URL, and every operation grouped by the spec's own tags. Built from the spec rather than run through the generic HTML→markdown pipeline, because the viewer's DOM carries almost no text. Accept: text/markdown on /api-reference rewrites to it.

Markdown 404s. The four markdown endpoints answered a miss with nine bytes of text/plain Not Found. A shared agentNotFoundResponse() now returns 404 with a short markdown body naming the missing path and linking llms.txt, the docs index, both sitemaps and the spec. Status stays 404 — this changes the body, not the code, so it does not create soft-404s.

MCP registry branding. Adds displayName, icon, icons, websiteUrl, and a server card at /.well-known/mcp/server-card.json.

llms.txt. Emits the > summary blockquote its own test already asserted (pre-existing red test), plus an ## API section linking the three new endpoints.

Issues closed by this PR

Part of the Agent Score epic (SigNoz/growth-pod#1170).

---

✅ Change Type

Select all that apply

• ✨ Feature
• 🐛 Bug fix (llms.txt blockquote — pre-existing failing test)
• ♻️ Refactor
• 🛠️ Infra / Tooling
• 🧪 Test-only

---

🧪 Testing Strategy

How was this change validated?

Tests added/updated: 16 new tests across openapi-markdown, mcp-discovery, agent-response-headers, llms-txt, proxy. The two proxy tests that encoded the old /api-reference passthrough are updated. Full suite: 285/285 node, 83/83 vitest, yarn lint clean, yarn check:stale-urls + yarn test:stale-urls pass, yarn build green with the new routes prerendered at 1d revalidate.

Manual verification — run against the preview deployment https://signoz-web-git-feat-agent-openapi-discovery-signoz.vercel.app:

#	What	Command	Result
1	Spec serves as JSON	curl -sI $P/openapi.json	200 application/json, 534,591 B, x-signoz-api-version: v0.139.0
2	Spec serves as YAML	curl -sI $P/openapi.yaml	200 application/yaml, 768,492 B, same version header
3	Markdown twin	curl -sI $P/api-reference.md	200 text/markdown, 19,685 B
4	MCP server card	curl -sI $P/.well-known/mcp/server-card.json	200 application/json, 2,011 B
5	Browser gets HTML	curl -sIL -H 'Accept: text/html' $P/api-reference	text/html ✅
6	Agent gets markdown	curl -sIL -H 'Accept: text/markdown' $P/api-reference	text/markdown ✅
7	Googlebot gets HTML	curl -sIL -A '…Googlebot/2.1…' $P/api-reference	text/html ✅
8	Spec cache policy	header check	public, s-maxage=86400, stale-while-revalidate=86400
9	Recovery 404 body	curl -s -H 'Accept: text/markdown' $P/nonexistent	404 text/markdown with the recovery link list

Reproduce with P=https://signoz-web-git-feat-agent-openapi-discovery-signoz.vercel.app.

Edge cases covered: trailing-slash variants; per-release specs at /api-reference/<version> unchanged; POST never rewritten; self-fetch recursion guard; nonexistent slugs; /llms.txt.md.

---

⚠️ Risk & Impact Assessment

What could break? How do we recover?

Blast radius

• Additive routes. /openapi.json, /openapi.yaml, /api-reference.md, /.well-known/mcp/server-card.json are new paths that previously 404'd. Nothing that worked before changes.
• One behavioural change on an existing URL: /api-reference with Accept: text/markdown now returns markdown instead of HTML. No browser and no search crawler sends that header (verified in test 7 — Googlebot gets HTML). Everything else about /api-reference is byte-identical, canonical included.
• Proxy runs site-wide, so the new predicates execute on every request — but they are pure pathname/header string checks and only fire on explicit markdown intent.
• Shared helper touches 4 existing endpoints: agentNotFoundResponse() changes the 404 body on /api/docs-markdown, /api/content-markdown, /api/page-markdown and /api-reference-openapi. Status code and cache semantics unchanged.
• Not touched: docs rendering, blog, HTML pages, redirects, data/docs/**.

Potential regressions

1. Cache poisoning between representations — one URL serving two content types. Mitigated by Vary: Accept. ⚠️ See the SEO section: this needs one reviewer decision.
2. Spec size — /openapi.yaml is 768 KB. Served at 1d revalidate + SWR, so it is a cache fill, not a per-request cost.
3. Upstream spec unavailable — getLatestOpenAPISpec() returning null yields agentNotFoundResponse (404) rather than a 500.
4. export const revalidate must be a literal — Next rejects API_SPEC_REVALIDATE_SECONDS, so the routes use 86400 with a comment, matching app/(site)/api-reference/page.tsx.

Rollback plan: revert. Every new path returns to 404 and /api-reference stops negotiating; nothing else is affected.

---

🔍 SEO Considerations

Verified on the preview, with production compared as the control:

Check	Result
New endpoints in sitemap.xml	0 hits — all 5 child sitemaps checked (blogs 808, docs 2,929, products 22, alternatives 12, corporate 10 = 3,781 URLs)
Canonical on /api-reference	https://signoz.io/api-reference/ — identical to production. The .md twin does not alter it
Googlebot receives HTML	✅ test 7
Markdown twin is noindex	✅ /api-reference.md uses agentResponse, which sets X-Robots-Tag: noindex (confirmed on prod /docs/introduction.md) — no duplicate-content risk against /api-reference
404s stay 404	Markdown bodies do not change the status code, so no soft-404s

Two items for reviewer decision:

1. /openapi.json and /openapi.yaml are not noindex. They build a raw NextResponse instead of going through agentResponse, so unlike every other agent endpoint they carry no X-Robots-Tag. (The noindex visible on the preview is Vercel's blanket preview header — confirmed, preview / and /docs/introduction/ carry it too while production does not. It is an environment artifact, not route behaviour.) A 534 KB JSON and 768 KB YAML are indexable as written. Agents find them via llms.txt, not search, so I'd add X-Robots-Tag: noindex to both for consistency — happy to do so if reviewers agree.

2. Vary: Accept does not reach the client on HTML responses. Preview /api-reference/ returns only vary: rsc, next-router-state-tree, …. The proxy appends Vary: Accept, but it does not survive on statically prerendered HTML. This is pre-existing, not introduced here — production /docs/introduction/ behaves identically, and docs have negotiated since before #3948. Markdown responses do carry vary: Accept (verified on /api-reference.md and prod /docs/introduction.md), and Vercel keys rewritten requests to different cache entries, so no poisoning has been observed. But this PR extends negotiation to a page that ranks, so it inherits the exposure. Worth a follow-up to set Vary: Accept at the next.config.js headers level, where it survives prerendering — out of scope here.

---

📝 Changelog

Field	Value
Deployment Type	Cloud / OSS / Enterprise
Change Type	Feature
Description	The SigNoz OpenAPI specification is now published at signoz.io/openapi.json and signoz.io/openapi.yaml, and the API reference is available as markdown at signoz.io/api-reference.md or by sending Accept: text/markdown.

---

📋 Checklist

• Tests added or explicitly not required
• Manually tested
• Breaking changes documented (none — additive; the only changed behaviour is markdown negotiation on /api-reference)
• Backward compatibility considered (per-release specs, existing docs negotiation and api-reference HTML all unchanged)

---

👀 Notes for Reviewers

Not addressed here. OAuth 2.0 / scoped permissions and unauthenticated MCP resources/list are product-side: the API authenticates with SigNoz-Api-Key, and the OAuth 2.1 flow lives on mcp.<region>.signoz.cloud, which already serves RFC 9728 metadata. mcp.us.signoz.cloud/mcp correctly returns 401 with a www-authenticate: Bearer resource_metadata=… challenge — so the is-agentic checks "MCP resources exposed" and "MCP resource quality" are structurally unpassable without exposing customer telemetry anonymously. They should be ignored, not chased. Organization JSON-LD contactPoint/address is separate, pending a decision on which entity address to publish.

Related PR. #4094 adds an API and access control section to llms.txt pointing at the spec's current URL (/api/api-reference-openapi/latest/). No file overlap with this PR. Once this merges, that link should be switched to /openapi.json.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
signoz-web	Ready	Preview	Sep 3, 2026 12:04pm UTC

[github-actions]

Claude finished @revmag's task in 2m 58s —— View job

---

Claude PR Review

Reviewed with the signoz-website-frontend-pr-review skill against contributing/site-code.md. This is a well-structured, well-tested PR — the new routes reuse existing helpers (agentResponse, apiReference fetchers, API_SPEC_REVALIDATE_SECONDS), the proxy rewrite ordering is correct, and the version-stamping rationale is sound. No correctness or architecture issues found. A few low-severity cleanups below.

Code findings

Low

• Operation count double-counts multi-tagged operations — utils/openapiMarkdown.ts:101. collectOperationsByTag lists each operation under every tag it declares, so operationCount (and the visible listing) counts multi-tagged operations more than once, potentially overstating the N operations across M paths header. Tests only cover single-tag ops. (inline)
• Unused yaml field — utils/openapiSpec.ts:14. LatestOpenAPISpec.yaml is fetched and stored but never read; /openapi.yaml serializes spec.document instead. Drop it or document why it's kept. (inline)

Nits (non-blocking, no inline)

• /openapi.json and /openapi.yaml routes duplicate the not-found + cache-header boilerplate; a tiny shared helper would DRY it, but it's minor.
• /api-reference.md sets varyAccept: true even though the route always returns markdown regardless of Accept; the extra Vary: Accept slightly fragments the CDN cache key for a fixed-path resource. Harmless, drop if convenient.
• /openapi.json / /openapi.yaml return a text/markdown 404 body (agentNotFoundResponse) for JSON/YAML endpoints — fine as an error path, just noting the content-type mismatch.

Verified good

• revalidate = 86400 literal correctly matches API_SPEC_REVALIDATE_SECONDS (86400) with an explanatory comment.
• MCP icon /svgs/icons/signoz.svg exists; siteMetadata.siteUrl has no trailing slash, so the constructed icon/website URLs are well-formed (matches the tests).
• Proxy: /api-reference (Accept markdown) → /api-reference.md, trailing-slash + bot-UA / per-release-spec paths, and Vary: Accept handling all covered by updated tests.
• stampSpecVersion preserves an existing version, backfills empty/missing ones, and handles a spec with no info block.

Not a blocker — the two Low items are worth a quick look before merge.

• Gather context (read changed files & diff)
• Apply signoz-website-frontend-pr-review skill
• Review new routes & utils (openapi, markdown, mcp discovery)
• Review proxy + routing changes
• Review tests
• Post inline findings & summary
  · branch feat/agent-openapi-discovery

[github-actions]

Claude finished @revmag's task in 2m 39s —— View job

---

Claude PR Review

Reviewed with the signoz-website-frontend-pr-review skill against contributing/site-code.md. This is a well-structured, well-tested PR — new routes reuse existing helpers (agentResponse, getLatestOpenAPISpec, API_SPEC_REVALIDATE_SECONDS), the proxy rewrite ordering/predicates are pure string checks that only fire on explicit markdown intent, and version-stamping is sound. No correctness or architecture issues found. A few low-severity items below.

Code findings

Low

• Unused yaml field — utils/openapiSpec.ts:14. LatestOpenAPISpec.yaml is stored (line 51) but never read anywhere (confirmed repo-wide); /openapi.yaml serializes spec.document so the stamped info.version is preserved. Dead field — drop it or document why it's kept. (inline)
• Raw NextResponse diverges from the shared agent pattern — app/(site)/openapi.json/route.ts:17, same in openapi.yaml. No X-Robots-Tag: noindex (both are indexable as written), and Cache-Control carries s-maxage/swr that Vercel strips at the client — the very split agentResponse/Vercel-CDN-Cache-Control exists to avoid. Consistency, not correctness. (inline)

Nits (non-blocking)

• /api-reference.md (app/(site)/api-reference.md/route.ts:12) sets varyAccept: true even though it always returns markdown regardless of Accept — the extra Vary: Accept slightly fragments the CDN key for a fixed-path resource. Drop if convenient.
• agentNotFoundResponse returns a text/markdown 404 body from the JSON/YAML endpoints — fine as an error path, just a content-type mismatch on that branch.
• Per-release-specs line (utils/openapiMarkdown.ts:126) points at the internal /api/api-reference-openapi/<release> rather than the .md convention llms.txt advertises. @crazyuploader's suggestion to switch to /api-reference/<release>.md is worth taking once feat(agents): serve markdown for versioned api-reference URLs #4100 lands.

Reviewer-decision items (raised in the PR body)

• noindex on /openapi.json + /openapi.yaml — agreed, add it. They're discovered via llms.txt, not search, and there's no reason for a 500 KB+ spec to be indexable. Bundled into the inline above.
• Vary: Accept not surviving on prerendered HTML — correctly diagnosed as pre-existing (docs already negotiate this way) and out of scope; a next.config.js headers-level fix is the right follow-up.

Verified good

• Operation-count double-count from the earlier review is fixed — openapiMarkdown.ts:103 now counts a Set of unique method route pairs, with an explanatory comment.
• stampSpecVersion backfills only empty/missing info.version, preserves an existing one, strips the leading v, and handles a missing info block.
• Proxy predicates: /api-reference (Accept markdown) → /api-reference.md, per-release spec + bot-UA paths, trailing-slash normalization, self-fetch recursion guard, and Vary: Accept scoping all covered by updated tests. revalidate = 86400 literal correctly matches API_SPEC_REVALIDATE_SECONDS.
• MCP icon /svgs/icons/signoz.svg exists; siteMetadata.siteUrl has no trailing slash, so constructed icon/website/docs URLs are well-formed.

Not a blocker — the two Low items are quick consistency cleanups worth a look before merge.
· branch feat/agent-openapi-discovery

[github-actions]

Unused yaml field — Low

LatestOpenAPISpec.yaml is populated at line 51 but never read: /openapi.yaml serializes spec.document via stringify(...) (correctly, so the stamped info.version is included), and no other consumer reads spec.yaml (confirmed repo-wide). Serving the raw YAML would re-introduce the empty info.version the stamping exists to fix, so this field is intentionally dead — drop it from the type and the return object, or add a one-liner explaining why it's retained, since the "raw YAML exactly as published" doc comment implies it's the served representation. (Flagged in the prior automated review and still present.)

[github-actions]

Raw NextResponse diverges from the shared agent-response pattern — Low

Both /openapi.json and /openapi.yaml build a raw NextResponse instead of going through agentResponse, so they diverge from every other agent endpoint in two ways:

1. No X-Robots-Tag: noindex. As you flagged in the SEO section, a 534 KB JSON / 768 KB YAML is indexable as written; agents reach them via llms.txt, not search. Recommend adding X-Robots-Tag: noindex to both for consistency with agentResponse.
2. Cache-Control carries s-maxage/stale-while-revalidate. Per the comment in utils/agentResponseHeaders.ts, Vercel strips those directives from Cache-Control before it reaches the client, leaving a bare public — which is exactly why the shared helper splits edge policy into Vercel-CDN-Cache-Control. These two routes reintroduce that split-brain. Consider mirroring the helper (or emitting Vercel-CDN-Cache-Control alongside a client-facing Cache-Control).

Neither is a correctness bug — just consistency with the pattern the rest of the agent surfaces already use.