Consider supporting 3rd party Prometheus HTTP API implementations via flag (ex: `--backend=thanos`)

[tjhop]

About

Prometheus is often a component within a larger collection of systems to store/manage/query metrics. As such, some 3rd party systems implement at least part of the Prometheus HTTP API. This is especially true for API endpoints like: query, range query, exemplar query, label names, label values, series, metric metadata, etc so that these 3rd party systems can serve as a "drop in" replacement for use with querying metrics in systems like grafana/perses.

Proposal

Add a flag --backend=$system that can potentially filter/tailor the list of tools available to the user based on the backend tool. This flag can be used to filter/change the tools that are exposed to the user based on the backend system being used.

Examples

Thanos Query

The querier implements the Prometheus HTTP v1 API according to the docs, but in practice it's not a 100% implementation:

• Thanos primarily uses flags rather than a main config file, so the config api endpoint isn't implemented
• Thanos has the stores endpoint that lists all valid store API servers, which is an additional API endpoint

These examples were off the top of my head; there may be other differences between Thanos/Prometheus APIs.

Grafana Mimir

Mimir has a much larger API footprint, including allowing for things like creating/updating/deleting rules via the API. This would be a significant number of new tools to add, if full support is the goal.

Cortex

Similar to Mimir, Cortex has a large API footprint and allows for lots of extra functionality. This would be a significant number of new tools to add, but at least they're pretty much the same between cortex and mimir.

VictoriaMetrics

VictoriaMetrics already has an existing VM community MCP server, so it would seem unnecessary to support VM here.

Others

There are also obviously tons of other services that expose a "Prometheus compatible API", but it's not feasible to support them all. In my experience, talking about Prometheus API support primarily centers around query and metadata/discovery endpoint support, so even without backend-specific support, the MCP server is already capable of supporting the vast majority of Prometheus API compatible tools/integrations.

[tjhop]

@oberkem I've been kicking around this idea the last few days and I think it may prove helpful/related to some of the rule management things we've discussed in #3, curious about your thoughts. I'm still unsure I'd even like to take the project down such a path, since it would take what is currently a tightly scoped project and make it very open-ended.

[oberkem]

I think there are basically 3 options at the table. Happy to hear if there are more options.

1. Unified approach: one mcp server that can handle prometheus compatible implementations

2. Composable MCP Servers: this MCP server only handles prometheus methods, and 3rd party options can implement their own specific tools while also disabling tools from this MCP implementation. Mimir has cardinality APIs as an example that no other tool or prometheus has implemented. Some 3rd party tools might have limited support for certain APIs such as examplars. These differences can be handled in 3rd party specific MCP servers while it composes this MCP server at build time assuming that runtime is cumbersome to maintain.

3. Backend specific approach: Either 3rd party maintainers write their own mcp servers or they might reuse the libraries from this mcp implementation. Grafana has one already https://github.com/grafana/mcp-grafana/tree/main/tools. We created a basic set of toolkit https://github.com/awslabs/mcp/blob/main/src/prometheus-mcp-server/awslabs/prometheus_mcp_server/server.py

I would evaluate these options against (possibly more to consider here):

• Security & Auth: some backends will have different auth model that needs to be supported but won't be applicable to all backends
• Development complexity & maintainability: some APIs won't be shared across 3rd party plugin, alert management is also different.
• Extensibility: Leverage tools across both prometheus and 3p but also across 3p
• Deployment complexity: Integration with Agents

We should hear from prometheus maintainers & 3rd party maintainers. Prometheus Dev summit might be a good place to discuss this but here is my initial thoughts

Requirement	Option 1: Unified	Option 2: Composable	Option 3: Backend Specific
Security & Auth	Need to be able to handle multiple auth mechanisms	auth specific code can live in 3rd party mcp servers	every MCP would handle auth for their own specific backend
Development complexity & Maintainability	medium complexity as we would have to handle all the flags in one place	have this repo compatible to prometheus and let other implementations handle additional and restricted tools as well as auth	there will most likely be duplicated code and optimizations / improvements would not be shared
Extensibility	3rd party tools could share each others tool additions	additional APIs would have to implemented in each 3rd party mcp server	opportunity to lose core improvements on top of cons for option 2
Deployment complexity	Low - single api endpoint	Low if single api endpoint via build time integration & medium if composed at run time (agent would have to deal with 2 endpoints and different tools)	this seems more simple compared to option 2