[KEP-4] Built-in HTTP & MCP Server for Pipeline Execution

[DimedS]

Related PR: #5370
KEP shepherd: @DimedS

Q1. What are we trying to do?

Add two built-in server interfaces to Kedro so that pipelines can be triggered over HTTP (by orchestrators, dashboards, CI/CD) and discovered/executed by AI agents (Claude, Copilot, etc.) via the Model Context Protocol (MCP) — without requiring users to write any glue code.

kedro server http start   →  REST API (FastAPI)
kedro server mcp start    →  MCP server (FastMCP)

Both are thin wrappers around the same KedroSession.create() → session.run() path that the CLI uses. Full details and implementation are in PR #5370.

Q2. What problem is this proposal NOT designed to solve?

• Not a production-grade application server — Phase 1 is synchronous, single-worker, no auth. Production hardening (auth, TLS, multi-worker, async execution) is deferred to future phases.
• Not a replacement for orchestrators — This does not replace Airflow, Prefect, or Dagster. It provides an HTTP trigger point that orchestrators can call.
• Not a full pipeline management API — There is no run history, status polling, or queue. Phase 1 is fire-and-execute-and-return.

Q3. How is it done today, and what are the limits of current practice?

Today pipelines are triggered via:

• kedro run CLI
• Programmatic KedroSession in Python code

If users need HTTP access, they must build their own wrapper (Flask/FastAPI app around KedroSession). If they want AI agent integration, there is no path at all — they'd need to build a custom MCP server from scratch. Every team reinvents the same boilerplate.

Q4. What is new in your approach and why do you think it will be successful?

• One shared execution function — Both servers call a single execute_pipeline() in runner.py, which creates a KedroSession and calls session.run(). No duplication, no divergence from CLI behavior.
• Full CLI parity — Every kedro run parameter is available as a JSON field (HTTP) or tool parameter (MCP).
• Extensible by design — create_http_server() returns a standard FastAPI app; create_mcp_server() returns a standard FastMCP instance. Users add auth, CORS, observability, or custom tools using patterns they already know. No Kedro-specific plugin system to learn.
• Zero cost if unused — FastAPI/Uvicorn and the MCP SDK are optional extras (kedro[http], kedro[mcp]). No new hard dependencies.

Q5. Who cares? If you are successful, what difference will it make?

• Platform/infra teams get a standard HTTP endpoint to trigger Kedro from any system.
• AI-first teams get instant MCP integration — Claude, Copilot, and other agents can discover and run pipelines out of the box.
• New users get a more accessible entry point — interact with pipelines via HTTP or AI chat instead of CLI only.

Q6. What are the risks?

• Maintenance surface — Two server implementations to maintain alongside the CLI. Mitigated by the shared execute_pipeline() core.
• Scope creep — Users may expect production features (auth, async, queue) in Phase 1. Clear documentation of Phase 1 limitations is important.
• MCP SDK stability — Pinned to mcp>=1.0.0,<2.0.0. A breaking v2 release would require a compatibility update.

Q7. How long will it take?

Phase 1 implementation is complete in PR #5370. Remaining work: unit tests, integration tests, documentation.

Q8. What are the mid-term and final "exams" to check for success?

Mid-term

• HTTP server starts, serves /health and /run with full CLI parity ✅
• MCP server starts, exposes 4 tools, connects to VS Code and Claude Desktop ✅
• Shared execution core with no code duplication ✅

Final

• Unit and integration test coverage
• Documentation published
• User feedback collected post-release
• Decision on Phase 2 scope (async execution, run history)

---

Core Decisions for TSC Vote

The implementation details can be discussed during PR review. This KEP asks for alignment on four high-level decisions:

1. Should we proceed with server functionality at all?

Should Kedro provide built-in HTTP and/or MCP access to pipeline execution, or should this remain a user-side concern?

2. Core or plugin?

Should this live in kedro core (as implemented in #5370) or in a separate kedro-server plugin?

Arguments for core: stays in sync with session.run() automatically, first-class discoverability (kedro server), zero cost if unused (optional extras).

Arguments for plugin: smaller core surface, independent release cycle.

Note: extensibility is equivalent in both cases — the factory functions return standard FastAPI/FastMCP objects regardless of where they live.

3. Should both HTTP and MCP be included?

Or should we ship only one? They share the same execution core and CLI namespace, but serve different audiences (programmatic integration vs. AI agents).

4. Are you fine with the overall architecture?

The full architecture, file structure, design decisions, and extensibility model are documented in the PR #5370 description. Please review and flag any concerns.

---

Please vote +1/−1 in comments, not the poll!

[merelcht]

Can you clarify how we expect this to be called programatically (so not through CLI)? What does the endpoint/function look like a user can call directly from python? For e.g. the Kedro integration with MLRun the team said explicitly that they definitely don't want to use CLI.

[DimedS]

@merelcht, do you mean adding an example of how to connect to an already running server from a Python script, e.g. with the requests library:

import requests

response = requests.post(
    "http://127.0.0.1:8000/run",
    json={
        "pipeline": "data_processing",
        "env": "base",
    },
)

print(response.json())

Or do you mean starting the server itself from a Python script?

[merelcht]

Thank you, this is what I meant!

[deepyaman]

Should Kedro provide built-in HTTP and/or MCP access to pipeline execution, or should this remain a user-side concern?

Exposing pipelines as HTTP endpoints is a common-enough user concern that I think it's very justifiable to provide an out-of-the-box solution.

Should this live in kedro core (as implemented in #5370) or in a separate kedro-server plugin?

I think you can never go wrong with implementing as a plugin here—you can always bring it into core, if there's sufficient adoption. The main question I'd have re adding it to core is how confident we feel about the approach being finalized.

To this end, I think:

1. HTTP server could belong in either core or plugins (there's sufficient evidence over the past 5+ years that people want a solution like this, between kedro-server, kedro-grpc-server, and kedro-boot, together with a lot of related questions in Slack, etc.).
2. MCP server is way too new to go in core (more on this later).

Should both HTTP and MCP be included?

I don't think MCP should be included, especially in core. The ecosystem around this is way too new.

For one, is MCP even the right solution? If the goal is to provide local agent access, why not Agent Skills? This is simpler (no server) and cheaper (no tokens for API calls). Again, this could shift to something new given how volatile the ecosystem is, but at this point in time there seems to be a movement from MCP to Skills for many use cases.

Furthermore, FastMCP is popular, but it's is much less prevalent than FastAPI.

Are you fine with the overall architecture?

Not answering on MCP, since I think there are questions re whether this is even the right path forward at this time.

For the HTTP server:

1. Does this get you to production, or a "production-ready" POC? I'd be interested to learn more about what a user would need to do to actually get running in production from this, and what limitations there are (if any). For example, I think one of the big challenges around deploying to production is clobbering intermediate datasets, leading to race conditions between parallel executions. Is this meant more for scheduled, non-overlapping runs? Are there certain guidelines for when and how it should be used?
2. Alternative approaches? Kedro gRPC Server could also be appealing, especially with the clients from all the languages supporting gRPC, and looking at how other projects (e.g. Spark Connect) leverage gRPC. kedro-boot also provides server capabilities via FastAPI, IIRC. I don't think there's anything wrong with this approach, but I'd also be curious about the rationale for not taking another approach (especially if considered the gRPC one).

[rashidakanchwala]

I’m supportive of introducing a simple HTTP API layer; that was the original motivation. At this stage, it would be preferable to keep the API layer intentionally minimal and additive. This is still new territory for us, and we likely don’t want Kedro to take on orchestration responsibilities.

Concurrency control and avoiding overlapping runs can remain the responsibility of the orchestrator or the user (e.g., versioned datasets, namespacing, max concurrency limits), at least for Phase 1. We can be very explicit in the documentation about these expectations and provide clear guidance on recommended patterns for safe usage.

On MCP, I agree with @deepyaman .... while promising, the ecosystem is evolving quickly. Since we already have kedro-mcp, it may make sense to continue iterating there for now.

So a possible path forward could be:

• HTTP server in core (minimal scope)
• MCP server in kedro-mcp (experimental)

[merelcht]

Definitely +1 from me to proceed with this work. More specifically:

1. Should we proceed with server functionality at all?
   Yes +1

2. Core or plugin?
   I think HTTP server should go in Core

3. Should both HTTP and MCP be included?
   I'm on the same page as @rashidakanchwala and @deepyaman here. For now, I'd only build the HTTP (as part of core) and make adding the MCP lower priority.

4. Are you fine with the overall architecture?
   I didn't look at the details of the PR, but I agree with the overall architecture.

[ravi-kumar-pilla]

We should have this feature. +1 on should we have it

1. Should we proceed with server functionality at all?

Yes

2. Core or plugin?

For the details and making it first class kedro native support, I feel we should have it in core.

Rationale: I see there are lot of new features and we make some experimental and some introduced to core. As @deepyaman mentioned it is always easy and quick to make it a plugin and you can experiment and cannot go wrong. But at the same time user experience on installing a new plugin + Kedro team maintaining a new plugin seems an overkill for a kedro server which seems an obvious feature to be within kedro.

3. Should both HTTP and MCP be included?

I agree on implementing http server for now as part of core.

4. Are you fine with the overall architecture?

Yes. I could not think of any complex use-case based on my knowledge to go for gRPC server. But it would be worth considering in future as we have evidence of latency issues with HTTP.

Thank you

[DimedS]

Closing Summary

Thank you everyone for the discussion and feedback.

Based on the comments in this thread, we are closing this KEP with the following agreed direction:

Decisions

1. Server functionality

   • There is consensus that Kedro should provide built-in server functionality to trigger pipelines programmatically.
   • This addresses a common need currently solved by users building their own wrappers.

2. HTTP server

   • We will proceed with implementing a minimal HTTP server in Kedro core.
   • The goal for Phase 1 is to provide a simple trigger interface for pipeline execution, not a full orchestration layer.

3. MCP server

   • The MCP ecosystem is still evolving quickly.
   • For now, MCP support will not be included in Kedro core and experimentation can continue in the kedro-mcp project.

4. Scope for Phase 1

   • The HTTP server should remain minimal and additive.
   • Production concerns such as authentication, concurrency control, scheduling, or orchestration are out of scope and expected to be handled by external systems or users.