OAuth updates for mcp-agent (lastmile-ai#547)

Allow OAuth for MCP servers used by agents

---------

Co-authored-by: Roman van der Krogt <[email protected]>

Author: 2 people

auth_spec.png

@@ -123,6 +123,54 @@ mcp-agent uses two configuration files:
   </Tab>
 </Tabs>
 
+## OAuth Configuration
+
+MCP Agent exposes two complementary OAuth configuration blocks:
+
+- `authorization` describes how the MCP Agent server validates inbound bearer tokens and publishes protected resource metadata.
+- `oauth` configures delegated authorization when the agent connects to downstream MCP servers.
+
+```yaml
+authorization:
+  enabled: true
+  issuer_url: https://auth.example.com
+  resource_server_url: https://agent.example.com/mcp
+  required_scopes: ["mcp.read", "mcp.write"]
+  introspection_endpoint: https://auth.example.com/oauth/introspect
+  introspection_client_id: ${INTROSPECTION_CLIENT_ID}
+  introspection_client_secret: ${INTROSPECTION_CLIENT_SECRET}
+
+oauth:
+  callback_base_url: https://agent.example.com
+  flow_timeout_seconds: 180
+  token_store:
+    backend: memory   # set to "redis" for multi-instance deployments
+    refresh_leeway_seconds: 90
+    redis_url: redis://localhost:6379
+    redis_prefix: mcp_agent:oauth_tokens
+
+mcp:
+  servers:
+    github:
+      transport: streamable_http
+      url: https://github.mcp.example.com/mcp
+      auth:
+        oauth:
+          enabled: true
+          scopes: ["repo", "user:email"]
+          client_id: ${GITHUB_MCP_CLIENT_ID}
+          client_secret: ${GITHUB_MCP_CLIENT_SECRET}
+          include_resource_parameter: false  # disable RFC 8707 resource param for providers like GitHub
+          redirect_uri_options:
+            - https://agent.example.com/internal/oauth/callback
+```
+
+- When `authorization.enabled` is true the MCP server advertises `/.well-known/oauth-protected-resource` and enforces bearer tokens using the provided introspection or JWKS configuration.
+- `oauth` enables delegated authorization flows; the default in-memory token store is ideal for local development while Redis is recommended for production clusters.
+- To use Redis for token storage, configure `token_store.backend: redis` and supply `redis_url` (see optional dependency `mcp-agent[redis]`).
+- Downstream servers opt into OAuth via `mcp.servers.<name>.auth.oauth`. Supplying a `client_id`/`client_secret` allows immediate usage; support for dynamic client registration is planned as a follow-up.
+- Some providers (including GitHub) reject the RFC 8707 `resource` parameter. Set `include_resource_parameter: false` in the client settings for those services.
+
 ## Configuration Reference
 
 ### Execution Engine

docs/configuration.mdx

@@ -0,0 +1,110 @@
+# MCP Agent OAuth Support
+
+## Goals
+- Protect MCP Agent Cloud servers using OAuth 2.1 so MCP clients obtain tokens via standard flows.
+- Enable MCP Agent runtimes to authenticate to downstream MCP servers that require OAuth access tokens.
+- Provide pluggable token storage for both local development (in-memory) and multi-instance deployments (Redis planned).
+- Maintain compatibility with MCP Authorization spec (RFC 8414, RFC 9728, OAuth 2.1 + PKCE, Resource Indicators) and the proposed delegated authorization SEP.
+
+## Architecture Overview
+
+### Components
+1. **Auth Server Integration** – Configure the FastMCP instance with `AuthSettings` and a custom `TokenVerifier` that calls MCP Agent Cloud auth services.
+2. **Protected Resource Metadata** – Serve `/.well-known/oauth-protected-resource` using FastMCP hooks so clients can discover the auth server.
+3. **Access Token Validation** – Enforce bearer tokens on every inbound MCP request via `RequireAuthMiddleware`, populating the request context with the authenticated user.
+4. **OAuth Token Service** – New `mcp_agent.oauth` package with:
+  - `TokenStore`/`TokenRecord` abstractions
+  - `InMemoryTokenStore` and Redis-backed implementation (optional for multi-instance)
+   - `TokenManager` orchestration (acquire, refresh, revoke)
+   - `OAuthHttpxAuth` for attaching tokens to downstream HTTP transports
+   - `AuthorizationFlowCoordinator` that interacts with the user via MCP `auth/request`.
+     When no upstream client session is available, a client-only loopback flow starts a
+     temporary local callback listener on 127.0.0.1 using a configurable fixed port list
+     (default: 33418, 33419, 33420), opens the browser, and completes the PKCE code flow.
+5. **Delegated Authorization UI Flow** – Extend the gateway/session relay so servers can send `auth/request` messages to MCP clients, capturing authorization codes via either:
+   - Client-returned callback URL (preferred, works with SEP-capable clients)
+   - MCP Agent hosted callback endpoint (`/internal/oauth/callback/{flow_id}`) as a fallback / native-app style loopback.
+6. **Configuration Surface** – Extend `Settings` and per-server `MCPServerAuthSettings` to describe OAuth behaviour (scopes, preferred auth server, redirect URIs, etc.) and global token-store configuration.
+
+### Key Data Flow
+1. **Inbound Requests**
+   - Client presents bearer token ⇒ `BearerAuthBackend` + `MCPAgentTokenVerifier` introspect token.
+   - Verified token populates context with `OAuthUserIdentity` (provider + subject + email).
+   - Context is propagated into workflows/sessions so downstream OAuth flows know the acting user.
+
+2. **Outbound HTTP (downstream MCP server)**
+   - `ServerRegistry` detects `auth.oauth` configuration.
+   - Wraps HTTP transport with `OAuthHttpxAuth` which requests an access token from `TokenManager`.
+   - `TokenManager` checks store; if missing/expired ⇒ `AuthorizationFlowCoordinator` performs RFC 9728 discovery, PKCE, delegated browser flow through MCP client, exchanges code for tokens, caches result.
+   - Requests automatically retry after token refresh when a response returns 401/invalid token.
+
+3. **Token Storage**
+   - Tokens stored per `(user_identity, resource, authorization_server)` tuple with metadata (scopes, expiry, refresh token, provider claims).
+   - Store implements optimistic locking to avoid concurrent refresh storms.
+   - Pluggable backend (`InMemoryTokenStore` initial, Redis follow-up).
+
+## Module Plan
+
+```
+src/mcp_agent/oauth/
+  __init__.py
+  identity.py           # OAuthUserIdentity, helpers to extract from auth context
+  records.py            # TokenRecord dataclass/pydantic model
+  store/base.py         # TokenStore protocol
+  store/in_memory.py    # Default store
+  manager.py            # TokenManager (get/refresh/invalidate)
+  flow.py               # AuthorizationFlowCoordinator
+  http/auth.py          # OAuthHttpxAuth (httpx.Auth implementation)
+  metadata.py           # RFC 8414 + RFC 9728 discovery helpers
+  pkce.py               # PKCE + state utilities
+  errors.py             # Custom exception hierarchy
+```
+
+Integration touchpoints:
+- `mcp_agent/config.py` – add OAuth settings models.
+- `mcp_agent/core/context.py` – add `token_manager`, `token_store`, `oauth_config` fields.
+- `mcp_agent/app.py` – initialize token store/manager based on settings.
+- `mcp_agent/server/app_server.py` – configure FastMCP auth settings, register callback route, surface user identity, extend relay to handle `auth/request`.
+- `mcp_agent/mcp/mcp_server_registry.py` & `mcp_agent/mcp/mcp_connection_manager.py` – wire `OAuthHttpxAuth` into HTTP transports and expose helper for manual token teardown.
+- `mcp_agent/mcp/client_proxy.py` – add proxy helpers for `auth/request`.
+- `SessionProxy` – add direct request helper for `auth/request` and ensure Temporal flow support.
+- `examples/mcp_agent_server/*` – demonstrate configuration changes.
+- Tests – new suite exercising token store, metadata discovery, flow orchestration (with mocked HTTP + client responses).
+
+## OAuth Flow Details
+1. **Discovery**
+   - If downstream server responds 401 with `WWW-Authenticate`, parse for `resource_metadata` ⇒ GET metadata ⇒ determine auth server URL(s).
+   - Fetch authorization server metadata (RFC 8414).
+   - Perform optional dynamic client registration when configured and supported.
+
+2. **Authorization Request**
+   - Generate PKCE challenge/verifier, secure `state`, choose `redirect_uri`.
+   - Build authorization URL including `resource` parameter (RFC 8707) + requested scopes.
+   - Invoke `auth/request` via SessionProxy → MCP client opens browser.
+
+3. **Callback Handling**
+   - Preferred: MCP client returns callback URL payload via request result.
+   - Fallback: Authorization server redirects to `/internal/oauth/callback/{flow_id}`.
+   - Coordinator validates `state`, extracts `code` (and errors).
+
+4. **Token Exchange / Storage**
+   - POST token endpoint with code + PKCE verifier + resource.
+   - Store access token, refresh token, expiry, scope, provider metadata.
+   - Associate tokens with user identity for reuse.
+
+5. **Refresh / Revocation**
+   - Manager refreshes when expiry within configurable grace window.
+   - Invalidate token on refresh failure or when server responses indicate revocation.
+   - Provide method to revoke tokens via authorization server when supported.
+
+## Open Questions / Follow-ups
+- Additional operational hardening (token rotation policies, rate limits).
+- How LastMile auth server exposes token introspection + JWKS; need concrete endpoint specs to finalize `MCPAgentTokenVerifier`.
+- MCP client adoption of `auth/request` SEP – need capability detection; until widely supported we rely on hosted callback fallback & manual instructions.
+- Access control DSL (include/exclude by email/domain) – to be evaluated once token identity payload finalized.
+
+## Testing Strategy
+- Unit tests for token store concurrency + expiry handling.
+- Metadata discovery + PKCE generation (pure python tests).
+- Integration-style test for delegated flow using mocked HTTP server + fake MCP client (ensures `auth/request` plumbing works end-to-end).
+- Tests around server 401 enforcement + WWW-Authenticate header.

docs/oauth_support_design.md

@@ -0,0 +1,40 @@
+# OAuth Basic MCP Agent example (client-only loopback)
+
+This example mirrors `mcp_basic_agent` but adds GitHub MCP with OAuth using the client-only loopback flow.
+
+## Setup
+
+1. Register a GitHub OAuth App and add redirect URIs (at least one of):
+
+   - `http://127.0.0.1:33418/callback`
+   - `http://127.0.0.1:33419/callback`
+   - `http://localhost:33418/callback`
+
+2. Copy the secrets template and fill in your API keys / OAuth client (or export the env vars manually):
+
+```bash
+cp mcp_agent.secrets.yaml.example mcp_agent.secrets.yaml
+```
+
+3. Configuration is loaded from `mcp_agent.config.yaml` and secrets from
+   `mcp_agent.secrets.yaml`. Populate the secrets file (or export the matching
+   environment variables) with your GitHub OAuth credentials before running.
+
+4. (Optional) To persist tokens across runs, start Redis and set `OAUTH_REDIS_URL`:
+
+```bash
+docker run --rm -p 6379:6379 redis:7-alpine
+export OAUTH_REDIS_URL="redis://127.0.0.1:6379"
+```
+
+5. Install deps and run:
+
+```bash
+uv pip install -r requirements.txt
+# If you populated the secrets file you can skip these exports.
+export GITHUB_CLIENT_ID=...
+export GITHUB_CLIENT_SECRET=...
+uv run main.py
+```
+
+On first run, a browser window opens to authorize GitHub; subsequent runs reuse the cached token.

examples/basic/oauth_basic_agent/README.md

@@ -0,0 +1,144 @@
+import asyncio
+import inspect
+import os
+import time
+
+from mcp_agent.app import MCPApp
+from mcp_agent.config import get_settings, OAuthTokenStoreSettings, OAuthSettings
+from mcp_agent.agents.agent import Agent
+from mcp_agent.workflows.llm.augmented_llm_anthropic import AnthropicAugmentedLLM
+from mcp_agent.workflows.llm.augmented_llm_openai import OpenAIAugmentedLLM
+from mcp_agent.tracing.token_counter import TokenSummary
+
+
+def _load_settings():
+    signature = inspect.signature(get_settings)
+    if "set_global" in signature.parameters:
+        return get_settings(set_global=False)
+    return get_settings()
+
+
+settings = _load_settings()
+
+redis_url = os.environ.get("OAUTH_REDIS_URL")
+if redis_url:
+    settings.oauth = settings.oauth or OAuthSettings()
+    settings.oauth.token_store = OAuthTokenStoreSettings(
+        backend="redis",
+        redis_url=redis_url,
+    )
+elif not getattr(settings.oauth, "token_store", None):
+    settings.oauth = settings.oauth or OAuthSettings()
+    settings.oauth.token_store = OAuthTokenStoreSettings()
+
+github_settings = (
+    settings.mcp.servers.get("github")
+    if settings.mcp and settings.mcp.servers
+    else None
+)
+github_oauth = (
+    github_settings.auth.oauth
+    if github_settings and github_settings.auth and github_settings.auth.oauth
+    else None
+)
+
+if not github_oauth or not github_oauth.client_id or not github_oauth.client_secret:
+    raise SystemExit(
+        "GitHub OAuth client_id/client_secret must be provided via mcp_agent.config.yaml or mcp_agent.secrets.yaml."
+    )
+
+app = MCPApp(
+    name="oauth_basic_agent", settings=settings, session_id="oauth-basic-agent"
+)
+
+
+@app.tool()
+async def example_usage() -> str:
+    async with app.run() as agent_app:
+        logger = agent_app.logger
+        context = agent_app.context
+        result = ""
+
+        logger.info("Current config:", data=context.config.model_dump())
+
+        context.config.mcp.servers["filesystem"].args.extend([os.getcwd()])
+
+        finder_agent = Agent(
+            name="finder",
+            instruction="""You are an agent with access to the filesystem,
+            as well as the ability to fetch URLs and GitHub MCP. Your job is to
+            identify the closest match to a user's request, make the appropriate tool
+            calls, and return useful results.""",
+            server_names=["fetch", "filesystem", "github"],
+        )
+
+        async with finder_agent:
+            logger.info("finder: Connected to server, calling list_tools...")
+            tools_list = await finder_agent.list_tools()
+            logger.info("Tools available:", data=tools_list.model_dump())
+
+            llm = await finder_agent.attach_llm(OpenAIAugmentedLLM)
+
+            # GitHub MCP server use
+            github_repos = await llm.generate_str(
+                message="Use the GitHub MCP server to find the top 3 public repositories for the GitHub organization lastmile-ai and list their names.",
+            )
+            logger.info(
+                f"Top 3 public repositories for the GitHub organization lastmile-ai: {github_repos}"
+            )
+
+            result += f"\n\nTop 3 public repositories for the GitHub organization lastmile-ai: {github_repos}"
+
+            # Filesystem MCP server use
+            config_contents = await llm.generate_str(
+                message="Print the contents of mcp_agent.config.yaml verbatim",
+            )
+            logger.info(f"mcp_agent.config.yaml contents: {config_contents}")
+            result += f"\n\nContents of mcp_agent.config.yaml: {config_contents}"
+
+            # Switch to Anthropic LLM
+            llm = await finder_agent.attach_llm(AnthropicAugmentedLLM)
+
+            # fetch MCP server use
+            mcp_introduction = await llm.generate_str(
+                message="Print the first 2 paragraphs of https://modelcontextprotocol.io/introduction",
+            )
+            logger.info(
+                f"First 2 paragraphs of Model Context Protocol docs: {mcp_introduction}"
+            )
+            result += f"\n\nFirst 2 paragraphs of Model Context Protocol docs: {mcp_introduction}"
+
+        await display_token_summary(agent_app)
+    return result
+
+
+async def display_token_summary(app_ctx: MCPApp, agent: Agent | None = None):
+    summary: TokenSummary = await app_ctx.get_token_summary()
+
+    print("\n" + "=" * 50)
+    print("TOKEN USAGE SUMMARY")
+    print("=" * 50)
+
+    print("\nTotal Usage:")
+    print(f"  Total tokens: {summary.usage.total_tokens:,}")
+    print(f"  Input tokens: {summary.usage.input_tokens:,}")
+    print(f"  Output tokens: {summary.usage.output_tokens:,}")
+    print(f"  Total cost: ${summary.cost:.4f}")
+
+    if summary.model_usage:
+        print("\nBreakdown by Model:")
+        for model_key, data in summary.model_usage.items():
+            print(f"\n  {model_key}:")
+            print(
+                f"    Tokens: {data.usage.total_tokens:,} (input: {data.usage.input_tokens:,}, output: {data.usage.output_tokens:,})"
+            )
+            print(f"    Cost: ${data.cost:.4f}")
+
+    print("\n" + "=" * 50)
+
+
+if __name__ == "__main__":
+    start = time.time()
+    asyncio.run(example_usage())
+    end = time.time()
+    print(f"Total run time: {end - start:.2f}s")

examples/basic/oauth_basic_agent/main.py

@@ -0,0 +1,36 @@
+$schema: ../../../schema/mcp-agent.config.schema.json
+
+execution_engine: asyncio
+logger:
+  transports: [console, file]
+  level: info
+  path_settings:
+    path_pattern: "logs/mcp-agent-{unique_id}.jsonl"
+    unique_id: "timestamp"
+
+oauth:
+  loopback_ports: [33418, 33419, 33420]
+
+mcp:
+  servers:
+    fetch:
+      command: "uvx"
+      args: ["mcp-server-fetch"]
+    filesystem:
+      command: "npx"
+      args: ["-y", "@modelcontextprotocol/server-filesystem"]
+    github:
+      transport: streamable_http
+      url: "https://api.githubcopilot.com/mcp/"
+      auth:
+        oauth:
+          enabled: true
+          scopes: ["read:org", "public_repo", "user:email"]
+          authorization_server: "https://github.com/login/oauth"
+          use_internal_callback: false
+          include_resource_parameter: false
+
+openai:
+  default_model: "gpt-4o-mini"
+anthropic:
+  default_model: claude-sonnet-4-20250514