Danny/mcp hosted servers (#16743)

dannyroosevelt · web-flow · commit 7918c48431f1 · 2025-05-20T23:19:23.000-07:00
* Adding info for remote servers

* Update pnpm-lock.yaml

* Updates
diff --git a/docs-v2/pages/connect/mcp/developers.mdx b/docs-v2/pages/connect/mcp/developers.mdx
@@ -1,42 +1,190 @@
-import { Steps } from 'nextra/components'
+import { Callout, Steps, Tabs } from 'nextra/components'
 
-# Deploy Pipedream MCP to your agent
-Deploy Pipedream's MCP servers to your application or agent and make authenticated request on behalf of your customers. Refer to our reference implementation [here](https://github.com/PipedreamHQ/pipedream/blob/master/modelcontextprotocol/README.md).
+# Add Pipedream MCP to your app or agent
 
-## Quickstart
+Add Pipedream's MCP server to your application or agent to make tool calls on behalf of your users to {process.env.PUBLIC_APPS}+ APIs and 10,000+ tools.
 
-<Steps>
+## Overview
 
-#### Prerequisites
+Pipedream's MCP server code is [publicly available on GitHub](https://github.com/PipedreamHQ/pipedream/blob/master/modelcontextprotocol/README.md), and you have two options for using Pipedream's MCP server in your app:
 
-To run your own MCP server, you'll need:
+1. [Use Pipedream's remote MCP server](#use-pipedreams-remote-mcp-server)
+2. [Self-host Pipedream's MCP server](#self-host-pipedreams-mcp-server)
+
+### Pipedream concepts to understand
+
+The MCP server accepts two route params:
+
+**`external_user_id`**
+
+- This is your user’s ID, in your system: whatever you use to uniquely identify them
+- Any requests made to that route are coupled to that end user and uses the connected account tied to that user
+- [See here](/connect/api/#external-users) for more detail
+
+**`app`**
+
+- The app's "name slug" (the unique identifier for the app)
+- [See below](#discover-available-mcp-servers) for discovering available apps
+
+## Getting started
+
+### Prerequisites
+
+For either option, you'll need:
 
 1. A [Pipedream account](https://pipedream.com/auth/signup)
 2. A [Pipedream project](/projects/#creating-projects). Accounts connected via MCP will be stored here.
 3. [Pipedream OAuth credentials](/rest-api/auth/#oauth)
 
-#### Set up environment variables
+#### Set up your environment
 
 Set the following environment variables:
 
 ```bash
 PIPEDREAM_CLIENT_ID=your_client_id
 PIPEDREAM_CLIENT_SECRET=your_client_secret
-PIPEDREAM_PROJECT_ID=your_project_id
-PIPEDREAM_ENVIRONMENT=development
+PIPEDREAM_PROJECT_ID=your_project_id # proj_xxxxxxx
+PIPEDREAM_ENVIRONMENT=development # development | production
+```
+
+### Authentication
+
+#### Developer authentication
+
+Your application authenticates with Pipedream using client credential OAuth. [See below](#api-authentication) for details.
+
+#### User account connections
+
+One of the core features of Pipedream Connect and our MCP product is the ability for your users to easily connect their accounts without having to build any of the authorization flow or handle token storage.
+
+You can handle account connections in one of two ways in your app:
+
+##### Add a button in your UI
+- Use Pipedream's [frontend SDK](/connect/managed-auth/quickstart/#use-the-pipedream-sdk-in-your-frontend) to let users connect their account directly in your UI
+- You can see an example of this when you connect any account in [mcp.pipedream.com](https://mcp.pipedream.com)
+
+##### Return a link
+- Use [Connect Link ](/connect/managed-auth/quickstart/#or-use-connect-link) to let your users open a Pipedream hosted page to connect their account
+- There is no implementation required for this option since it's already handled in Pipedream's MCP server
+- If a user doesn't have a connected account that's required for a given tool call, we'll return a URL in the tool call response. For example:
+
+```
+https://pipedream.com/_static/connect.html?token=ctok_xxxxxxx&connectLink=true&app={appSlug}
+```
+
+### Discover available MCP servers
+
+Pipedream provides [{process.env.PUBLIC_APPS}+ APIs as MCP servers](https://mcp.pipedream.com). Each server corresponds to an app integration (like Notion, Gmail, or Slack) and has its own specific set of tools that you can expose to OpenAI.
+
+<Tabs items={['List all apps', 'Search for a specific app']}>
+<Tabs.Tab>
+```javascript
+// Get all available apps (paginated)
+const apps = await pd.getApps();
+
+// Each app has these key properties:
+// - name_slug: Used in the MCP server URL (e.g., "notion", "gmail", "slack")
+// - name: Display name (e.g., "Notion", "Gmail", "Slack")
+```
+</Tabs.Tab>
+<Tabs.Tab>
+```javascript
+// Search by app name
+const notionApps = await pd.getApps({ q: "notion" });
+const gmailApps = await pd.getApps({ q: "gmail" });
+const slackApps = await pd.getApps({ q: "slack" });
 ```
+</Tabs.Tab>
+</Tabs>
+
+### Use Pipedream's remote MCP server
+
+<Callout type="info">
+Pipedream MCP server supports both SSE and streamable HTTP transport types, with no configuration required by the developer or MCP client.
+</Callout>
+
+#### Base URL
+
+```
+https://mcp.pipedream.net
+```
+
+#### API Authentication
+
+To authenticate requests to Pipedream's MCP server, you need to include an access token with every HTTP request. Here's how to get it:
 
-#### Run the MCP server
+<Tabs items={['Node.js', 'cURL']}>
+<Tabs.Tab> 
+```javascript
+import { createBackendClient } from "@pipedream/sdk/server";
 
-Pipedream's MCP servers can run in two modes:
+// Initialize the Pipedream client with the SDK
+const pd = createBackendClient({
+  environment: PIPEDREAM_ENVIRONMENT,
+  credentials: {
+    clientId: PIPEDREAM_CLIENT_ID,
+    clientSecret: PIPEDREAM_CLIENT_SECRET,
+  },
+  projectId: PIPEDREAM_PROJECT_ID
+});
 
-##### Stdio (for local testing)
+// Get access token for MCP server auth
+const accessToken = await pd.rawAccessToken();
 
+console.log(accessToken);
+```
+</Tabs.Tab>
+<Tabs.Tab>
 ```bash
-npx @pipedream/mcp stdio --app slack --external-user-id user123
+curl -s -X POST https://api.pipedream.com/v1/oauth/token \
+  -H "Content-Type: application/json" \
+  -d '{
+    "grant_type": "client_credentials", 
+    "client_id": "'$PIPEDREAM_CLIENT_ID'", 
+    "client_secret": "'$PIPEDREAM_CLIENT_SECRET'"
+  }'
+```
+</Tabs.Tab>
+</Tabs>
+
+#### Required headers
+
+Include these headers in every HTTP request:
+
+```javascript
+{
+  Authorization: `Bearer ${PIPEDREAM_ACCESS_TOKEN}`,
+  "x-pd-project-id": PIPEDREAM_PROJECT_ID, // proj_xxxxxxx
+  "x-pd-environment": PIPEDREAM_ENVIRONMENT // development | production
+}
 ```
 
-##### SSE (for hosting)
+### Self-host Pipedream's MCP server
+
+#### Using the `Dockerfile`
+
+If you have Docker installed locally, you can build and run the container from the [reference implementation](https://github.com/PipedreamHQ/pipedream/blob/master/modelcontextprotocol/Dockerfile):
+
+```console
+> docker build -t pipedream-connect .
+> docker run -d --name pd-mcp -p 3010:3010 --env-file .env pipedream-connect:latest
+```
+
+This exposes a generic MCP server at [http://localhost:3010/:external_user_id/:app](http://localhost:3010/:external_user_id/:app).
+
+#### Start the server with Streamable HTTP Transport
+
+```bash
+pnpm dev:http
+```
+
+You can use the optional env var `PD_SDK_DEBUG` to print out all the requests and responses going to the Connect API:
+
+```bash
+PD_SDK_DEBUG=true pnpm dev:http
+```
+
+#### Start the server with SSE
 
 ```bash
 npx @pipedream/mcp sse
@@ -46,13 +194,9 @@ This exposes routes at:
 - `GET /:external_user_id/:app` - App-specific SSE connection endpoint
 - `POST /:external_user_id/:app/messages` - App-specific message handler
 
-Where:
-- `:external_user_id` is your user's unique identifier in your system
-- `:app` is the app slug (e.g., "slack", "github")
-
 #### Customize for your needs
 
-You can host and customize the MCP server for your specific requirements:
+You can also host and customize the MCP server for your specific requirements:
 
 ```bash
 # Clone the repo
@@ -71,4 +215,24 @@ npm run start:sse:prod
 
 See the [MCP server README](https://github.com/PipedreamHQ/pipedream/blob/master/modelcontextprotocol/README.md) for detailed instructions on customization options.
 
-</Steps>
+### Use the MCP inspector
+
+The [MCP inspector](https://modelcontextprotocol.io/docs/tools/inspector) can be helpful when debugging tool calls.
+
+```bash
+npx @modelcontextprotocol/inspector
+```
+
+Enter the server URL:
+
+If using Pipedream's remote server:
+
+```
+https://mcp.pipedream.net/{external_user_id}/{app_slug}
+```
+
+If running locally:
+
+```
+http://localhost:3010/{external_user_id}/{app_slug}
+```
diff --git a/docs-v2/pages/connect/mcp/openai.mdx b/docs-v2/pages/connect/mcp/openai.mdx
@@ -5,10 +5,10 @@ import { Callout, Tabs, Steps } from 'nextra/components'
 Access {process.env.PUBLIC_APPS}+ APIs and 10,000+ tools in OpenAI using Pipedream Connect. MCP makes it easy to extend the capabilities of any LLM or agent, and Pipedream offers drop-in support for [calling tools in OpenAI](https://platform.openai.com/docs/guides/tools?api-mode=responses) .
 
 <Callout type="info">
-Pipedream Connect includes built-in user authentication for [every MCP server](https://mcp.pipedream.com), which means you don't need to build any authorization flows or deal with token storage and refresh in order to make authenticated requests on behalf of your users. [Learn more below](#account-connection).
+Pipedream Connect includes built-in user authentication for [every MCP server](https://mcp.pipedream.com), which means you don't need to build any authorization flows or deal with token storage and refresh in order to make authenticated requests on behalf of your users. [Learn more here](/connect/mcp/developers/#user-account-connections).
 </Callout>
 
-### Getting started
+## Getting started
 
 <Steps>
 
@@ -30,34 +30,13 @@ Now set the following environment variables (learn more about environments in Pi
 OPENAI_API_KEY=your_openai_api_key
 PIPEDREAM_CLIENT_ID=your_client_id
 PIPEDREAM_CLIENT_SECRET=your_client_secret
-PIPEDREAM_PROJECT_ID=your_project_id
-PIPEDREAM_ENVIRONMENT=development
+PIPEDREAM_PROJECT_ID=your_project_id # proj_xxxxxxx
+PIPEDREAM_ENVIRONMENT=development # development | production
 ```
 
 ### Discover available MCP servers
 
-Pipedream provides [{process.env.PUBLIC_APPS}+ APIs as MCP servers](https://mcp.pipedream.com) that can be used with OpenAI's tool calls. Each server corresponds to an app integration (like Notion, Gmail, or Slack) and has its own specific set of tools that you can expose to OpenAI.
-
-<Tabs items={['List all apps', 'Search for a specific app']}>
-<Tabs.Tab>
-```javascript
-// Get all available apps (paginated)
-const apps = await pd.getApps();
-
-// Each app has these key properties:
-// - name_slug: Used in the MCP server URL (e.g., "notion", "gmail", "slack")
-// - name: Display name (e.g., "Notion", "Gmail", "Slack")
-```
-</Tabs.Tab>
-<Tabs.Tab>
-```javascript
-// Search by app name
-const notionApps = await pd.getApps({ q: "notion" });
-const gmailApps = await pd.getApps({ q: "gmail" });
-const slackApps = await pd.getApps({ q: "slack" });
-```
-</Tabs.Tab>
-</Tabs>
+[See here](/connect/mcp/developers/#discover-available-mcp-servers) for guidance on discovering the apps Pipedream has available as MCP servers.
 
 ### Generate a model response in OpenAI with Pipedream MCP
 
@@ -120,8 +99,6 @@ console.log(response);
 </Tabs.Tab>
 <Tabs.Tab>
 ```bash
-# Complete example from start to finish
-
 # Step 1: Get access token from Pipedream
 ACCESS_TOKEN=$(curl -s -X POST https://api.pipedream.com/v1/oauth/token \
   -H "Content-Type: application/json" \
@@ -164,24 +141,5 @@ curl -X POST https://api.openai.com/v1/chat/completions \
   }'
 ```
 </Tabs.Tab>
-</Tabs> 
+</Tabs>
 </Steps>
-
-## Account Connection
-
-One of the core features of Pipedream Connect and our MCP product is the ability for your users to easily connect their accounts without having to build any of the authorization flow or handle token storage.
-
-You can handle account connections in one of two ways in your app:
-
-### Add a button in your UI
-- Use Pipedream's [frontend SDK](/connect/managed-auth/quickstart/#use-the-pipedream-sdk-in-your-frontend) to let users connect their account directly in your UI
-- You can see an example of this when you connect any account in [mcp.pipedream.com](https://mcp.pipedream.com)
-
-### Return a link
-- Use [Connect Link ](/connect/managed-auth/quickstart/#or-use-connect-link) to let your users open a Pipedream hosted page to connect their account
-- There is no implementation required for this option since it's already handled in Pipedream's MCP server
-- If a user doesn't have a connected account that's required for a given tool call, we'll return a URL in the tool call response. For example:
-
-```
-https://pipedream.com/_static/connect.html?token=ctok_xxxxxxx&connectLink=true&app=notion
-```
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
