Merge branch 'main' into fweinberger/sep-1034-client

Author: pcarleton

.github/workflows/ci.yml

@@ -3,6 +3,8 @@ on:
     branches:
       - main
   pull_request:
+  release:
+    types: [published]
 
 permissions:
   contents: read
@@ -19,9 +21,34 @@ jobs:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
         with:
-          node-version: 22
+          node-version: 24
           cache: npm
 
       - run: npm ci
+      - run: npm run check
+      - run: npm run build
       - run: npm test
-      - run: npm run lint
+
+  publish:
+    runs-on: ubuntu-latest
+    if: github.event_name == 'release'
+    environment: release
+    needs: [test]
+
+    permissions:
+      contents: read
+      id-token: write
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+          cache: npm
+          registry-url: 'https://registry.npmjs.org'
+
+      - run: npm ci
+
+      - run: npm publish --provenance --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/pr-publish.yml

@@ -0,0 +1,27 @@
+name: Publish PR to pkg.pr.new
+permissions:
+  contents: read
+on:
+  pull_request:
+  push:
+    branches:
+      - '**'
+    tags:
+      - '!**'
+
+jobs:
+  pr-publish:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+          cache: npm
+
+      - run: npm ci
+      - name: Build
+        run: npm run build
+      - name: Publish
+        run: npx pkg-pr-new publish

.gitignore

@@ -1,3 +1,4 @@
 node_modules
 results/
-lefthook-local.yml
+lefthook-local.yml
+dist

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Anthropic, PBC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,6 +1,6 @@
 # MCP Conformance Test Framework
 
-A framework for testing MCP (Model Context Protocol) client implementations against the specification.
+A framework for testing MCP (Model Context Protocol) client and server implementations against the specification.
 
 > [!WARNING] This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord.
 
@@ -9,56 +9,110 @@ A framework for testing MCP (Model Context Protocol) client implementations agai
 ### Testing Clients
 
 ```bash
-npm install
-npm run start -- --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
+npx @modelcontextprotocol/conformance client --command "tsx examples/clients/typescript/test1.ts" --scenario initialize
 ```
 
 ### Testing Servers
 
 ```bash
-npm run test:server -- --server-url http://localhost:3000/mcp --all
+# Run all server scenarios (default)
+npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
+
+# Run a single scenario
+npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize
+```
+
+### List Available Scenarios
+
+```bash
+npx @modelcontextprotocol/conformance list
 ```
 
 ## Overview
 
-The conformance test framework validates MCP client implementations by:
+The conformance test framework validates MCP implementations by:
+
+**For Clients:**
 
 1. Starting a test server for the specified scenario
 2. Running the client implementation with the test server URL
 3. Capturing MCP protocol interactions
 4. Running conformance checks against the specification
 5. Generating detailed test results
 
+**For Servers:**
+
+1. Connecting to the running server as an MCP client
+2. Sending test requests and capturing responses
+3. Running conformance checks against server behavior
+4. Generating detailed test results
+
 ## Usage
 
+### Client Testing
+
 ```bash
-npm run start -- --command "<client-command>" --scenario <scenario-name>
+npx @modelcontextprotocol/conformance client --command "<client-command>" --scenario <scenario-name> [options]
 ```
 
+**Options:**
+
 - `--command` - The command to run your MCP client (can include flags)
 - `--scenario` - The test scenario to run (e.g., "initialize")
+- `--timeout` - Timeout in milliseconds (default: 30000)
+- `--verbose` - Show verbose output
 
 The framework appends the server URL as the final argument to your command.
 
+### Server Testing
+
+```bash
+npx @modelcontextprotocol/conformance server --url <url> [--scenario <scenario>]
+```
+
+**Options:**
+
+- `--url` - URL of the server to test
+- `--scenario <scenario>` - Test scenario to run (e.g., "server-initialize". Runs all available scenarios by default
+
 ## Test Results
 
-Results are saved to `results/<scenario>-<timestamp>/`:
+**Client Testing** - Results are saved to `results/<scenario>-<timestamp>/`:
 
 - `checks.json` - Array of conformance check results with pass/fail status
 - `stdout.txt` - Client stdout output
 - `stderr.txt` - Client stderr output
 
+**Server Testing** - Results are saved to `results/server-<scenario>-<timestamp>/`:
+
+- `checks.json` - Array of conformance check results with pass/fail status
+
 ## Example Clients
 
 - `examples/clients/typescript/test1.ts` - Valid MCP client (passes all checks)
 - `examples/clients/typescript/test-broken.ts` - Invalid client missing required fields (fails checks)
 
 ## Available Scenarios
 
+### Client Scenarios
+
 - **initialize** - Tests MCP client initialization handshake
   - Validates protocol version
   - Validates clientInfo (name and version)
   - Validates server response handling
+- **tools-call** - Tests tool invocation
+- **auth/basic-dcr** - Tests OAuth Dynamic Client Registration flow
+- **auth/basic-metadata-var1** - Tests OAuth with authorization metadata
+
+### Server Scenarios
+
+Run `npx @modelcontextprotocol/conformance list --server` to see all available server scenarios, including:
+
+- **server-initialize** - Tests server initialization and capabilities
+- **tools-list** - Tests tool listing endpoint
+- **tools-call-\*** - Various tool invocation scenarios
+- **resources-\*** - Resource management scenarios
+- **prompts-\*** - Prompt management scenarios
 
 ## Architecture
 
@@ -67,8 +121,13 @@ See `src/runner/DESIGN.md` for detailed architecture documentation.
 ### Key Components
 
 - **Runner** (`src/runner/`) - Orchestrates test execution and result generation
-- **Scenarios** (`src/scenarios/`) - Test scenarios with custom server implementations
-- **Checks** (`src/checks.ts`) - Conformance validation functions
+  - `client.ts` - Client testing implementation
+  - `server.ts` - Server testing implementation
+  - `utils.ts` - Shared utilities
+  - `index.ts` - Public API exports
+- **CLI** (`src/index.ts`) - Command-line interface using Commander.js
+- **Scenarios** (`src/scenarios/`) - Test scenarios with expected behaviors
+- **Checks** (`src/checks/`) - Conformance validation functions
 - **Types** (`src/types.ts`) - Shared type definitions
 
 ## Adding New Scenarios

SERVER_REQUIREMENTS.md

@@ -1022,8 +1022,8 @@ MCP Conformance Test Server running on http://localhost:3000
 
 ```bash
 # Single test
-npm run test:server -- --server-url http://localhost:3000/mcp --scenario server-initialize
+npm run start -- server --url http://localhost:3000/mcp --scenario server-initialize
 
 # All tests
-npm run test:server -- --server-url http://localhost:3000/mcp --all
+npm run start -- server --url http://localhost:3000/mcp
 ```

examples/servers/typescript/README.md

@@ -147,8 +147,7 @@ This server implements the MCP Server Conformance Requirements specified in `../
 To run conformance tests against this server:
 
 ```bash
-cd ../../../
-npm run test:server -- --server-url http://localhost:3000/mcp --all
+npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp
 ```
 
 ## Implementation Notes