Skip to content

Commit b15427c

Browse files
author
Jeff Bulger
committed
Initial commit
0 parents  commit b15427c

File tree

1,497 files changed

+548149
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,497 files changed

+548149
-0
lines changed

‎.gitignore‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Node and build outputs
2+
node_modules/
3+
gateway/node_modules/
4+
gateway/ui/node_modules/
5+
gateway/ui/dist/
6+
7+
# Logs
8+
*.log
9+
gateway.log
10+
11+
# Python caches/venvs
12+
__pycache__/
13+
*.pyc
14+
servers/venv_*/
15+
16+
# RAG data (user content)
17+
data/rag/uploads/
18+
data/rag/saved_chats/
19+
data/rag/images/
20+
data/rag/indexes.pkl
21+
22+
# Local indexes/cache
23+
data/rag/indexes/
24+
25+
# OS cruft
26+
.DS_Store
27+
Thumbs.db

‎AGENT.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# AGENT.MD: FullStack MCP Hub snapshot (2025-12-15)
2+
3+
## 1) Progress summary
4+
- Gateway + UI stable on :3333 (tools, blocklist, RAG browser, presets, editable descriptions).
5+
- Tool stack ~53 tools across 12 servers: filesystem, shell, Playwright, sqlite, local_rag, websearch, websearch_adv (deep search + advanced extract), scrape, research (wiki/arXiv/Wikimedia images), python_repl (venv), pollinations, coingecko.
6+
- local_rag scoped to `data/rag`; `save_chat` writes raw+summary without overwriting; `list_indexes` added.
7+
- Python REPL owns its venv and `pip_install`; research adds Wikimedia Commons image search.
8+
- Scraper server for fast HTML→text; blocklist persisted (`tool-blocklist.json`) and managed via UI.
9+
- Allowlists: pollinations (generateImageUrl/listImageModels), coingecko (4 tools), websearch_adv (deep search + single-page extract).
10+
11+
---
12+
13+
## 2. Project Vision
14+
15+
To create a **universal, model-agnostic, and extensible tool-use architecture**. This system will allow any AI model (Gemini, OpenAI's GPT, Anthropic's Claude, Grok, etc.) and any custom interface (like the Francine GUI) to seamlessly access a single, powerful, and ever-growing stack of tools.
16+
17+
The core principle is **"Write a tool once, use it from any model."**
18+
19+
## 3. Core Architecture
20+
21+
The system is composed of three primary layers, ensuring maximum separation of concerns and scalability.
22+
23+
```
24+
+----------------+ +----------------+ +----------------+
25+
| Gemini API | | OpenAI API | | Francine GUI |
26+
+----------------+ +----------------+ +----------------+
27+
| | |
28+
+-----------------------+-----------------------+
29+
|
30+
+------------------------------+
31+
| Universal API Gateway | <-- THE KEY TO UNIVERSALITY
32+
| (Google Cloud Run/Function) |
33+
+------------------------------+
34+
| - /gemini/v1/execute | (Gemini Tool Spec)
35+
| - /openai/v1/openapi.json | (OpenAI Plugin Spec)
36+
| - /anthropic/v1/execute | (Anthropic Tool Spec)
37+
| - /francine/v1/mcp | (Custom MCP Spec)
38+
+------------------------------+
39+
|
40+
| (Standardized Internal Request)
41+
v
42+
+--------------------------+
43+
| MCP Hub | (Borrowed from Fran1)
44+
+--------------------------+
45+
| - Tool Registry |
46+
| - Connection Manager |
47+
| - Execution Router |
48+
+--------------------------+
49+
| | |
50+
(stdio/sse) | | |
51+
+-----------------+ +----------------+ +-----------------+
52+
| RAG MCP Server | | Finch MCP Serv | | Playwright MCP |
53+
| (Python) | | (Node.js) | | Server (OSS) |
54+
+-----------------+ +----------------+ +-----------------+
55+
56+
```
57+
58+
### 3.1. Layer 1: Tool Servers (The "Hands")
59+
60+
- **Standard:** Each tool is an independent process that communicates using the **Model Context Protocol (MCP)** over `stdio` (for local tools) or `sse` (for networked tools), just like in `Fran1`.
61+
- **Responsibilities:** A tool server is responsible for one thing only: exposing its capabilities (`tools/list`) and executing them (`tools/call`).
62+
- **Examples:** A server for browsing the web with Playwright, a server for reading local files (our RAG tool), a server for interacting with APIs.
63+
64+
### 3.2. Layer 2: The MCP Hub (The "Brainstem")
65+
66+
- **Logic:** We will adopt the robust logic from your `Fran1` project.
67+
- **Responsibilities:**
68+
1. **Registry:** Reads a `master.json` file to discover all available tool servers.
69+
2. **Connection:** Manages the lifecycle of connections to these tool servers.
70+
3. **Routing:** Provides a single internal endpoint to execute any registered tool by name.
71+
72+
### 3.3. Layer 3: The Universal API Gateway (The "Translator")
73+
74+
- **This is the most critical new component for universal compatibility.** It is a public-facing API that acts as a multi-headed adaptor.
75+
- **Responsibilities:**
76+
1. **Expose Model-Specific Endpoints:** It will have different endpoints that conform to the *exact* tool-use specifications of different AI providers.
77+
2. **Translate and Delegate:** When a request comes in from a specific model (e.g., Gemini), the gateway translates the model-specific request into a standardized call to the **MCP Hub**.
78+
3. **Format and Return:** It receives the result from the MCP Hub and formats it back into the response structure the original AI model expects.
79+
- **Deployment:** This gateway is the perfect candidate for deployment as a serverless **Google Cloud Run** service, integrating it directly with the environment we've set up.
80+
81+
## 4. Development Roadmap
82+
83+
This plan is designed for parallel work. Different agents can tackle different MCP servers simultaneously once the core is in place.
84+
85+
1. **Setup Core Infrastructure:**
86+
- `[✅]` Initialize the `MASTER_MCP` project structure (e.g., `/servers`, `/hub`, `/gateway`).
87+
- `[✅]` Port the `McpHub` logic from `Fran1` into this new project.
88+
- `[✅]` Port the `master.json` registry and create a directory for tool server configurations.
89+
90+
2. **Integrate First Tool (Proof of Concept):**
91+
- `[✅]` Find and integrate an existing open-source MCP-compatible tool (e.g., for Playwright or a similar web browser tool).
92+
- `[✅]` Register it in `master.json` and confirm the `McpHub` can connect to and list its tools.
93+
94+
3. **Build the Local RAG Tool:**
95+
- `[✅]` Create the new `rag_mcp_server.py`.
96+
- `[✅]` Implement `list_directory`, `read_file`, and `search_files` functions.
97+
- `[✅]` Implement `create_index` and `search_index` with **named collection support**.
98+
- `[✅]` Register it with the `McpHub`.
99+
100+
4. **Build the Universal Gateway (Gemini First):**
101+
- `[✅]` Create the initial API Gateway project (e.g., using Node.js/Express or Python/FastAPI).
102+
- `[✅]` Implement the `/gemini/v1/execute` endpoint. This endpoint will accept a request body matching the Gemini API's `FunctionCall` format.
103+
- **Status:** Fully functional and tested.
104+
105+
5. **Expand and Integrate:**
106+
- `[✅]` Integrate **Shell MCP** (`mcp-server-commands`) for command-line access.
107+
- `[✅]` Integrate **Playwright MCP** (`mcp-server-playwright`) for browser automation.
108+
- `[✅]` Integrate **SQLite MCP** (`mcp-server-sqlite`) for persistent memory.
109+
- `[✅]` Initialize `tool_runs` logging table in SQLite.
110+
- `[✅]` Organize documentation into `~/.master_mcp/data/raw/mcp_docs` and create RAG index.
111+
112+
---
113+
114+
## 5. System Status (Live)
115+
116+
Approx tools: 53
117+
118+
| Server | Transport | Tools | Notes |
119+
| :--- | :--- | :--- | :--- |
120+
| filesystem | stdio (npx) | 14 | sandbox `/home/jeff` |
121+
| local_rag | stdio (python) | 6 | save_chat, list_indexes; sandbox `data/rag` |
122+
| shell | stdio (node) | 1 | commands |
123+
| playwright | stdio (npx) | 10 | browser automation |
124+
| sqlite | stdio (python venv) | 6 | db ops |
125+
| websearch | stdio (node) | 1 | fast DDG |
126+
| websearch_adv | stdio (node) | 2 | deep search + single-page extract (allowlisted) |
127+
| scrape | stdio (node) | 1 | fast HTML→text |
128+
| research | stdio (python) | 3 | wiki, arxiv, Commons images |
129+
| python_repl | stdio (python) | 3 | exec/reset/pip (venv) |
130+
| pollinations | stdio (npx) | 2 | image URL + list models |
131+
| coingecko | sse | 4 | allowlisted to search/price/markets/range |

‎README.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# FullStack MCP Hub
2+
3+
This repo bundles the MCP hub/gateway plus a browser UI for discovering and running MCP tools.
4+
5+
## What’s here
6+
- `hub/`: connects to configured MCP servers, lists tools, executes calls.
7+
- `gateway/`: HTTP/SSE surface for the hub (`/tools`, `/gemini/v1/execute`, etc.) and serves the built UI.
8+
- `gateway/ui/`: React UI (Vite) for browsing tools, editing descriptions, adding servers, running calls, saving presets, and managing blocked tools.
9+
- `tool-registry/master.json`: MCP server registry (stdio/SSE).
10+
`tool-registry/tool-overrides.json`: description overrides.
11+
`tool-registry/tool-blocklist.json`: persisted blocklist (managed via UI Blocked tab).
12+
- `servers/`: bundled servers (local_rag, sqlite, python_repl with venv, research incl. Wikipedia/ArXiv/Wikimedia images, scrape, pollinations, coingecko, advanced web search clone).
13+
- Key shipped servers:
14+
- `local_rag`: chunked search, fuzzy/filters, `save_chat`, `save_image`.
15+
- `sqlite`
16+
- `python_repl` (with its own venv; use `pip_install` to add packages like pandas/numpy without touching system Python)
17+
- `research` (Wikipedia, ArXiv, Wikimedia Commons images)
18+
- `scrape` (HTML→text)
19+
- `pollinations` (image URL + models)
20+
- `coingecko` (SSE, curated 4-tool allowlist: search, price, markets, range chart)
21+
- `websearch` (fast DDG)
22+
- `websearch_adv` (deep multi-engine search + single-page extract, local clone)
23+
- `playwright`, `shell`, `filesystem`
24+
25+
## Why (intent)
26+
- Make MCP approachable: start the gateway, open the UI, add servers with a guided form, test, and go.
27+
- Serve ops/dev workflows (LLM ops / “llmOPS” vibes) with a single pane to discover, run, and tune tools.
28+
- No-required-API-key defaults: ships with stdio-friendly examples like Playwright for browser/search.
29+
30+
## Contact / collab
31+
- Built by Jeff Bulger — https://jeffbulger.dev | [email protected] | GitHub: https://github.com/jbulger82
32+
- Looking for collaborators who want to build/extend open LLM OPS tooling (MCP servers, local-first flows, RAG, search, automation).
33+
34+
## Prereqs
35+
- Node 18+ (gateway/UI), npm.
36+
- The MCP servers you want to run (stdio commands or SSE endpoints).
37+
38+
## Start everything
39+
```bash
40+
export MCP_ROOT=/path/to/Fullstack_MCP_hub # set to your clone path
41+
cd gateway
42+
npm start
43+
# UI served at http://localhost:3333
44+
```
45+
46+
First-time setup (deps + UI build):
47+
```bash
48+
cd gateway && npm install
49+
cd ui && npm install && npm run build
50+
```
51+
52+
One-shot setup helper (does the installs/builds/playwright browsers):
53+
```bash
54+
export MCP_ROOT=/path/to/Fullstack_MCP_hub # set to your clone path
55+
./setup.sh
56+
```
57+
58+
Quick start after setup:
59+
```bash
60+
export MCP_ROOT=/path/to/Fullstack_MCP_hub
61+
./start.sh
62+
```
63+
64+
If port 3333 is busy: `lsof -i :3333` then `kill <pid>` and retry.
65+
66+
## Using the UI
67+
Open `http://localhost:3333`.
68+
69+
- **Add MCP server (guided)**
70+
- Click “Open form” under Servers (left pane).
71+
- Choose transport:
72+
- `stdio`: fill Command (e.g., `npx`), Args (e.g., `-y @automatalabs/mcp-server-playwright`), optional CWD.
73+
- `sse`: fill full SSE URL (e.g., `http://localhost:4000/sse`).
74+
- Click **Test connection** (runs a lightweight connect + tools/list).
75+
- On success, click **Add server** to persist to `tool-registry/master.json` and connect live. Tools appear in the list.
76+
77+
- **Block/restore tools**
78+
- Tools tab: “Block tool” hides the selected tool (persisted to `tool-blocklist.json`).
79+
- Blocked tab: view/restore blocked tools.
80+
81+
- **Browse & run tools**
82+
- Select a tool in the left list; right pane shows description + input schema.
83+
- Enter JSON payload (or keep `{}`) and click **Run tool**.
84+
- Responses show in “Result”; status chip shows timing.
85+
86+
- **Presets**
87+
- Save current payload (“Save current”); apply or delete per tool. Stored locally in browser storage.
88+
89+
- **Edit descriptions**
90+
- Tool detail pane has an editable description. **Save** writes to `tool-registry/tool-overrides.json`. **Restore default** removes the override.
91+
92+
- **RAG tab**
93+
- Browse `data/rag/uploads`, `saved_chats`, `indexes`; drag/drop “ADD FILE”; search filenames/paths; delete with confirm. Uploads auto-refresh the `uploads` index.
94+
- Indexing: text files are chunked (~500 words + overlap); indexes persist to disk (`indexes.pkl`) so they survive restarts. `search_index` supports `fuzzy`, `path_contains`, `tag` (from a `#tags:` line), and mtime filters. Results return matching chunks (with file/chunk info).
95+
- Save tools: `save_chat` writes raw+summary to `data/rag/saved_chats` (no overwrites); `save_image` writes base64 images to `data/rag/images`.
96+
- Profiles: a starter template lives at `data/rag/profile_template/profile_public.md`. Copy/rename to your own folder (e.g., `data/rag/profile_me/profile_public.md`) and edit with your details; use `#tags:` if you want tag filtering.
97+
98+
## Common stdio examples (no API key)
99+
- Playwright (browser automation/search/screenshot):
100+
- Command: `npx`
101+
- Args: `-y @automatalabs/mcp-server-playwright`
102+
- CWD: `servers`
103+
- DuckDuckGo Websearch (included):
104+
- Command: `node`
105+
- Args: `servers/websearch/server.js`
106+
- CWD: repo root (`MASTER_MCP`)
107+
- Python REPL (persistent session + its own venv):
108+
- Command: `python3`
109+
- Args: `python_repl_mcp.py`
110+
- CWD: `servers`
111+
- Research (Wikipedia, ArXiv, Wikimedia Commons images):
112+
- Command: `python3`
113+
- Args: `research_mcp_server.py`
114+
- CWD: `servers`
115+
- Scraper (fast HTML fetch/clean):
116+
- Command: `node`
117+
- Args: `scrape_mcp_server.js`
118+
- CWD: `servers`
119+
- Advanced web search (multi-engine + extraction; cloned locally):
120+
- Command: `node`
121+
- Args: `dist/index.js`
122+
- CWD: `servers/web-search-mcp`
123+
- Coingecko (SSE):
124+
- Transport: `sse`
125+
- URL: `https://mcp.api.coingecko.com/sse`
126+
127+
## Project scripts
128+
Gateway (`/gateway`):
129+
- `npm start` – start gateway + serve built UI.
130+
- `npm run build:ui` – build UI assets into `gateway/ui/dist`.
131+
- `npm run dev:ui` – UI dev server with proxy to gateway.
132+
133+
UI (`/gateway/ui`):
134+
- `npm run dev` – Vite dev server (proxies to 3333).
135+
- `npm run build` – production build.
136+
137+
## Paths of interest
138+
- Gateway entry: `gateway/server.js`
139+
- Hub logic: `hub/McpHub.js`
140+
- UI entry: `gateway/ui/src/App.jsx`
141+
- Registry: `tool-registry/master.json`
142+
- Description overrides: `tool-registry/tool-overrides.json`
143+
- Blocklist: `tool-registry/tool-blocklist.json`
144+
- RAG data: `data/rag/` (uploads, saved_chats, indexes)
145+
- Includes `data/rag/images` for saved screenshots via `local_rag__save_image`.
146+
- Advanced search clone: `servers/web-search-mcp/`
147+
148+
## Troubleshooting
149+
- Port in use: `lsof -i :3333` → `kill <pid>` → restart.
150+
- Test fails when adding server: check command/args or SSE URL; rerun **Test connection** to see error.
151+
- No tools after add: refresh tools (left pane Refresh) or restart gateway after fixing registry.
152+
- If a server keeps failing: check paths/CWD, block noisy tools via Blocked tab, then reconnect.
153+
- Connecting from hosted UIs (e.g., ChatGPT dev mode): if the local URL is marked unsafe, tunnel with ngrok (e.g., ngrok v3/stable 3.34.1) to expose `http://localhost:3333` over HTTPS, then use the ngrok URL.

‎data/rag/profile_template/profile_public.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Profile (Public) – Starter Template
2+
3+
Name: <your name>
4+
Handle: <optional handle>
5+
Location: <city/state or region>
6+
7+
Personality & vibe:
8+
- <how you like answers> (e.g., direct/concise or detailed/friendly)
9+
- <comfort with tone> (e.g., casual/professional)
10+
- <boundaries> (e.g., no NSFW, no long-winded filler)
11+
12+
Preferences:
13+
- <tools you prefer to use> (e.g., local LLMs, certain search tools)
14+
- <formatting you like> (e.g., bullet summaries, include timestamps)
15+
- <other preferences> (e.g., keep responses short/long)
16+
17+
Current focus/projects:
18+
- <project 1 + short note>
19+
- <project 2 + short note>
20+
21+
System basics (optional):
22+
- OS / environment:
23+
- Notable paths:
24+
- Anything an agent should know:
25+
26+
Usage notes for agents:
27+
- Treat this file as stable profile context for any model.
28+
- If the user says “remember this” or “add to profile,” append here or to a notes file under `data/rag/`.
29+
30+
#tags: profile, user-context

‎gateway/index.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
// Convenience entrypoint: keeps `node .` or `npm start` aligned with the real server.
2+
import './server.js';

0 commit comments

Comments
 (0)