feat: add JSON extension support with comprehensive tests, new APIs and caching behavior for list tool network requests

Author: Gioee

API.md

@@ -27,23 +27,33 @@ sqlite> SELECT mcp_version();
 
 ---
 
-### `mcp_connect(server_url, [legacy_sse], [headers_json])`
+### `mcp_connect(server_url, [legacy_sse], [headers_json], [use_json_ext])`
 
-Connects to an MCP server using either Streamable HTTP (default) or SSE transport, with optional custom HTTP headers.
+Connects to an MCP server using either Streamable HTTP (default) or SSE transport, with optional custom HTTP headers and JSON extension mode.
 
 **Syntax:**
 ```sql
 SELECT mcp_connect(server_url);
 SELECT mcp_connect(server_url, legacy_sse);
 SELECT mcp_connect(server_url, legacy_sse, headers_json);
+SELECT mcp_connect(server_url, legacy_sse, headers_json, use_json_ext);
 ```
 
 **Parameters:**
 - `server_url` (TEXT) - URL of the MCP server (e.g., "http://localhost:8000/mcp")
 - `legacy_sse` (INTEGER, optional) - 1 to use SSE transport (legacy), 0 for Streamable HTTP (default)
 - `headers_json` (TEXT, optional) - JSON string with custom HTTP headers (e.g., `{"Authorization": "Bearer token"}`)
+- `use_json_ext` (INTEGER, optional) - 1 to enable JSON extension mode (returns results as JSON type), 0 for regular text mode (default)
 
-**Returns:** `TEXT` - JSON object with connection status
+**Returns:**
+
+SQLite return codes from `sqlite3_step()`:
+- **On success**: `SQLITE_ROW`
+  - Regular mode (`use_json_ext=0`): Row contains JSON object with connection status
+  - JSON mode (`use_json_ext=1`): Row contains NULL (silent success)
+- **On failure**: `SQLITE_ERROR` - Error message set via `sqlite3_result_error()`
+
+**Note:** SQLite scalar functions always return exactly one row on success. They cannot return `SQLITE_DONE` (zero rows).
 
 **Examples:**
 ```sql
@@ -59,6 +69,17 @@ SELECT mcp_connect(
   0,
   '{"Authorization": "Bearer ghp_your_token", "X-MCP-Readonly": "true"}'
 );
+
+-- Connect with JSON extension mode enabled
+SELECT mcp_connect('http://localhost:8000/mcp', 0, NULL, 1);
+
+-- Connect with both custom headers and JSON extension mode
+SELECT mcp_connect(
+  'http://localhost:8000/mcp',
+  0,
+  '{"Authorization": "Bearer token"}',
+  1
+);
 ```
 
 **Response:**
@@ -169,6 +190,198 @@ SELECT mcp_call_tool('test', '{}');
 
 ---
 
+## Virtual Tables
+
+The extension provides virtual tables that automatically parse MCP responses into structured rows. These are ideal for SQL queries that need to process multiple tools or results.
+
+### `mcp_tools_table`
+
+A virtual table that returns each tool as a row with structured columns.
+
+**Syntax:**
+```sql
+SELECT * FROM mcp_tools_table;
+SELECT name, description FROM mcp_tools_table;
+SELECT * FROM mcp_tools_table WHERE name LIKE 'airbnb%';
+```
+
+**Columns:**
+- `name` (TEXT) - Unique identifier for the tool
+- `title` (TEXT) - Optional human-readable name of the tool
+- `description` (TEXT) - Human-readable description of functionality
+- `inputSchema` (TEXT) - JSON Schema defining expected parameters
+- `outputSchema` (TEXT) - Optional JSON Schema defining expected output structure
+- `annotations` (TEXT) - Optional properties describing tool behavior
+
+**Example:**
+```sql
+-- Connect to server
+SELECT mcp_connect('http://localhost:8000/mcp');
+
+-- Query tools as rows
+SELECT name, description
+FROM mcp_tools_table
+WHERE name LIKE 'airbnb%';
+```
+
+**Result:**
+```
+name              description
+----------------  ------------------------------------------
+airbnb_search     Search for Airbnb listings with filters
+airbnb_details    Get details for a specific listing
+```
+
+---
+
+### `mcp_call_tool_table`
+
+A virtual table that extracts text results from tool calls. Returns one row for each `type="text"` content item in the result.
+
+**Syntax:**
+```sql
+SELECT text FROM mcp_call_tool_table
+WHERE tool_name = '<tool>' AND arguments = '<json>';
+```
+
+**Columns:**
+- `tool_name` (TEXT, hidden) - Name of tool to call (required in WHERE clause)
+- `arguments` (TEXT, hidden) - JSON arguments for the tool (required in WHERE clause)
+- `text` (TEXT) - Text content from each result item
+
+**Example:**
+```sql
+-- Call tool and get text results as rows
+SELECT text FROM mcp_call_tool_table
+WHERE tool_name = 'airbnb_search'
+AND arguments = '{"location": "Rome", "maxPrice": 100}';
+```
+
+**Result:**
+```
+text
+----------------------------------------------------
+Found 5 listings in Rome under $100
+Listing 1: Cozy Apartment - $85/night
+Listing 2: Historic Studio - $95/night
+```
+
+**Important Notes:**
+- The `tool_name` and `arguments` columns are "hidden" parameters that must be provided via the WHERE clause
+- This table automatically extracts only `type="text"` results from the MCP response
+- Each text item in the response becomes a separate row
+
+---
+
+## Function Variants
+
+The extension provides multiple ways to access MCP functionality:
+
+### Scalar Functions (Return JSON strings)
+
+- `mcp_list_tools()` - Returns JSON string of all tools
+- `mcp_call_tool(tool_name, arguments)` - Returns JSON string of tool result
+- `mcp_tools_json()` - Alias for `mcp_list_tools()`
+- `mcp_call_tool_json(tool_name, arguments)` - Alias for `mcp_call_tool()`
+
+**Behavior:**
+- Returns the complete JSON response from MCP
+- When JSON extension mode is enabled (`use_json_ext=1`), results are marked with JSON subtype for seamless use with SQLite JSON functions
+- When JSON extension mode is disabled, results are returned as plain text
+
+Use these when you need the raw JSON response or want to process with `json_extract()`.
+
+### Virtual Tables (Return structured rows)
+
+- `mcp_tools_table` - Returns tools as rows with named columns
+- `mcp_call_tool_table` - Returns text results as rows
+
+**Behavior:**
+- Always extract and parse the JSON response using SQLite's `json_each()` function
+- Return structured rows regardless of JSON extension mode setting
+- `mcp_tools_table` extracts the `$.tools` array and returns each tool as a row
+- `mcp_call_tool_table` extracts the `$.result.content` array and returns each `type="text"` item as a row
+
+Use these when you need to process multiple items, filter with WHERE clauses, or join with other tables.
+
+**Example comparing approaches:**
+
+```sql
+-- Scalar function approach
+SELECT json_extract(value, '$.name')
+FROM json_each((SELECT mcp_list_tools()), '$.tools')
+WHERE json_extract(value, '$.name') LIKE 'airbnb%';
+
+-- Virtual table approach (simpler)
+SELECT name
+FROM mcp_tools_table
+WHERE name LIKE 'airbnb%';
+```
+
+---
+
+## JSON Extension Mode
+
+### Benefits
+
+- **Direct JSON Operations**: Use `json_extract()`, `json_each()`, `json_array_length()` and other SQLite JSON functions directly on results
+- **Type Safety**: SQLite recognizes results as JSON type, not plain text
+- **Cleaner Queries**: No need to wrap results in `json()` function calls
+- **Better Performance**: SQLite can optimize JSON operations when the type is known
+
+### Usage Examples
+
+**Without JSON Extension Mode (default):**
+```sql
+-- Connect in regular text mode
+SELECT mcp_connect('http://localhost:8000/mcp', 0, NULL, 0);
+
+-- Need to explicitly parse as JSON
+SELECT json_extract(json(mcp_list_tools()), '$.tools[0].name');
+```
+
+**With JSON Extension Mode:**
+```sql
+-- Connect with JSON extension mode enabled
+SELECT mcp_connect('http://localhost:8000/mcp', 0, NULL, 1);
+
+-- Results are automatically recognized as JSON
+SELECT json_extract(mcp_list_tools(), '$.tools[0].name');
+
+-- Extract connection status
+SELECT json_extract(
+  mcp_connect('http://localhost:8000/mcp', 0, NULL, 1),
+  '$.status'
+) as status;
+
+-- List all available tools using json_each
+SELECT
+  json_extract(value, '$.name') as tool_name,
+  json_extract(value, '$.description') as description
+FROM json_each((SELECT mcp_list_tools()), '$.tools');
+
+-- Extract tool result content
+SELECT json_extract(
+  mcp_call_tool('airbnb_search', '{"location": "Rome"}'),
+  '$.result.content[0].text'
+) as result_text;
+```
+
+### When to Use JSON Extension Mode
+
+**Use JSON extension mode when:**
+- You need to extract specific fields from results using `json_extract()`
+- You want to iterate over arrays in results using `json_each()`
+- You're building queries that process JSON results
+- You want type safety for JSON operations
+
+**Use regular text mode when:**
+- You just need the raw JSON string
+- You're displaying results to users without processing
+- You're integrating with systems that expect text output
+
+---
+
 ## Transport Protocols
 
 The extension supports two MCP transport protocols:

README.md

@@ -8,9 +8,7 @@ A SQLite extension that integrates the [Model Context Protocol (MCP)](https://mo
 
 #### Pre-built Binaries
 
-Download for your platform: **macOS**, **Linux**, **Windows**, **Android**, and **iOS**
-
-Or build from source (see [Building from Source](#building-from-source)).
+Download for your platform: **macOS**, **Linux**, **Windows**, **Android**, and **iOS**.
 
 ### Basic Usage
 
@@ -88,57 +86,7 @@ Server-Sent Events for compatibility with older servers.
 SELECT mcp_connect('http://localhost:8931/sse', 1);
 ```
 
-## 🚦 Usage with Different Languages
-
-### Python
-
-```python
-import sqlite3
-
-conn = sqlite3.connect('data.db')
-conn.enable_load_extension(True)
-conn.load_extension('./dist/mcp')
-
-cursor = conn.cursor()
-
-# Connect to MCP server
-cursor.execute("SELECT mcp_connect('http://localhost:8000/mcp')")
-
-# List tools
-result = cursor.execute("SELECT mcp_list_tools()").fetchone()[0]
-tools = json.loads(result)
-print(f"Available tools: {len(tools['tools'])}")
-
-# Call a tool
-cursor.execute("""
-    SELECT mcp_call_tool('airbnb_search',
-                         '{"location": "Paris", "maxPrice": 150}')
-""")
-result = cursor.fetchone()[0]
-print(result)
-```
-
-### Node.js
-
-```javascript
-const Database = require('better-sqlite3');
-const db = new Database(':memory:');
-
-db.loadExtension('./dist/mcp');
-
-// Connect to MCP server
-const result = db.prepare("SELECT mcp_connect('http://localhost:8000/mcp')").get();
-console.log(result);
-
-// Call tool
-const listings = db.prepare(`
-    SELECT mcp_call_tool('airbnb_search',
-                         '{"location": "Tokyo", "adults": 2}')
-`).get();
-console.log(listings);
-```
-
-### C
+## 🚦 Quick Usage Example
 
 ```c
 #include <sqlite3.h>
@@ -174,6 +122,8 @@ int main() {
 }
 ```
 
+See [USAGE.md](USAGE.md) for complete usage examples.
+
 ## 🤝 Contributing
 
 Contributions are welcome! Please feel free to submit a Pull Request.
@@ -184,8 +134,8 @@ MIT License - See [LICENSE](LICENSE) for details
 
 ## 🔗 Related Projects
 
-- [sqlite-agent](https://github.com/sqlitecloud/sqlite-agent) - A powerful AI agent for SQLite
-- [sqlite-ai](https://github.com/sqlitecloud/sqlite-ai) - LLM integration for SQLite
-- [sqlite-vector](https://github.com/sqlitecloud/sqlite-vector) - Vector search extension
-- [sqlite-sync](https://github.com/sqlitecloud/sqlite-sync) - Cloud synchronization
-- [sqlite-js](https://github.com/sqlitecloud/sqlite-js) - JavaScript engine integration
+- [sqlite-agent](https://github.com/sqliteai/sqlite-agent) - A powerful AI agent for SQLite
+- [sqlite-ai](https://github.com/sqliteai/sqlite-ai) - LLM integration for SQLite
+- [sqlite-vector](https://github.com/sqliteai/sqlite-vector) - Vector search extension
+- [sqlite-sync](https://github.com/sqliteai/sqlite-sync) - Cloud synchronization
+- [sqlite-js](https://github.com/sqliteai/sqlite-js) - JavaScript engine integration