feat(jobs): Add cancel_dataproc_job tool with comprehensive job cancellation support (#34)

🎯 Overview

This PR adds a new **`cancel_dataproc_job`** MCP tool to provide emergency job cancellation capabilities for the Dataproc MCP Server. This addresses a critical need for users to stop runaway or long-running jobs to control costs and manage resources.

✨ Key Features:
- Emergency job cancellation with minimal parameters (only jobId required)
- Intelligent state handling (only attempts cancellation for PENDING/RUNNING jobs)
- Comprehensive error handling with clear messages for all job states
- Job tracking integration and knowledge base indexing
- Enhanced documentation with 17 total tools (was 16)

🧪 Testing:
- 15 comprehensive unit tests covering all scenarios
- All 26 unit tests passing
- Golden Command validation: `npm run pre-push` passed
- Build, lint, format, type-check: All passed
- Security audit: Clean
- Documentation: All 209 links validated

💡 Use Cases:
- Emergency cost control: Stop expensive runaway jobs
- Pipeline management: Cancel dependent jobs when upstream fails
- Development workflows: Quick cancellation during testing
- Resource management: Free up cluster resources

Author: dipseth

.gitignore

@@ -131,3 +131,4 @@ dataproc-ops-test-report.json
 enhanced-prompt-demo.js
 test-spark-job.py
 verification-report.json
+state/dataproc-state.json

README.md

@@ -87,7 +87,7 @@ npx @dipseth/dataproc-mcp-server@latest
 ## ✨ Features
 
 ### 🎯 **Core Capabilities**
-- **21 Production-Ready MCP Tools** - Complete Dataproc management suite
+- **22 Production-Ready MCP Tools** - Complete Dataproc management suite
 - **🧠 Knowledge Base Semantic Search** - Natural language queries with optional Qdrant integration
 - **🚀 Response Optimization** - 60-96% token reduction with Qdrant storage
 - **🔄 Generic Type Conversion System** - Automatic, type-safe data transformations
@@ -132,7 +132,7 @@ npx @dipseth/dataproc-mcp-server@latest
 - **Troubleshooting Guides** - Common issues and solutions
 - **IDE Integration** - TypeScript support
 
-## 🛠️ Complete MCP Tools Suite (21 Tools)
+## 🛠️ Complete MCP Tools Suite (22 Tools)
 
 > **🔄 Enhanced with Generic Type Conversion**: All tools now benefit from automatic, type-safe data transformations with intelligent compression and field mapping.
 
@@ -148,11 +148,12 @@ npx @dipseth/dataproc-mcp-server@latest
 | `delete_cluster` | Delete existing clusters | ✅ Project/region defaults | Safe deletion |
 | `get_zeppelin_url` | Get Zeppelin notebook URL | ✅ Auto-discovery | Web interface access |
 
-### 💼 **Job Management (6 Tools)**
+### 💼 **Job Management (7 Tools)**
 | Tool | Description | Smart Defaults | Key Features |
 |------|-------------|----------------|--------------|
 | `submit_hive_query` | Submit Hive queries to clusters | ✅ 70% fewer params | Async support, timeouts |
 | `submit_dataproc_job` | Submit Spark/PySpark/Presto jobs | ✅ 75% fewer params | Multi-engine support, **Local file staging** |
+| `cancel_dataproc_job` | Cancel running or pending jobs | ✅ JobID only needed | **Emergency cancellation**, cost control |
 | `get_job_status` | Get job execution status | ✅ JobID only needed | Real-time monitoring |
 | `get_job_results` | Get job outputs and results | ✅ Auto-pagination | Result formatting |
 | `get_query_status` | Get Hive query status | ✅ Minimal params | Query tracking |
@@ -216,7 +217,7 @@ my-company-analytics-prod-1234:
 - **[Knowledge Base Semantic Search](https://dipseth.github.io/dataproc-mcp/KNOWLEDGE_BASE_SEMANTIC_SEARCH/)** - Natural language queries and setup
 - **[Generic Type Conversion System](docs/GENERIC_TYPE_CONVERTER.md)** - Architectural design and implementation
 - **[Generic Converter Migration Guide](docs/GENERIC_TYPE_CONVERTER.md)** - Migration from manual conversions
-- **[API Reference](https://dipseth.github.io/dataproc-mcp/api/)** - Complete tool documentation
+- **[API Reference](https://dipseth.github.io/dataproc-mcp/API_REFERENCE/)** - Complete tool documentation
 - **[Configuration Examples](https://dipseth.github.io/dataproc-mcp/CONFIGURATION_EXAMPLES/)** - Real-world configurations
 - **[Security Guide](https://dipseth.github.io/dataproc-mcp/security/)** - Best practices and compliance
 - **[Installation Guide](https://dipseth.github.io/dataproc-mcp/INSTALLATION_GUIDE/)** - Detailed setup instructions

docs/API_REFERENCE.md

@@ -7,13 +7,13 @@ permalink: /API_REFERENCE/
 
 # 📚 API Reference
 
-Complete reference for all 16 Dataproc MCP Server tools with practical examples and usage patterns.
+Complete reference for all 17 Dataproc MCP Server tools with practical examples and usage patterns.
 
 ## Overview
 
-The Dataproc MCP Server provides 16 comprehensive tools organized into four categories:
+The Dataproc MCP Server provides 17 comprehensive tools organized into four categories:
 - **Cluster Management** (6 tools)
-- **Job Execution** (5 tools)
+- **Job Execution** (6 tools)
 - **Profile Management** (3 tools)
 - **Monitoring & Utilities** (2 tools)
 
@@ -542,9 +542,78 @@ Gets the results of a completed Dataproc job.
 }
 ```
 
+### 12. cancel_dataproc_job
+
+Cancels a running or pending Dataproc job with intelligent status handling and job tracking integration.
+
+**Parameters:**
+- `jobId` (string, required): The ID of the Dataproc job to cancel
+- `projectId` (string, optional): GCP project ID (uses defaults if not provided)
+- `region` (string, optional): Dataproc region (uses defaults if not provided)
+- `verbose` (boolean, optional): Return full response without filtering (default: false)
+
+**🛑 CANCELLATION WORKFLOW:**
+- Attempts to cancel jobs in PENDING or RUNNING states
+- Provides informative messages for jobs already in terminal states
+- Updates internal job tracking when cancellation succeeds
+
+**📊 STATUS HANDLING:**
+- **PENDING/RUNNING** → Cancellation attempted
+- **DONE/ERROR/CANCELLED** → Informative message returned
+- **Job not found** → Clear error message
+
+**💡 MONITORING:**
+After cancellation, use `get_job_status("jobId")` to confirm the job reaches CANCELLED state.
+
+**Example:**
+```json
+{
+  "tool": "cancel_dataproc_job",
+  "arguments": {
+    "jobId": "Clean_Places_sub_group_base_1_cleaned_places_13b6ec3f"
+  }
+}
+```
+
+**Successful Cancellation Response:**
+```json
+{
+  "content": [
+    {
+      "type": "text",
+      "text": "🛑 Job Cancellation Status\n\nJob ID: Clean_Places_sub_group_base_1_cleaned_places_13b6ec3f\nStatus: 3\nMessage: Cancellation request sent for job Clean_Places_sub_group_base_1_cleaned_places_13b6ec3f."
+    }
+  ]
+}
+```
+
+**Job Already Completed Response:**
+```json
+{
+  "content": [
+    {
+      "type": "text",
+      "text": "Cannot cancel job Clean_Places_sub_group_base_1_cleaned_places_13b6ec3f in state: 'DONE'; cancellable states: '[PENDING, RUNNING]'"
+    }
+  ]
+}
+```
+
+**Use Cases:**
+- **Emergency Cancellation**: Stop runaway jobs consuming excessive resources
+- **Pipeline Management**: Cancel dependent jobs when upstream processes fail
+- **Cost Control**: Terminate expensive long-running jobs
+- **Development Workflow**: Cancel test jobs during development iterations
+
+**Best Practices:**
+1. **Monitor job status** before and after cancellation attempts
+2. **Use with get_job_status** to verify cancellation completion
+3. **Handle gracefully** when jobs are already in terminal states
+4. **Consider dependencies** before cancelling pipeline jobs
+
 ## Profile Management Tools
 
-### 12. list_profiles
+### 13. list_profiles
 
 Lists available cluster configuration profiles.
 
@@ -573,7 +642,7 @@ Lists available cluster configuration profiles.
 }
 ```
 
-### 13. get_profile
+### 14. get_profile
 
 Gets details for a specific cluster configuration profile.
 
@@ -590,7 +659,7 @@ Gets details for a specific cluster configuration profile.
 }
 ```
 
-### 14. list_tracked_clusters
+### 15. list_tracked_clusters
 
 Lists clusters that were created and tracked by this MCP server.
 
@@ -609,7 +678,7 @@ Lists clusters that were created and tracked by this MCP server.
 
 ## Monitoring & Utilities
 
-### 15. get_zeppelin_url
+### 16. get_zeppelin_url
 
 Gets the Zeppelin notebook URL for a cluster (if enabled).
 
@@ -642,7 +711,7 @@ Gets the Zeppelin notebook URL for a cluster (if enabled).
 }
 ```
 
-### 16. check_active_jobs
+### 17. check_active_jobs
 
 🚀 Quick status check for all active and recent jobs with intelligent response optimization.
 

docs/QUICK_START.md

@@ -191,6 +191,16 @@ Show me all my Dataproc clusters
 Submit a Spark job to process data from gs://my-bucket/data.csv
 ```
 
+### Cancel a Running Job
+```
+Cancel the job with ID "my-long-running-job-12345"
+```
+
+### Monitor Job Status
+```
+Check the status of job "my-job-67890"
+```
+
 ### Try Semantic Search (if Qdrant enabled)
 ```
 Show me clusters with machine learning packages installed

scripts/setup.js

@@ -140,6 +140,7 @@ async function createMCPTemplate() {
         "submit_dataproc_job",
         "get_job_status",
         "get_job_results",
+        "cancel_dataproc_job",
         "get_zeppelin_url"
       ],
       "env": {

src/handlers/cluster-handlers.ts

@@ -5,6 +5,7 @@
 
 import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
 import { logger } from '../utils/logger.js';
+import { deepMerge } from '../utils/object-utils.js';
 import SecurityMiddleware from '../security/middleware.js';
 import {
   StartDataprocClusterSchema,
@@ -767,7 +768,7 @@ export async function handleCreateClusterFromYaml(args: any, deps: HandlerDepend
 
     // Apply overrides if provided
     if (overrides && typeof overrides === 'object') {
-      clusterConfig = { ...clusterConfig, ...overrides };
+      clusterConfig = deepMerge(clusterConfig, overrides);
     }
 
     // Use existing cluster creation logic with properly extracted config
@@ -824,7 +825,7 @@ export async function handleCreateClusterFromProfile(args: any, deps: HandlerDep
 
     // Apply overrides if provided
     if (overrides && typeof overrides === 'object') {
-      clusterConfig = { ...clusterConfig, ...overrides };
+      clusterConfig = deepMerge(clusterConfig, overrides);
     }
 
     // Use existing cluster creation logic

src/handlers/index.ts

@@ -20,6 +20,7 @@ import {
   handleGetQueryResults,
   handleGetJobResults,
   handleCheckActiveJobs,
+  handleCancelDataprocJob,
 } from './job-handlers.js';
 import {
   handleListProfiles,
@@ -97,6 +98,8 @@ export async function handleToolCall(toolName: string, args: any, deps: AllHandl
       return handleGetJobResults(args, deps);
     case 'check_active_jobs':
       return handleCheckActiveJobs(args, deps);
+    case 'cancel_dataproc_job':
+      return handleCancelDataprocJob(args, deps);
 
     // Profile handlers
     case 'list_profiles':