Merge pull request #28 from SemClone/fix/download-and-scan-json-parsing

fix: Critical bug in download_and_scan_package JSON parsing (v1.5.8)

Author: oscarvalenzuelab

CHANGELOG.md

@@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.5.8] - 2025-01-13
+
+### Fixed & Redesigned
+
+#### Critical Bug + Complete Redesign: download_and_scan_package
+
+**Problem 1 - Tool was completely broken (v1.5.7):**
+The `download_and_scan_package` tool returned JSON parsing errors:
+```
+"metadata_error": "the JSON object must be str, bytes or bytearray, not CompletedProcess"
+"scan_error": "the JSON object must be str, bytes or bytearray, not CompletedProcess"
+```
+
+**Root Cause:**
+The `_run_tool()` helper returns `subprocess.CompletedProcess` objects, but the code tried to parse them directly as JSON instead of using `.stdout`.
+
+**Problem 2 - Incorrect workflow (v1.5.7):**
+Original implementation tried to use `upmex` and `osslili` with PURLs directly, but these tools require local file paths.
+
+**NEW IMPLEMENTATION - Correct Multi-Method Workflow:**
+
+**Workflow (tries methods in order until sufficient data is collected):**
+1. **Primary**: Use `purl2notices` to download and analyze (fastest, most comprehensive)
+2. **Deep scan**: If incomplete, use `purl2src` to get download URL → download artifact → run `osslili` for deep license scanning + `upmex` for metadata
+3. **Online fallback**: If still incomplete, use `upmex --api clearlydefined` for online metadata
+
+**New Dependencies:**
+- Added `purl2src>=1.2.3` to translate PURLs to download URLs for Step 2
+
+**Changes:**
+```python
+# OLD (v1.5.7) - Broken
+upmex_result = _run_tool("upmex", [purl])  # ❌ upmex needs file path, not PURL
+metadata = json.loads(upmex_result)  # ❌ CompletedProcess is not JSON
+
+# NEW (v1.5.8) - Correct multi-method workflow
+# Step 1: Try purl2notices (downloads internally, extracts from cache file)
+purl2notices_result = _run_tool("purl2notices", ["-i", purl, "--cache", temp_cache])
+cache_data = json.loads(open(temp_cache).read())  # ✅ Read cache file
+
+# Step 2: If incomplete, get download URL and download artifact
+purl2src_result = _run_tool("purl2src", [purl, "--format", "json"])
+download_url = json.loads(purl2src_result.stdout)[0]["download_url"]
+urllib.request.urlretrieve(download_url, local_file)  # Download artifact
+osslili_result = _run_tool("osslili", [local_file])  # ✅ Run on local file
+upmex_result = _run_tool("upmex", ["extract", local_file])  # ✅ Run on local file
+
+# Step 3: If still incomplete, use online APIs
+upmex_online = _run_tool("upmex", ["extract", temp_file, "--api", "clearlydefined"])
+```
+
+**Impact:**
+- ✅ Tool now works correctly with proper multi-method fallback
+- ✅ Uses the correct workflow: purl2notices → download+osslili+upmex → online APIs
+- ✅ Returns `method_used` field showing which method succeeded
+- ✅ Proper error handling with `methods_attempted` tracking
+- ✅ JSON parsing fixed (uses `.stdout` correctly)
+
+**Thanks:**
+User feedback identified the bugs and clarified the correct workflow design
+
 ## [1.5.7] - 2025-01-13
 
 ### Added

README.md

@@ -14,23 +14,34 @@ mcp-semclone integrates the complete SEMCL.ONE toolchain to provide LLMs with po
 - **Binary Analysis**: Analyze compiled binaries (APK, EXE, DLL, SO, JAR) for OSS components and licenses
 - **Vulnerability Assessment**: Query multiple vulnerability databases for security issues
 - **Package Discovery**: Identify packages from source code and generate PURLs
-- **SBOM Generation**: Create Software Bill of Materials in SPDX/CycloneDX formats
+- **SBOM Generation**: Create Software Bill of Materials in CycloneDX format
 - **Policy Validation**: Check license compatibility and organizational compliance
 
 ## Features
 
 ### Tools
+
+**Analysis & Scanning:**
 - `scan_directory` - Comprehensive directory scanning for packages, licenses, and vulnerabilities
 - `scan_binary` - Analyze compiled binaries (APK, EXE, DLL, SO, JAR) for OSS components
 - `check_package` - Check specific packages for licenses and vulnerabilities
+- `download_and_scan_package` - Download package source from registries and perform deep license/copyright scanning
+
+**Legal Notices & Documentation:**
+- `generate_legal_notices` - Generate legal notices by scanning source code directly (fast, recommended)
+- `generate_legal_notices_from_purls` - Generate legal notices from PURL list (downloads from registries)
+- `generate_sbom` - Generate Software Bill of Materials in CycloneDX format
+
+**License & Policy Validation:**
 - `validate_policy` - Validate licenses against organizational policies
 - `validate_license_list` - Quick license safety validation for distribution types
 - `get_license_obligations` - Get detailed compliance requirements for licenses
 - `check_license_compatibility` - Check if two licenses can be mixed
 - `get_license_details` - Get comprehensive license information including full text
 - `analyze_commercial_risk` - Assess commercial distribution risks
+
+**Complete Workflows:**
 - `run_compliance_check` - Universal one-shot compliance workflow for any project type
-- `generate_sbom` - Generate SBOM for projects
 
 ### Resources
 - `license_database` - Access license compatibility information
@@ -225,7 +236,7 @@ asyncio.run(main())
 1. **Scan project structure** to identify components
 2. **Extract metadata** for each component
 3. **Detect licenses** and copyright information
-4. **Format as SBOM** (SPDX or CycloneDX)
+4. **Format as SBOM** (CycloneDX 1.4 JSON)
 5. **Validate completeness** of the SBOM
 
 ## Architecture

examples/strands-agent-ollama/requirements.txt

@@ -2,7 +2,7 @@
 # Python 3.10+ required
 
 # MCP server with SEMCL.ONE compliance tools
-mcp-semclone>=1.5.7
+mcp-semclone>=1.5.8
 
 # MCP SDK for connecting to MCP servers
 mcp>=1.0.0

guides/IDE_INTEGRATION_GUIDE.md

@@ -67,17 +67,19 @@ The SEMCL.ONE MCP server works with any IDE that supports the Model Context Prot
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }
@@ -95,17 +97,19 @@ The SEMCL.ONE MCP server works with any IDE that supports the Model Context Prot
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }
@@ -201,17 +205,19 @@ Kiro is Amazon's new agentic AI IDE with native MCP support.
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }
@@ -229,17 +235,19 @@ Kiro is Amazon's new agentic AI IDE with native MCP support.
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }
@@ -283,9 +291,9 @@ Kiro is Amazon's new agentic AI IDE with native MCP support.
 The `autoApprove` field allows these tools to run without prompting the user:
 
 - **License Analysis**: `get_license_details`, `get_license_obligations`, `check_license_compatibility`
-- **Package Scanning**: `scan_directory`, `check_package`, `scan_binary`
+- **Package Scanning**: `scan_directory`, `scan_binary`, `check_package`, `download_and_scan_package`
 - **Policy & Risk**: `validate_policy`, `analyze_commercial_risk`, `validate_license_list`
-- **Documentation**: `generate_legal_notices`, `generate_sbom`
+- **Documentation**: `generate_legal_notices`, `generate_legal_notices_from_purls`, `generate_sbom`
 - **Complete Workflow**: `run_compliance_check` (one-shot compliance check for any project type)
 
 **Note**: Only include tools you trust to run automatically. You can remove sensitive tools if needed.
@@ -316,17 +324,19 @@ Cline is a popular AI coding extension for VS Code with native MCP support.
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }
@@ -344,17 +354,19 @@ Cline is a popular AI coding extension for VS Code with native MCP support.
          "disabled": false,
          "autoApprove": [
            "scan_directory",
+           "scan_binary",
            "check_package",
+           "download_and_scan_package",
+           "generate_legal_notices",
+           "generate_legal_notices_from_purls",
+           "generate_sbom",
            "validate_policy",
+           "validate_license_list",
            "get_license_obligations",
            "check_license_compatibility",
            "get_license_details",
            "analyze_commercial_risk",
-           "validate_license_list",
-           "run_compliance_check",
-           "generate_legal_notices",
-           "generate_sbom",
-           "scan_binary"
+           "run_compliance_check"
          ]
        }
      }

mcp_semclone/__init__.py

@@ -1,5 +1,5 @@
 """MCP SEMCL.ONE - Model Context Protocol server for OSS compliance."""
 
-__version__ = "1.5.7"
+__version__ = "1.5.8"
 __author__ = "Oscar Valenzuela B."
 __email__ = "oscar.valenzuela.b@gmail.com"