LLM non-JSON output causes real issues to be dropped and reported as “no issues”

[IamYJLee]

Environment

llm_provider:
  name: "ollama"
  model: "llama3.1:8b"
  base_url: "http://localhost:11434/v1"
  code_embedding_model: "all-minilm:22m"
  docs_embedding_model: "all-minilm:22m"

Problem

The LLM is required to return strict JSON, but sometimes returns explanatory text mixed with JSON.
When JSON parsing fails, the system silently falls back to treating the result as “no issues.”
As a result, even when the LLM clearly reports issues in the logs, those issues are not written to the final JSON results file.

Example Log

$ uv run metis --codebase-path ./test_code --non-interactive --command "review_code" -v --log-level DEBUG 2>&1 | tee review.log

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2026-01-14 15:09:36,741 - INFO - Loading default metis.yaml
2026-01-14 15:09:37,014 - WARNING - Langchain Ollama integration requires an non-empty api_key, using a default.
2026-01-14 15:09:37,320 - INFO - Loading default plugins.yaml
2026-01-14 15:09:37,344 - DEBUG - Loaded plugin from entry point: c -> <class 'metis.plugins.c_plugin.CPlugin'>
2026-01-14 15:09:37,344 - DEBUG - Loaded plugin from entry point: cpp -> <class 'metis.plugins.cpp_plugin.CppPlugin'>
2026-01-14 15:09:37,344 - DEBUG - Loaded plugin from entry point: go -> <class 'metis.plugins.go_plugin.GoPlugin'>
2026-01-14 15:09:37,344 - DEBUG - Loaded plugin from entry point: javascript -> <class 'metis.plugins.javascript_plugin.JavaScriptPlugin'>
2026-01-14 15:09:37,344 - DEBUG - Loaded plugin from entry point: php -> <class 'metis.plugins.php_plugin.PHPPlugin'>
2026-01-14 15:09:37,345 - DEBUG - Loaded plugin from entry point: python -> <class 'metis.plugins.python_plugin.PythonPlugin'>
2026-01-14 15:09:37,345 - DEBUG - Loaded plugin from entry point: rust -> <class 'metis.plugins.rust_plugin.RustPlugin'>
2026-01-14 15:09:37,345 - DEBUG - Loaded plugin from entry point: tablegen -> <class 'metis.plugins.tb_plugin.TableGenPlugin'>
2026-01-14 15:09:37,345 - DEBUG - Loaded plugin from entry point: terraform -> <class 'metis.plugins.terraform_plugin.TerraformPlugin'>
2026-01-14 15:09:37,345 - DEBUG - Loaded plugin from entry point: typescript -> <class 'metis.plugins.typescript_plugin.TypeScriptPlugin'>
Reviewing codebase...
2026-01-14 15:09:37,347 - INFO - MetisIgnore file not loaded .metisignore
2026-01-14 15:09:37,347 - INFO - MetisIgnore file not loaded .metisignore
2026-01-14 15:09:37,403 - INFO - Chroma vector components initialized.
2026-01-14 15:10:19,246 - WARNING - LLM fallback returned non-JSON response: Based on the FILE and RELEVANT_CONTEXT, I have identified potential security issues:

```json
{
  "reviews": [
    {
      "issue": "Potential buffer overflow vulnerability due to unvalidated pointer arithmetic",
      "code_snippet": "for (uint32_t* ptr = start; ptr < end; ptr++) {\n    uint32_t value = *ptr;\n    ...",
      "reasoning": "The `start` and `end` pointers are used for iterating over memory addresses, but their values are not validated. If these pointers are manipulated to point to an arbitrary location in memory, it could lead to a buffer overflow vulnerability.",
      "mitigation": "Validate the `start` and `end` pointers to ensure they point to valid memory locations within the specified range.",
      "confidence": 0.89,
      "cwe": "CWE-119",
      "severity": "MEDIUM"
    },
    {
      "issue": "Potential security issue due to externally controlled values for remapping",
      "code_snippet": "if (value >= OLD_REGION_BASE && value < OLD_REGION_BASE + REGION_SIZE) {\n    value = value - OLD_REGION_BASE + NEW_REGION_BASE;\n}",
      "reasoning": "The `OLD_REGION_BASE`, `REGION_SIZE`, and `NEW_REGION_BASE` values are used for remapping memory addresses. If these values are not properly sanitized or validated, they could potentially lead to security issues.",
      "mitigation": "Sanitize and validate the `OLD_REGION_BASE`, `REGION_SIZE`, and `NEW_REGION_BASE` values before using them for remapping.",
      "confidence": 0.79,
      "cwe": "CWE-20",
      "severity": "LOW"
    }
  ]
}```
No issues in test.c
Results saved to results/review_code_20260114_150937.json