bug: judge reports terminal ERROR on openai-compatible providers that return valid-but-empty JSON in json_object mode

[arunSunnyKVS]

Component

MCP Server / Judge (LLM-as-judge)

Description

What happened:
When running opfor run against an MCP target with an openai-compatible provider (e.g. attackerLlm/judgeLlm configured with a custom baseURL), some attacks end with verdict: "ERROR" and errorMessage: "unparseable judge output: {}" in the JSON report, even though the tool call itself succeeded and OPFOR received a normal HTTP 200 from the LLM.

Root cause: chatCompletionJsonContent (core/src/llm/openaiCompatible.ts) requests response_format: { type: "json_object" }. Some OpenAI-compatible providers honor this shallowly — they guarantee the response is syntactically valid JSON, but don't guarantee the model actually populated the fields asked for in the prompt. The provider returned a literal {} body.

That {} passes extractJson()'s "starts with {" check (openaiCompatible.ts:40) as if it were well-formed judge output, gets forwarded to verdictParser.parseJson(), which does JSON.parse("{}") successfully, finds no verdict key, falls back to line-parsing (parseLines), finds no Verdict: label either, and terminally reports ERROR: unparseable judge output: {}.

What I expected:
The 400-response retry path in chatCompletionJsonContent (core/src/llm/openaiCompatible.ts:102-111) already retries once without response_format: json_object when the provider rejects the param outright (HTTP 400). I'd expect a similar retry to kick in when the provider returns HTTP 200 with syntactically-valid-but-semantically-empty JSON (no verdict key) — i.e. treat "valid JSON missing the required field" the same as "provider rejected json_object," and retry once with jsonMode: false + plain-text extraction, before giving up and reporting ERROR.

Right now a single judge call to a provider with shallow JSON-mode support terminally fails the whole attack's verdict, with no retry.

Steps to reproduce

1. Configure an MCP target (transport: "stdio" or "url") with attackerLlm/judgeLlm set to "provider": "openai-compatible", pointing baseURL at a provider whose json_object mode returns valid-but-empty JSON on some prompts (rather than erroring or fully complying).
2. opfor run --config <path-to-config> against any MCP evaluator suite (e.g. mcp-smoke).
3. Observe the terminal output — some attacks show ⚠ ERROR (score 0/10) while others (typically the baseline scans with simpler judge prompts) show ✓ PASS.
4. Open the JSON report and inspect the judge object for the errored attacks:

   "judge": {
     "verdict": "ERROR",
     "score": 0,
     "confidence": 0,
     "evidence": "N/A",
     "reasoning": "Judge output contained no parseable Verdict line.",
     "errorMessage": "unparseable judge output: {}"
   }

Notably, the attacks that fail this way tend to be the ones with longer/more complex judge prompts (e.g. multi-field evaluators like SSRF, resource-exposure, content-injection), while simpler judge prompts (tool-poisoning baseline scans) succeed — consistent with the provider's JSON-mode compliance being prompt-length/complexity dependent rather than a hard error.

Config file (redacted)

{
  "target": {
    "kind": "mcp",
    "name": "Example MCP Server",
    "transport": "stdio",
    "command": "node",
    "args": ["/path/to/mcp-server/dist/index.js"]
  },
  "selection": {
    "mode": "suite",
    "suite": "mcp-smoke"
  },
  "attackerLlm": {
    "provider": "openai-compatible",
    "model": "<redacted-model-name>",
    "baseURL": "https://<redacted-openai-compatible-endpoint>/v1",
    "apiKeyEnv": "REDACTED_API_KEY"
  },
  "effort": "adaptive",
  "turns": 1,
  "turnMode": "single"
}

Relevant log output

Results: 4 passed, 0 failed, 3 errors
Warning:
⚠️  3 attack(s) failed due to errors — results may be incomplete.
Safety score: 57%

JSON report excerpt (repeated across the 3 errored attacks):

{
  "judge": {
    "verdict": "ERROR",
    "score": 0,
    "confidence": 0,
    "evidence": "N/A",
    "reasoning": "Judge output contained no parseable Verdict line.",
    "errorMessage": "unparseable judge output: {}"
  }
}

Opfor version

0.9.0

Operating system

Windows (WSL)

Node.js version

v22.23.0

LLM provider

OpenAI-compatible (LiteLLM, OpenRouter, Ollama, etc.)

Additional context

Relevant code paths:

• core/src/llm/openaiCompatible.ts — chatCompletionJsonContent() builds the request and only retries on HTTP 400 (unsupported json_object/temperature params); it does not retry on HTTP 200 responses containing valid-but-empty/incomplete JSON.
• core/src/evaluators/verdictParser.ts — parseJson() correctly falls back to parseLines() when JSON has no usable verdict, but by that point the raw provider response was already just {}, so the line-parser has nothing to recover either. This part of the parser is behaving correctly — the gap is upstream, in not retrying the LLM call itself.

Suggested fix direction: in chatCompletionJsonContent, after parsing a 200 response, check whether the extracted JSON actually contains a usable verdict key (or reuse verdictParser's tolerant verdict check) before returning it as final. If it's missing, retry once with jsonMode: false (same fallback already used for the 400 case), the same way the existing 400-driven retry works — rather than surfacing the empty object straight to the caller as terminal judge output.

This is not unique to any single provider — any OpenAI-compatible endpoint with shallow json_object enforcement (accepts the param, guarantees syntax, doesn't guarantee content) will hit this.