bug: broken judge/attacker LLM crashes the whole run instead of a clean error during MCP baseline scans

[arunSunnyKVS]

Component

Core (@keyvaluesystems/agent-opfor-core) — MCP baseline scanner / OpenAI-compatible LLM client

Description

What happened:

Running opfor run against an MCP target with an invalid attackerLlm.model (routed through an OpenAI-compatible gateway, e.g. LiteLLM) intermittently crashes the whole run with a raw, unhandled TypeError: fetch failed instead of the clean Error: LLM HTTP 400: ... message the gateway actually returned.

Observed stack trace:

Error: LLM HTTP 400: {"error":{"message":"litellm.BadRequestError: You passed in model=Openai. There are no healthy deployments for this model. ...

vs., intermittently, on the same config:

TypeError: fetch failed
    at node:internal/deps/undici/undici:13510:13
    at async chatCompletionJsonContent (...)
    at async judgeToolResponse (...)
    at async scanToolDescriptions (...)
    at async runAll (...)

Root cause (two compounding issues):

1. chatCompletionJsonContent (core/src/llm/openaiCompatible.ts) retries up to 3 times on HTTP 400 (stripping json_object mode, then temperature) to work around providers that reject those params. It has no way to distinguish "this param isn't supported" from "this model doesn't exist" — an invalid model returns 400 on every attempt regardless of these params. Each retry fires immediately, back-to-back, without ever draining the previous (discarded) response body. An unconsumed fetch response body holds its connection open; issuing several rapid requests to the same host without draining prior bodies can exhaust the connection pool and throw an opaque TypeError: fetch failed that completely masks the real, informative LLM HTTP 400: ... error the function is designed to produce.

2. Independently, nothing between chatCompletionJsonContent and the CLI's top-level handler catches this failure. judgeToolResponse (core/src/run/judge.ts) calls it with no try/catch, and both scanResources and scanToolDescriptions in core/src/execute/baselineScanner.ts call judgeToolResponse for every resource/tool with no try/catch either — so any transient failure during the MCP pre-flight baseline scans (which run before the real evaluator suite even starts) takes down the entire run/report instead of failing just that one scan.

Steps to reproduce:

1. Configure an MCP target with a suite selected (e.g. owasp-mcp-top10).
2. Set attackerLlm to an openai-compatible provider pointed at a gateway (e.g. LiteLLM), with an invalid model name.
3. Run opfor run --config <path>.
4. Usually see a clean Error: LLM HTTP 400: .... Intermittently (network/connection-timing dependent), see an unhandled TypeError: fetch failed with a full stack trace instead, and the whole run aborts rather than reporting the scan as errored.

Impact:

• Misleading/unhelpful error message for a very common misconfiguration (wrong model name).
• A single transient network hiccup during baseline scans aborts the entire assessment rather than degrading gracefully.

Fix:

• Drain each discarded response body in chatCompletionJsonContent's 400-retry path before firing the next request.
• Wrap judgeToolResponse calls in scanResources and scanToolDescriptions so a judge/LLM failure produces an ERROR-verdict result (consistent with the existing mcpErrorJudge pattern already used elsewhere in baselineScanner.ts) and the scan continues, instead of crashing the whole run.