P0 bug(rtk curl): large JSON responses truncated mid-stream producing invalid JSON, breaks all API agent pipelines

[anhermon-dealhub]

Summary

When rtk curl receives a JSON response larger than the internal passthrough_max_chars threshold (~510 chars), it truncates the raw response body mid-stream — inserting ... (N bytes total) directly inside a JSON string value. This produces invalid, unparseable JSON on stdout, silently breaking every agent pipeline that pipes rtk curl to a JSON parser.

This is a distinct bug from #1152 (which covers schema conversion of small JSON). This truncation happens at a lower layer — the raw body is cut before schema conversion can run — and is far more destructive because it produces invalid JSON rather than schema-typed JSON.

Environment

• RTK version: 0.37.2
• OS: macOS Darwin 25.4.0
• Hook mode: Claude Code PreToolUse hook (rtk hook claude)

Reproduction

# Small JSON response (~200 bytes) — passes through fine:
rtk curl -s 'http://127.0.0.1:3101/api/health'
# stdout: {"status":"ok","version":"0.3.1","deploymentMode":"local_trusted",...}
# Valid JSON ✅

# Large JSON response (~4000 bytes) — truncated mid-stream:
rtk curl -s 'http://127.0.0.1:3101/api/companies/COMPANY_ID/issues?status=in_progress&limit=2'
# stdout: [{"id":"9366a21a-...","title":"...","description":"Infrastructure scope...\n\n**What to do:**... (4048 bytes total)
#         [full output: ~/Library/Application Support/rtk/tee/1777194358_curl.log]
# INVALID JSON ❌

# Confirm breakage:
rtk curl -s '...' | python3 -c "import sys,json; json.load(sys.stdin)"
# json.decoder.JSONDecodeError: Invalid control character at: line 1 column 521 (char 520)

What actually appears on stdout

[{"id":"9366a21a-...","companyId":"...","title":"DEA-187 follow-up: ...","description":"Infrastructure scope extracted from DEA-187 (Prometheus alerting rules).\n\n**Selected approach: Option A**\n\n**What to do:**... (4048 bytes total)
[full output: ~/Library/Application Support/rtk/tee/1777194358_curl.log]

The truncation marker ... (4048 bytes total) is inserted inside a JSON string value (the description field), immediately breaking JSON syntax. The full response is only in the tee log file; stdout is non-recoverable.

Root Cause

The passthrough_max_chars limit in LimitsConfig is applied to the raw HTTP body before any JSON processing. When the body exceeds ~510 chars, the body is cut and the truncation marker is appended mid-stream — directly into whatever JSON string was being written, regardless of structure.

This is different from the schema conversion path in src/cmds/cloud/curl_cmd.rs (tracked in #1152). That path converts small-enough JSON to schema form. This truncation fires when the response is too large to enter the schema path at all.

Impact

P0 — breaks all agent JSON API pipelines:

• rtk curl ... | python3 -c "import json,sys; json.load(sys.stdin)" → JSONDecodeError
• rtk curl ... | jq '.' → parse error (Invalid numeric literal at line 1, column 521)
• Any agent using the Claude Code hook (rtk hook claude) that calls a local REST API with non-trivial responses gets silent data loss — the pipeline appears to run but yields no parseable data
• Currently blocks RTK pilot rollout to autonomous coding agents — agents cannot call their orchestration APIs (Paperclip, local services) via the hook

Expected Behaviour

When rtk curl produces JSON output that exceeds passthrough_max_chars:

Option A (preferred): Pass JSON through unmodified. JSON is already structured and compact; mid-stream truncation provides zero token savings and silently corrupts the pipeline.

Option B: Emit a valid JSON document describing the truncation:

{"_rtk": {"truncated": true, "total_bytes": 4048, "tee_log": "~/Library/Application Support/rtk/tee/1777194358_curl.log"}}

Option C: Respect passthrough_max_chars = 0 or the no_truncation = true flag proposed in #1313.

Workaround

# Bypass rtk filtering entirely (loses token savings):
rtk proxy curl -s 'http://127.0.0.1:3101/api/.../issues?status=in_progress'

Related Issues

• bug(rtk curl): JSON schema conversion destroys API response values, breaks browser CDP connections #1152 — JSON schema conversion destroys values (schema conversion path, distinct bug)
• Feature request: lossless-only mode + honest account of silent failures truncation causes in agent contexts #1313 — Feature request: lossless-only mode (overlapping solution space)
• feat: config for curl filter bypass #219 — Config for curl filter bypass (resolved-pending-close, solution would help)
• curl subcommand truncates output when piped to bash, breaking curl | bash install patterns #1311 — curl truncation breaks piped bash install (CLOSED, same truncation mechanism)

[pszymkowiak]

[w] wshm · Automated triage by AI

🔍 Automated Triage

🐛 Category	bug
🔴 Priority	critical
🎯 Confidence	93%

Summary

rtk curl truncates large JSON responses mid-stream via passthrough_max_chars, producing invalid JSON that breaks all downstream parsers and agent pipelines. Needs fix to either skip truncation for JSON content-type or emit a valid truncation envelope.

📁 Relevant files

• src/cmds/cloud/curl_cmd.rs

---

Triaged automatically by wshm · This is an automated analysis, not a human review.