Skip to content

perf: use json.Marshal instead of json.MarshalIndent for non-multipart request bodies #825

@dangrondahl

Description

@dangrondahl

Problem

internal/requests/requests.go:122 still uses json.MarshalIndent when building non-multipart JSON request bodies:

jsonBytes, err := json.MarshalIndent(p.Payload, "", "    ")

This is the exact same class of bug that was fixed for multipart payloads in #822 / commit 251f3bc. Pretty-printing adds whitespace that inflates every JSON payload sent over the wire — roughly 30–100% larger depending on nesting depth — for no benefit since the server deserializes regardless of formatting.

Affected commands

Every command that sends a JSON body through the non-multipart path is affected. This is the majority of the CLI — approximately 28 commands including:

Category Commands
Snapshots kosli snapshot docker, kosli snapshot k8s, kosli snapshot ecs, kosli snapshot lambda, kosli snapshot s3, kosli snapshot azure-apps, kosli snapshot server, kosli snapshot path
Reporting kosli report artifact, kosli report approval
Creation kosli create flow, kosli create environment
Admin kosli rename flow, kosli rename environment, kosli join environment, kosli attach policy, kosli detach policy, kosli allow artifact, kosli tag, kosli disable beta

The 4 commands that use multipart form uploads (kosli attest * variants that attach files, kosli create policy, kosli create attestation-type) were already fixed in #822.

Impact

  • Bandwidth: Every API call sends ~30–100% more bytes than necessary. For snapshot commands that run frequently in CI/CD pipelines, this adds up.
  • Latency: Larger payloads mean slightly slower uploads, especially on constrained CI runners or high-latency connections.
  • Consistency: The multipart path was fixed in fix: use json.Marshal instead of json.MarshalIndent for multipart data_json field #822, but the non-multipart path still pretty-prints — inconsistent behavior.

Payload example: kosli snapshot k8s

The K8S snapshot sends a K8sEnvRequest containing an array of PodData structs. For a cluster with 2 pods, the indented payload is ~970 bytes vs ~630 bytes compact — ~54% inflation. The ratio stays constant regardless of pod count:

Cluster size Indented Compact Waste
10 pods ~4.5 KB ~2.9 KB ~1.6 KB (55%)
100 pods ~45 KB ~29 KB ~16 KB (55%)
500 pods ~225 KB ~145 KB ~80 KB (55%)

Server-side verification

An audit of all ~106 endpoints in the Kosli server confirmed compact JSON is safe across the board:

  • Standard Pydantic body parsing (~99 endpoints): FastAPI deserializes JSON into Pydantic models. Format-agnostic — only the JSON structure matters, not whitespace.
  • Multipart JsonModel parsing (attestation endpoints): Uses Pydantic's model_validate_json() internally — also format-agnostic. Already validated when compact multipart JSON shipped in fix: use json.Marshal instead of json.MarshalIndent for multipart data_json field #822.
  • Webhook HMAC verification (3 endpoints — LaunchDarkly, Sonar, Descope): Computes HMAC on raw bytes, but this is safe because both sides see the same bytes regardless of formatting.
  • No raw-body inspection elsewhere: Zero json.loads calls in the app code, zero regex/string matching on raw JSON payloads, zero custom decoders. All validation happens on deserialized Python objects.
  • Server integration tests already use compact JSON: test/integration/api/test_report_k8s.py (and others) call json.dumps() which produces compact output by default.

Conclusion: No server endpoint depends on JSON indentation, whitespace, or formatting.

Fix

One-line change on internal/requests/requests.go:122:

// Before
jsonBytes, err := json.MarshalIndent(p.Payload, "", "    ")

// After
jsonBytes, err := json.Marshal(p.Payload)

The PayloadOutput function (used for --dry-run and --debug) already handles pretty-printing at the logging site (lines 316–321), so human-readable output is preserved.

Verification

Metadata

Metadata

Assignees

Labels

performanceIssues related to performance bottlenecks or improvements

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions