feat: add JSON Stream (NDJSON) body processor

[fzipi]

what

Implements a new body processor for handling streaming JSON formats with per-record rule evaluation:

• NDJSON (Newline Delimited JSON)
• JSON Lines
• JSON Sequence (RFC 7464)

Streaming body processor

• Line-by-line processing for memory efficiency
• Each JSON object indexed by record number (json.0.field, json.1.field)
• Built-in DoS protection with 1024 recursion limit
• Support for nested objects and arrays
• Auto-detection of format (NDJSON vs RFC 7464) by peeking at the first 4KB
• Registered under JSONSTREAM, NDJSON, and JSONLINES aliases

Per-record rule evaluation

Instead of parsing all records into ArgsPost and evaluating Phase 2 rules once, rules are now evaluated after each complete JSON record:

• Malicious records are caught immediately without processing the rest of the stream
• ArgsPost is cleared between records so each record is evaluated in isolation
• TX variables (e.g. anomaly scores) persist across records, enabling cross-record correlation — a stream where 3 records each score below the threshold individually can
  still trigger a block when the accumulated score exceeds it
• Eval() is safe to call multiple times per phase — AllowTypePhase, Skip, and the transformation cache all reset correctly between calls
• Non-streaming body processors are completely unaffected

New StreamingBodyProcessor interface

Extends BodyProcessor with ProcessRequestRecords/ProcessResponseRecords methods that yield parsed records one at a time via callback. The transaction detects this
interface via type assertion and switches to the per-record evaluation path automatically.

Streaming relay support

Added ProcessRequestBodyFromStream(input, output) / ProcessResponseBodyFromStream(input, output) methods on the transaction for integrators building custom streaming
middleware. These read records from input, evaluate rules per record, and write clean records to output. Exposed via an experimental StreamingTransaction interface.

Usage example

SecRule REQUEST_HEADERS:Content-Type "^application/x-ndjson" \
    "id:'200007',phase:1,pass,nolog,ctl:requestBodyProcessor=JSONSTREAM"

Testing

• 24 unit tests for the body processor (single/multiple lines, nested objects, arrays, error cases, recursion limits, TX variable storage, large tokens, format
  auto-detection)
• 7 unit tests for callback-based record processing (interruption stops processing, field prefixes, RFC 7464, backward compatibility)
• 4 integration tests with real WAF rules (interruption at bad record, clean passthrough, TX variable accumulation across records, below-threshold no-block)
• Benchmark: ~5,000 ops/sec for 100-object streams

Benchmark Results (Apple M2)

ProcessRequest (buffered) vs Callback (streaming)

Scenario	Records	ProcessRequest	Callback	Throughput Speedup	Alloc Reduction
small	1	2.00 MB/s, 146 allocs	2.66 MB/s, 17 allocs	1.3x	88%
small	10	11.30 MB/s, 283 allocs	16.70 MB/s, 134 allocs	1.5x	53%
small	100	22.38 MB/s, 1,644 allocs	33.01 MB/s, 1,304 allocs	1.5x	21%
small	1,000	22.84 MB/s, 16,671 allocs	36.20 MB/s, 14,493 allocs	1.6x	13%
medium	1	8.50 MB/s, 176 allocs	11.92 MB/s, 39 allocs	1.4x	78%
medium	10	26.20 MB/s, 579 allocs	41.03 MB/s, 354 allocs	1.6x	39%
medium	100	31.71 MB/s, 4,563 allocs	52.34 MB/s, 3,505 allocs	1.7x	23%
medium	1,000	31.54 MB/s, 51,097 allocs	53.93 MB/s, 41,712 allocs	1.7x	18%
nested	1	8.53 MB/s, 178 allocs	12.04 MB/s, 41 allocs	1.4x	77%
nested	10	25.22 MB/s, 599 allocs	38.13 MB/s, 374 allocs	1.5x	38%
nested	100	30.71 MB/s, 4,763 allocs	48.66 MB/s, 3,705 allocs	1.6x	22%
nested	1,000	30.49 MB/s, 53,101 allocs	48.92 MB/s, 43,713 allocs	1.6x	18%

RFC 7464 (JSON Sequence) via Callback

Scenario	Records	Throughput	Allocs/op
small	10	16.61 MB/s	134
small	100	34.37 MB/s	1,304
medium	100	47.02 MB/s	3,505
nested	100	48.06 MB/s	3,705

Key Takeaways

• The callback-based streaming path is consistently 1.5–1.7x faster in throughput.
• Allocation counts are 13–88% lower (most dramatic at low record counts where per-collection overhead dominates).
• RFC 7464 format performance is comparable to NDJSON at the same record counts, confirming negligible format auto-detection overhead.
• The callback path avoids populating TransactionVariables collections (ArgsPost, TX vars, raw body via TeeReader), which accounts for the reduced allocations.

Record Templates

• small: {"id":1,"name":"Alice"} (24 bytes)
• medium: {"user_id":1234567890,"name":"User Name","email":"user@example.com","role":"admin","active":true,"tags":["tag1","tag2","tag3"]} (128 bytes)
• nested: {"user":{"name":"Alice","address":{"city":"NYC","zip":"10001"}},"scores":[95,87,92],"meta":{"created":"2026-01-01","active":true}} (131 bytes)

[codecov]

Codecov Report

❌ Patch coverage is 52.69461% with 158 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.07%. Comparing base (537abf9) to head (d4502d1).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/corazawaf/transaction.go	10.17%	148 Missing and 2 partials ⚠️
experimental/bodyprocessors/jsonstream.go	95.15%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1481      +/-   ##
==========================================
- Coverage   85.30%   84.07%   -1.24%     
==========================================
  Files         174      175       +1     
  Lines        8461     8811     +350     
==========================================
+ Hits         7218     7408     +190     
- Misses        994     1146     +152     
- Partials      249      257       +8     

Flag	Coverage Δ
coraza.rule.case_sensitive_args_keys	84.04% <52.69%> (-1.24%)	⬇️
coraza.rule.mandatory_rule_id_check	84.06% <52.69%> (-1.24%)	⬇️
coraza.rule.multiphase_evaluation	83.80% <52.69%> (-1.23%)	⬇️
coraza.rule.no_regex_multiline	84.06% <52.69%> (-1.24%)	⬇️
default	84.07% <52.69%> (-1.24%)	⬇️
examples+	21.03% <24.55%> (+4.54%)	⬆️
examples+coraza.rule.case_sensitive_args_keys	83.97% <52.69%> (-1.23%)	⬇️
examples+coraza.rule.mandatory_rule_id_check	84.06% <52.69%> (-1.24%)	⬇️
examples+coraza.rule.multiphase_evaluation	83.60% <52.69%> (-1.22%)	⬇️
examples+coraza.rule.no_regex_multiline	83.91% <52.69%> (-1.23%)	⬇️
examples+memoize_builders	84.01% <52.69%> (-1.24%)	⬇️
examples+no_fs_access	81.75% <52.69%> (-1.14%)	⬇️
ftw	84.07% <52.69%> (-1.24%)	⬇️
memoize_builders	84.18% <52.69%> (-1.25%)	⬇️
no_fs_access	83.61% <52.69%> (-1.22%)	⬇️
tinygo	84.05% <52.69%> (-1.24%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds support for JSON Stream (NDJSON) body processing to Coraza WAF, enabling line-by-line processing of streaming JSON formats. The implementation includes a new body processor that handles NDJSON, JSON Lines, and claims support for JSON Sequence (RFC 7464).

Changes:

• New jsonStreamBodyProcessor that processes JSON objects line-by-line with memory-efficient streaming
• Built-in DoS protection via configurable recursion limits (default 1024)
• TX variable storage for raw body and line count to enable custom validation rules
• Configuration rules in coraza.conf-recommended for NDJSON content types

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File	Description
internal/bodyprocessors/jsonstream.go	Core implementation of NDJSON body processor with line-by-line parsing, recursion limits, and TX variable storage
internal/bodyprocessors/jsonstream_test.go	Comprehensive test suite with 13 test cases covering single/multiple lines, nested objects, arrays, error cases, and benchmarks
coraza.conf-recommended	Configuration rules for enabling NDJSON processing based on Content-Type headers, with optional line count limiting

[jcchavezs]

Could we add this to experimental?

[fzipi]

I thought I mentioned this. Yes, that was my idea.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[fzipi]

@copilot Fix the conflicts first. Then add e2e tests to tests for this new body processor.

[Copilot]

@fzipi I've opened a new pull request, #1563, to work on those changes. Once the pull request is ready, I'll request review from you.