feat: add JSON Stream (NDJSON) body processor#1563
Merged
fzipi merged 38 commits intofeat/add-streaming-json-processorfrom Mar 21, 2026
Merged
feat: add JSON Stream (NDJSON) body processor#1563fzipi merged 38 commits intofeat/add-streaming-json-processorfrom
fzipi merged 38 commits intofeat/add-streaming-json-processorfrom
Conversation
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* docs(actions): update format and add package Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * fix: update documentation for package Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * fix: go fmt Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
* fix: SecRuleUpdateActionById should replace disruptive actions Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * fix: multiphase test with bad expectations Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * tests: improve coverage on engine Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * refactor: address SecRuleUpdateActionById review comments (#1484) * Initial plan * Address code review comments: improve documentation, fix double parsing, and fix range logic Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * Refactor: Extract hasDisruptiveActions helper to avoid code duplication Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * docs: Improve applyParsedActions documentation Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * docs: Clarify body parsing logic in SetRawRequest Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * refactor: address review comments on SecRuleUpdateActionById - Rename ClearActionsOfType to ClearDisruptiveActions - Add comments explaining quote trimming in action parsing - Remove empty line after function brace in updateActionBySingleID - Split engine_test.go: move output/helper tests to engine_output_test.go * Apply suggestions from code review Co-authored-by: Matteo Pace <pace.matteo96@gmail.com> * fix: use index-based iteration for SecRuleUpdateActionById range updates The range loop variable copied each Rule, so modifications to disruptive actions were lost. Use index-based iteration to modify rules in place. Also adds a test case exercising the range update path. --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Matteo Pace <pace.matteo96@gmail.com>
* refactor: remove root package dependency on experimental Replace experimental.Options with corazawaf.Options in waf.go, breaking the import cycle that prevented the experimental package from importing the root coraza package. This unblocks PR #1478 and lets experimental helpers use coraza.WAFConfig with proper type safety instead of any. * Update waf.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* No content wants no body * Update .github/workflows/regression.yml Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com> * one more place --------- Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com>
* feat: add optional rule observer callback to WAF config Introduce an optional rule observer callback that is invoked for each rule successfully added to the WAF during initialization. The observer receives rule metadata via the existing RuleMetadata interface. * Move to the experimental package * Do not use reflection to keep the compatibility with older Go versions * Use coraza.WAFConfig, move the test to where it belongs. --------- Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com> Co-authored-by: José Carlos Chávez <jcchavezs@gmail.com>
Add WAFWithRules interface with RulesCount()
* fix(deps): update module golang.org/x/net to v0.51.0 [security] * chore: update go.work to 1.25.0 Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * chore: update golang to 1.25.0 Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Felipe Zipitria <felipe.zipitria@owasp.org>
…ns (#1505) * fix: lowercase regex patterns for case-insensitive variable collections When a rule uses regex-based variable selection (e.g. TX:/PATTERN/), the regex pattern was compiled from the raw uppercase string before any case normalization. Since TX collection keys are stored lowercase, the uppercase regex would never match, causing rules like CRS 922110 (which uses TX:/MULTIPART_HEADERS_CONTENT_TYPES_*/) to silently fail. Now AddVariable and AddVariableNegation lowercase the regex pattern before compilation for case-insensitive variables, matching the existing behavior for string keys in newRuleVariableParams. * chore: update coreruleset to v4.24.0 Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
* chore: update libinjection-go and deps Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * chore: update coreruleset v4.24.0 Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org>
…1495) * Initial plan * Fix ruleRemoveTargetById to support removing entire collection (empty key) Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
* feat: add SecRequestBodyJsonDepthLimit directive Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * Apply suggestions from code review * fix: mage format Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * Update internal/bodyprocessors/json_test.go * Update internal/bodyprocessors/json_test.go * fix: bad char Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * fix: gofmt Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * docs: add clarifying comments for JSON recursion limit behavior - Explain why ResponseBodyRecursionLimit = -1 (unlimited for responses) - Document dual purpose of body reading (TX vars + ARGS_POST) - Clarify DoS protection mechanism in readItems() - Note how negative values bypass recursion check * fix: address PR review comments for JSON depth limit - Always enforce a positive recursion limit: change ResponseBodyRecursionLimit from -1 (unlimited) to 1024, matching the request body default - Rename test case "broken1" to "unbalanced_brackets" for clarity - Extract error check from the key iteration loop in TestReadJSON * test: add benchmarks for gjson.Valid pre-validation overhead Measures the cost of gjson.Valid() in the full readJSON pipeline. gjson.Parse is lazy (~9ns), so the real overhead is Valid vs the readItems traversal. Results show ~10-16% overhead for validation, which is acceptable for WAF safety. No single-pass alternative exists in the gjson API. * Apply suggestions from code review * Apply suggestion from @fzipi --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: José Carlos Chávez <jcchavezs@gmail.com>
* fix: conflate the constants for recursion limit * fix: value setting
* Initial plan * fix: replace panic with error return in parser.go evaluateLine Co-authored-by: jptosso <1236942+jptosso@users.noreply.github.com> * fix: revert go.sum changes - do not modify go.sum files in this PR Co-authored-by: jptosso <1236942+jptosso@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jptosso <1236942+jptosso@users.noreply.github.com>
Replace dynamic 64-permutation tag matrix with a curated static list of 13 build-flag combinations. Run all combos on Go 1.25.x and only baseline + kitchen-sink on Go 1.26.x. Add concurrency groups to regression, lint, tinygo, and codeql workflows so stale PR runs are auto-cancelled on new pushes.
…1453) * Ignore unexpected EOF in MIME multipart request body processor We need this behavior since we need to process an incomplete MIME multipart request body when SecRequestBodyLimitAction is set to ProcessPartial. * fix: add copilot code review comments Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: José Carlos Chávez <jcchavezs@gmail.com> Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com> Co-authored-by: Felipe Zipitria <felipe.zipitria@owasp.org>
Fix two bugs where transformation functions modified the input string but did not report changed=true: - removeComments: entering a C-style (/* */) or HTML (<!-- -->) comment block did not set changed=true, causing the multi-match optimization to skip the transformed result. - escapeSeqDecode: unrecognized escape sequences (e.g. \z) dropped the backslash but did not set changed=true. Add test coverage for both fixes including a new remove_comments_test.go and an additional unrecognized-escape test case for escape_seq_decode.
* perf: use map for ruleRemoveByID for O(1) lookup
Replace []int slice with map[int]struct{} for the per-transaction
rule exclusion list. The rule evaluation loop checks this list for
every rule in every phase, making O(1) map lookup significantly
faster than O(n) linear scan when rules are excluded via ctl actions.
* test: add TestRemoveRuleByID for map-based rule exclusion
* bench: add BenchmarkRuleEvalWithRemovedRules
* refactor: use real unconditionalMatch operator from registry in tests
* Fix HTTP middleware to process all Transfer-Encoding values Co-authored-by: jptosso <1236942+jptosso@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jptosso <1236942+jptosso@users.noreply.github.com> Co-authored-by: Matteo Pace <pace.matteo96@gmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…expanding to int slices (#1538) * Initial plan * Optimize ruleRemoveById range handling to avoid generating massive int slices - Replace rangeToInts (which allocated []int of all matching rule IDs) with parseRange and parseIDOrRange helpers that return start/end integers - For ctlRuleRemoveByID with ranges: store the range in Transaction.ruleRemoveByIDRanges ([][2]int) and check it in the rule evaluation loop, avoiding both the intermediate []int and potentially large map expansions - For ctlRuleRemoveTargetByID: iterate rules once directly, eliminating the intermediate []int allocation - Add RemoveRuleByIDRange method to Transaction - Reset ruleRemoveByIDRanges on transaction pool reuse - Replace TestCtlParseRange with TestCtlParseIDOrRange to test the new helpers Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * Improve test coverage for range-based rule removal - Add TestRemoveRuleByIDRange in transaction_test.go: - range is stored in ruleRemoveByIDRanges - rules in range are skipped during Eval - multiple ranges work correctly - ruleRemoveByIDRanges is reset on transaction pool reuse - Add TestCtlParseRange in ctl_test.go to cover parseRange directly (including the no-separator and start>end error paths) - Add GetRuleRemoveByIDRanges() accessor on Transaction for cross-package test assertions - Enhance "ruleRemoveById range" TestCtl case to verify the range is stored - Add "ruleRemoveTargetById range" TestCtl case to verify range path works Coverage changes: parseRange: 83.3% → 100% parseIDOrRange: 100% (unchanged) RemoveRuleByIDRange: 0% → 100% Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
Redesign the transformation cache to share intermediate results across rules with common transformation prefixes (e.g. rules using t:lowercase,t:urlDecodeUni reuse the t:lowercase result cached by an earlier rule using just t:lowercase). Key changes: - Add transformationPrefixIDs to Rule for backward prefix search - Cache every intermediate transformation step, not just the final result - Store cache values inline (not pointers) to avoid heap allocations - Fix ClearTransformations (t:none) to reset transformationsID Benchmarked against full CRS v4 ruleset (8 runs, benchstat): Allocations: -2% (small) to -19% (30 params) Memory: -2% (small) to -12% (30 params) Timing: -5% (small/large), neutral (medium) No regressions on any metric. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* perf: bulk-allocate MatchData in collection Find methods Pre-allocate a contiguous []corazarules.MatchData buffer and take pointers into it instead of individually heap-allocating each MatchData. This reduces per-result allocations from N to 2 (one buf slice + one result slice), improving GC pressure for large result sets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * perf: avoid double regex evaluation in FindRegex Collect matching data slices during the counting pass so the second pass only iterates over already-matched entries, eliminating redundant MatchString calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * bench: add FindAll/FindRegex/FindString benchmarks --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com>
* perf: use FindStringSubmatchIndex to avoid capture allocations Replace FindStringSubmatch (allocates a []string slice per match) with FindStringSubmatchIndex (returns index pairs). Substrings passed to CaptureField become slices of the original input — zero allocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add BenchmarkRxCapture for submatch allocation comparison Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…les (#1549) * add detectedInterruption var for DetectionOnly mode * IsDetectionOnly, refactor, populate matchedRules * nit * Apply suggestions from code review Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: Romain SERVIERES <romain@madeformed.com> Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* more fuzztime * go mod
- Add top-level `permissions: {}` (deny-all) to every workflow
- Add scoped per-job permissions granting only what each job needs
- Fix expression injection in regression.yml by using env instead of
inline shell interpolation for BUILD_TAGS
- Restrict regression.yml pull_request trigger to main branch only
- Add explicit permissions to fuzz.yml (issues: write for failure reports)
- Add security-events: write to CodeQL workflow
* feat: enable regex memoize by default Memoization of regex and aho-corasick builders was previously opt-in via the `memoize_builders` build tag. Most users didn't know to enable it, missing a critical performance optimization. This commit: - Enables memoization by default (opt-out via `coraza.no_memoize` tag) - Refactors internal/memoize from package-level Do() to Memoizer struct - Adds Memoizer interface to plugintypes.OperatorOptions - Wires WAF's Memoizer through to all operator and rule consumers - Replaces `memoize_builders` build tag with `coraza.no_memoize` opt-out Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: document cache tradeoffs and add noop memoize test - Update README and memoize README to document global cache behavior and point to WAF.Close() for live-reload scenarios. - Add test file for coraza.no_memoize build variant to verify no-op behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add WAF.Close() with per-owner memoize cache tracking and scale benchmarks (#1541) * feat: add WAF.Close() with per-owner memoize cache tracking Add WAFCloser interface and per-owner tracking to the memoize cache so that long-lived processes can release compiled regex entries when a WAF instance is destroyed. Each WAF gets a uint64 ID; Release() removes the owner and tombstones entries with no remaining owners. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add memoize scale benchmarks and CRS integration tests Add benchmarks demonstrating memoize value at scale (1-100 WAFs × 300 patterns) and CRS integration tests verifying Close() releases memory. Results show ~27x speedup for 100 WAFs and 27MiB released on Close(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add WAF.Close() calls to e2e and CRS tests Demonstrate proper WAFCloser usage in integration tests: e2e test, CRS FTW test, CRS benchmarks, and crsWAF helper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * test: extend coraza.no_memoize coverage in noop_test.go (#1555) * Initial plan * test: extend noop_test.go coverage for coraza.no_memoize build tag Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * fix: check error return of m.Do in benchmark to resolve errcheck lint failure (#1556) * Initial plan * fix: check error return of m.Do in benchmark test to fix errcheck lint Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * fix: skip memoize scale tests in short mode The scale tests (TestMemoizeScaleMultipleOwners, TestCacheGrowthWithoutClose, TestCacheBoundedWithClose) compile hundreds of regexes across many owners/cycles. Under TinyGo's slower regex engine these take hours when run in CI with -short. Gate all three scale tests behind testing.Short() in both sync_test.go and nosync_test.go so TinyGo CI (which passes -short) completes in reasonable time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(memoize): avoid deadlock in TinyGo's sync.Map during Release and Reset TinyGo's sync.Map.Range() holds its internal lock for the entire iteration. Calling cache.Delete() inside the Range callback tries to re-acquire the same non-reentrant lock, causing a deadlock. Defer all cache.Delete() calls until after Range returns by collecting keys first. This also fixes t.Skip() in tests which does not halt execution in TinyGo due to unimplemented runtime.Goexit(). On standard Go this is a net performance win for Release (up to 60% faster at 100 owners) with negligible temporary memory (~9KB slice). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Felipe Zipitría <3012076+fzipi@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat: implement SecUploadKeepFiles with RelevantOnly support Add UploadKeepFilesStatus type supporting On, Off, and RelevantOnly values for the SecUploadKeepFiles directive. When set to On, uploaded files are preserved after transaction close. When set to RelevantOnly, files are kept only if rules matched during the transaction. Closes #1550 * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @M4tteoP Co-authored-by: Matteo Pace <pace.matteo96@gmail.com> * docs: update SecUploadKeepFiles in coraza.conf-recommended Remove the "not supported" note and document the RelevantOnly option. * fix: filter nolog rules in RelevantOnly upload keep files check RelevantOnly now only considers rules with Log enabled, matching the same filtering used for audit log part K. This prevents CRS initialization rules (nolog) from making RelevantOnly behave like On. * fix: require SecUploadDir when SecUploadKeepFiles is enabled Add validation in WAF.Validate() to ensure SecUploadDir is configured when SecUploadKeepFiles is set to On or RelevantOnly, matching the ModSecurity requirement. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix: directive docs Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Matteo Pace <pace.matteo96@gmail.com> * fix: correct two compile errors in SecUploadKeepFiles implementation (#1560) * Initial plan * fix: correct lint errors - HasAccessToFS is a bool not a function, fix wrong constant name Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * fix: gofmt Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> * fix: skip SecUploadKeepFiles tests when no_fs_access build tag is set The upload keep files tests expected success for On/RelevantOnly modes, but the implementation correctly rejects these when filesystem access is disabled. Guard these test cases behind environment.HasAccessToFS. --------- Signed-off-by: Felipe Zipitria <felipe.zipitria@owasp.org> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Matteo Pace <pace.matteo96@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
…ByTag, and ruleRemoveTargetByMsg collection keys (#1561) * Initial plan * Add regex support to ctl:ruleRemoveTargetById for URI-scoped exclusions Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * Use memoization for regex compilation in parseCtl Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * Add benchmarks for short and medium regex exceptions in GetField Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * refactor: add HasRegex shared utility and use it in rule.go and ctl.go Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * test: add POST JSON body test for ruleRemoveTargetById regex key exclusion Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * docs: update RemoveRuleTargetByID comment to document keyRx parameter Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * docs: update ctl action doc comment to describe regex key syntax with example Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * test: add ruleRemoveTargetByTag and ruleRemoveTargetByMsg regex key integration tests Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * style: apply gofmt to internal/actions/ctl.go Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> * test: add memoizer coverage to TestParseCtl for ctl regex path Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com>
…ot/sub-pr-1481 # Conflicts: # coraza.conf-recommended # internal/corazawaf/transaction.go # waf.go
Co-authored-by: fzipi <3012076+fzipi@users.noreply.github.com> Agent-Logs-Url: https://github.com/corazawaf/coraza/sessions/bebca76e-344f-4966-8675-8bf4e5fda0cb
Copilot
AI
changed the title
[WIP] Fix: Add JSON Stream (NDJSON) body processor
feat: add JSON Stream (NDJSON) body processor
Mar 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a streaming JSON body processor (
JSONSTREAM) for NDJSON, JSON Lines, and RFC 7464 (JSON Sequence) formats, with per-record rule evaluation to catch malicious records without buffering the full stream.Body Processor (
experimental/bodyprocessors)bufio.Scannerwith configurable max token sizejson.<N>.<field>inARGS_POST0x1E) byte in first 4KBJSONSTREAM,NDJSON, andJSONLINESaliasesPer-Record Rule Evaluation (
internal/corazawaf/transaction.go)processRequestBodyStreaming/processResponseBodyStreamingclear and repopulateArgsPostper record, evaluating Phase 2 rules after each one. TX variables (e.g. anomaly scores) persist across records enabling cross-record correlation.Eval()is safe to call multiple times per phase —AllowTypePhase,Skip, and transformation cache all reset correctly between calls.Streaming Relay (
experimental/streaming.go)ProcessRequestBodyFromStream(input, output)/ProcessResponseBodyFromStream(input, output)on the transaction for integrators building streaming middleware. Exposed viaexperimental.StreamingTransaction.Configuration
Commented-out example rules added to
coraza.conf-recommendedalongside the newSecRequestBodyJsonDepthLimit 1024directive from main.E2E Tests (
testing/e2e/ndjson_e2e_test.go)HTTP-level tests through
txhttp.WrapHandler:application/jsonlinescontent type⌨️ Start Copilot coding agent tasks without leaving your editor — available in VS Code, Visual Studio, JetBrains IDEs and Eclipse.