feat: implement JSON export functionality#2228
feat: implement JSON export functionality#2228khs-alt wants to merge 3 commits intoprojectdiscovery:devfrom
Conversation
- Add JSON export option with -je flag - Use json.Marshal for safe JSON generation - Store results in memory and write at completion - Maintain code consistency with other output formats
WalkthroughAdds a new CLI option Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant Runner
participant JSONExporter
participant FS
User->>CLI: run with --json-export <file>
CLI->>Runner: Options{JSONExport: <file>}
Runner->>Runner: start enumeration
alt JSONExport set
Runner->>JSONExporter: NewJSONExporter(File: <file>)
JSONExporter-->>Runner: exporter instance
loop per result
Runner->>JSONExporter: Export(result)
JSONExporter-->>Runner: ack / error
end
Runner->>JSONExporter: Close()
JSONExporter->>FS: write JSON array to <file>
FS-->>JSONExporter: write result / error
end
Runner-->>User: finish
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
runner/options.go (1)
478-480: Nit: keep short-flag conventions consistentMost short options are 1–2 characters (
-j,-rr,-svrc). Using-jefor Json Export is fine but slightly asymmetric with existing single-letter-j. Double-letter shorthands increase the chance of clashes later.If backward-compatibility permits, consider making
-Jthe short form:-flagSet.StringVarP(&options.JSONExport, "json-export", "je", "", "file to export results in JSON format") +flagSet.StringVarP(&options.JSONExport, "json-export", "J", "", "file to export results in JSON format")Purely stylistic, feel free to skip if the project already embraces multi-letter shorthands elsewhere.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
runner/options.go(2 hunks)runner/runner.go(2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
runner/runner.go (1)
runner/types.go (1)
Result(35-103)
🔇 Additional comments (1)
runner/runner.go (1)
783-784: LGTM!The variable declarations for JSON export functionality are properly placed and typed correctly.
| StoreResponse bool | ||
| JSONOutput bool | ||
| JSONExport string | ||
| CSVOutput bool |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add validation & conflict checks for the new JSONExport field
The option is parsed and stored but never validated.
Consider extending ValidateOptions() to:
- Fail fast when the provided path is not writable / its parent directory does not exist.
- Warn or error when
-json-exportis combined with streaming JSONL output (-json) – the two formats differ and may confuse users / downstream tooling. - Guard against passing a directory path instead of a file.
Example patch:
@@ func (options *Options) ValidateOptions() error {
+ // sanity-check json-export
+ if options.JSONExport != "" {
+ // reject if used together with json lines
+ if options.JSONOutput {
+ return fmt.Errorf("flags -json and -json-export are mutually exclusive")
+ }
+ dir := filepath.Dir(options.JSONExport)
+ if dir != "." && !fileutil.DirExists(dir) {
+ return fmt.Errorf("directory for json-export (%s) does not exist", dir)
+ }
+ if strings.HasSuffix(options.JSONExport, string(os.PathSeparator)) {
+ return fmt.Errorf("json-export expects a file path, got directory '%s'", options.JSONExport)
+ }
+ }This prevents late-runtime errors and clarifies expected usage.
🤖 Prompt for AI Agents
In runner/options.go around lines 223 to 226, the new JSONExport field lacks
validation and conflict checks. Extend the ValidateOptions() function to verify
that the JSONExport path's parent directory exists and is writable, return an
error if not. Add a check to prevent using JSONExport together with JSONOutput,
returning an error or warning about the conflicting output formats. Also, ensure
the JSONExport path is not a directory by checking its file info and returning
an error if it is. These validations will prevent runtime errors and clarify
correct usage.
There was a problem hiding this comment.
@khs-alt Thanks for the PR! Let me know what you think!
runner/runner.go
Outdated
There was a problem hiding this comment.
We can pass r.options.JSONExport instead of introducing new var.
There was a problem hiding this comment.
Thanks for the feedback! I'll update it.
runner/runner.go
Outdated
There was a problem hiding this comment.
What do you think of copying this and adding batch support to be nice on memory and make it easier/flexible to play with the implementation details later on. Since the relatively large data could cause memory issues, and we will make sure most of the data is written to the file.
There was a problem hiding this comment.
Thank you for your suggestion, it looks correct!
I have two questions:
- httpx repo doesn't have a pkg folder, so should I make the same as nuclei
pkg/reporting/exporters/jsonexporter/jsonexporter.goand add to use this code, or any path to putjsonexporter.gocode? - Current jsonexporter is a little different to use httpx, should it be okay to change
jsonexporter.gocode for using httpx, or change httpx to usejsonexporter.gocode format
There was a problem hiding this comment.
re 1: let's not go with structural change for now, we can just introduce a new go file
re 2: we can adapt it for the httpx
|
Hi! @dogancanbakir I'm changing the code and the readme file. And also, I'm trying to check four tags |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (6)
runner/jsonexporter.go (2)
32-40: Misleading comment: no batching implemented.The comment says “writes batch when full” but the code only appends to an in-memory slice. Update the comment to match behavior.
-// Export adds result and writes batch when full +// Export adds the result to the in-memory buffer func (exporter *JSONExporter) Export(result *Result) error {
11-15: Use value Mutex instead of pointer.sync.Mutex should be embedded by value to avoid nil-pointer risks and reduce allocations. It’s safe and conventional.
type JSONExporter struct { options *JSONExportOptions - mutex *sync.Mutex + mutex sync.Mutex ... }Then update locks: exporter.mutex.Lock() stays the same.
runner/runner.go (4)
875-875: Initialize JSON exporter once before the loop; always produce a file (even with zero results).Currently, the exporter is lazily created inside the loop. If there are no results, the JSON file is never created. Initialize it once before ranging over output and register a single deferred Close.
Apply this diff right before the
for resp := range output {:+ // Initialize JSON exporter (if requested) before processing any results + if r.options.JSONExport != "" && jsonExporter == nil { + exportOptions := &JSONExportOptions{ + File: r.options.JSONExport, + } + var err error + jsonExporter, err = NewJSONExporter(exportOptions) + if err != nil { + gologger.Error().Msgf("Failed to create JSON exporter: %s", err) + } else { + defer func() { + if closeErr := jsonExporter.Close(); closeErr != nil { + gologger.Error().Msgf("Failed to close JSON exporter: %s", closeErr) + } + }() + } + }
1176-1193: Remove per-iteration initialization and defer; keep just the Export call.With the exporter created once before the loop, this block becomes unnecessary. It also avoids registering a defer inside the processing loop.
- if r.options.JSONExport != "" { - if jsonExporter == nil { - exportOptions := &JSONExportOptions{ - File: r.options.JSONExport, - } - - var err error - jsonExporter, err = NewJSONExporter(exportOptions) - if err != nil { - gologger.Error().Msgf("Failed to create JSON exporter: %s", err) - } else { - defer func() { - if closeErr := jsonExporter.Close(); closeErr != nil { - gologger.Error().Msgf("Failed to close JSON exporter: %s", closeErr) - } - }() - } - } - - if jsonExporter != nil { - if err := jsonExporter.Export(&resp); err != nil { - gologger.Error().Msgf("Failed to export JSON result: %s", err) - } - } - } + if jsonExporter != nil { + if err := jsonExporter.Export(&resp); err != nil { + gologger.Error().Msgf("Failed to export JSON result: %s", err) + } + }
1195-1199: Trim large payloads in JSON export to match JSONOutput behavior.JSONOutput uses Result.JSON(&scanopts) which trims ResponseBody to MaxResponseBodySizeToSave. JSONExport currently exports the full struct, which can bloat memory and file size (especially with large bodies or screenshots). Mirror the trimming logic before export.
- if jsonExporter != nil { - if err := jsonExporter.Export(&resp); err != nil { - gologger.Error().Msgf("Failed to export JSON result: %s", err) - } - } + if jsonExporter != nil { + exportResp := resp + if r.scanopts.MaxResponseBodySizeToSave > 0 && len(exportResp.ResponseBody) > r.scanopts.MaxResponseBodySizeToSave { + exportResp.ResponseBody = exportResp.ResponseBody[:r.scanopts.MaxResponseBodySizeToSave] + } + // Respect NoScreenshotBytes if set (already applied earlier for stored responses), keep as-is otherwise. + if err := jsonExporter.Export(&exportResp); err != nil { + gologger.Error().Msgf("Failed to export JSON result: %s", err) + } + }
1268-1333: Note: graceful shutdown vs. finalized JSON.Exporter Close is deferred within the output routine and will execute on normal completion. If the process is terminated abruptly (SIGKILL) or crashes, the JSON array may be left unterminated. If this is a concern, consider signal handling to trigger a graceful shutdown path that closes the exporter.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
README.md(1 hunks)runner/jsonexporter.go(1 hunks)runner/runner.go(2 hunks)
✅ Files skipped from review due to trivial changes (1)
- README.md
🔇 Additional comments (3)
runner/runner.go (3)
783-785: LGTM: clean integration point and isolation.Declaring a dedicated jsonExporter scoped to the output routine keeps concerns separated and avoids cross-goroutine synchronization.
1121-1128: Good: honoring NoScreenshotBytes to avoid embedding large binary blobs.Zeroing ScreenshotBytes when NoScreenshotBytes is set helps keep JSON and CSV outputs smaller. This will also benefit JSON export.
Also applies to: 1114-1119
1024-1057: JSON-only flags apply to both –j and –jeI’ve verified that the include-response-header (
-irh), include-response (-irr), include-response-base64 (-irrb) and include-chain (--include-chain) flags are bound unconditionally tooptions.ResponseHeadersInStdout,options.ResponseInStdout,options.Base64ResponseInStdoutandoptions.ChainInStdoutinrunner/options.go, and those in turn setscanopts.*fields inrunner/runner.goregardless of whether-jsonor-json-exportis used. The JSON exporter (NewJSONExporter→Export(&resp)) simply serializes the samerespstruct populated based on thosescanoptsflags. No additional wiring is needed—these flags already affect the JSON-export file.
| // NewJSONExporter creates a new JSON exporter | ||
| func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) { | ||
|
|
||
| exporter := &JSONExporter{ | ||
| mutex: &sync.Mutex{}, | ||
| options: options, | ||
| rows: []Result{}, | ||
| } | ||
| return exporter, nil | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Validate options early and prepare the filesystem.
Constructor doesn’t validate options or ensure the directory exists. If options.File is empty or its parent directory is missing, Close will fail late. Validate upfront and create the directory.
Apply this diff to validate inputs and prepare directories:
func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) {
-
- exporter := &JSONExporter{
- mutex: &sync.Mutex{},
- options: options,
- rows: []Result{},
- }
- return exporter, nil
+ if options == nil || options.File == "" {
+ return nil, errors.New("json exporter: file path is required")
+ }
+ // Ensure parent directory exists
+ if dir := filepath.Dir(options.File); dir != "." && dir != "" {
+ if err := os.MkdirAll(dir, 0o755); err != nil {
+ return nil, errors.Wrap(err, "json exporter: failed to create output directory")
+ }
+ }
+
+ exporter := &JSONExporter{
+ mutex: &sync.Mutex{},
+ options: options,
+ rows: []Result{},
+ }
+ return exporter, nil
}Also add imports:
import (
+ "path/filepath"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // NewJSONExporter creates a new JSON exporter | |
| func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) { | |
| exporter := &JSONExporter{ | |
| mutex: &sync.Mutex{}, | |
| options: options, | |
| rows: []Result{}, | |
| } | |
| return exporter, nil | |
| } | |
| // NewJSONExporter creates a new JSON exporter | |
| func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) { | |
| if options == nil || options.File == "" { | |
| return nil, errors.New("json exporter: file path is required") | |
| } | |
| // Ensure parent directory exists | |
| if dir := filepath.Dir(options.File); dir != "." && dir != "" { | |
| if err := os.MkdirAll(dir, 0o755); err != nil { | |
| return nil, errors.Wrap(err, "json exporter: failed to create output directory") | |
| } | |
| } | |
| exporter := &JSONExporter{ | |
| mutex: &sync.Mutex{}, | |
| options: options, | |
| rows: []Result{}, | |
| } | |
| return exporter, nil | |
| } |
| // NewJSONExporter creates a new JSON exporter | |
| func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) { | |
| exporter := &JSONExporter{ | |
| mutex: &sync.Mutex{}, | |
| options: options, | |
| rows: []Result{}, | |
| } | |
| return exporter, nil | |
| } | |
| import ( | |
| // ... other imports ... | |
| "path/filepath" | |
| ) |
| // Close writes remaining data and closes | ||
| func (exporter *JSONExporter) Close() error { | ||
| exporter.mutex.Lock() | ||
| defer exporter.mutex.Unlock() | ||
|
|
||
| // Convert the rows to JSON byte array | ||
| obj, err := json.Marshal(exporter.rows) | ||
| if err != nil { | ||
| return errors.Wrap(err, "failed to generate JSON report") | ||
| } | ||
|
|
||
| // Attempt to write the JSON to file specified in options.JSONExport | ||
| if err := os.WriteFile(exporter.options.File, obj, 0644); err != nil { | ||
| return errors.Wrap(err, "failed to create JSON file") | ||
| } | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Avoid OOM by streaming JSON to disk instead of buffering all results.
Accumulating every Result in memory and marshalling at Close is risky for large scans and can crash with OOM or lead to very long Close() times. Stream the JSON array to disk incrementally.
Proposed minimal streaming exporter (open file once; write “[”, then “,” + object per Export; write “]” on Close):
@@
-package runner
+package runner
import (
+ "bufio"
"encoding/json"
"os"
"sync"
"github.com/pkg/errors"
+ "path/filepath"
)
@@
type JSONExporter struct {
options *JSONExportOptions
- mutex *sync.Mutex
- rows []Result
+ mutex *sync.Mutex
+ // streaming state
+ file *os.File
+ writer *bufio.Writer
+ enc *json.Encoder
+ first bool // true until first element is written
}
@@
func NewJSONExporter(options *JSONExportOptions) (*JSONExporter, error) {
- exporter := &JSONExporter{
- mutex: &sync.Mutex{},
- options: options,
- rows: []Result{},
- }
- return exporter, nil
+ if options == nil || options.File == "" {
+ return nil, errors.New("json exporter: file path is required")
+ }
+ if dir := filepath.Dir(options.File); dir != "." && dir != "" {
+ if err := os.MkdirAll(dir, 0o755); err != nil {
+ return nil, errors.Wrap(err, "json exporter: failed to create output directory")
+ }
+ }
+ f, err := os.OpenFile(options.File, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600)
+ if err != nil {
+ return nil, errors.Wrap(err, "json exporter: failed to open output file")
+ }
+ w := bufio.NewWriter(f)
+ // start JSON array
+ if _, err := w.WriteString("["); err != nil {
+ _ = f.Close()
+ return nil, errors.Wrap(err, "json exporter: failed to initialize json array")
+ }
+ enc := json.NewEncoder(w)
+ enc.SetEscapeHTML(false)
+ exporter := &JSONExporter{
+ mutex: &sync.Mutex{},
+ options: options,
+ file: f,
+ writer: w,
+ enc: enc,
+ first: true,
+ }
+ return exporter, nil
}
@@
-// Export adds result and writes batch when full
+// Export writes one result into the JSON array
func (exporter *JSONExporter) Export(result *Result) error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
- exporter.rows = append(exporter.rows, *result)
+ if exporter.file == nil || exporter.writer == nil || exporter.enc == nil {
+ return errors.New("json exporter: not initialized")
+ }
+ // write comma if not the first element
+ if !exporter.first {
+ if _, err := exporter.writer.WriteString(","); err != nil {
+ return errors.Wrap(err, "json exporter: failed to write delimiter")
+ }
+ }
+ exporter.first = false
+ if err := exporter.enc.Encode(result); err != nil {
+ return errors.Wrap(err, "json exporter: failed to encode result")
+ }
return nil
}
@@
-// Close writes remaining data and closes
+// Close finalizes the JSON array and closes the file
func (exporter *JSONExporter) Close() error {
exporter.mutex.Lock()
defer exporter.mutex.Unlock()
- // Convert the rows to JSON byte array
- obj, err := json.Marshal(exporter.rows)
- if err != nil {
- return errors.Wrap(err, "failed to generate JSON report")
- }
-
- // Attempt to write the JSON to file specified in options.JSONExport
- if err := os.WriteFile(exporter.options.File, obj, 0644); err != nil {
- return errors.Wrap(err, "failed to create JSON file")
- }
-
- return nil
+ if exporter.writer == nil || exporter.file == nil {
+ return nil
+ }
+ // close the JSON array
+ if _, err := exporter.writer.WriteString("]"); err != nil {
+ return errors.Wrap(err, "json exporter: failed to finalize json array")
+ }
+ if err := exporter.writer.Flush(); err != nil {
+ return errors.Wrap(err, "json exporter: failed to flush writer")
+ }
+ if err := exporter.file.Close(); err != nil {
+ return errors.Wrap(err, "json exporter: failed to close file")
+ }
+ exporter.writer = nil
+ exporter.file = nil
+ return nil
}Benefits:
- Constant memory (O(1) w.r.t. number of results).
- Ensures a valid JSON array even on long-running scans.
- Immediate I/O backpressure; earlier failures surface sooner.
If you prefer the nuclei-style exporter with batch size/rolling flush, we can adapt that instead.
|
Thanks for taking the time to work on this! After reviewing the changes and the discussion in #2218 We appreciate your effort and contribution, but we’ll go ahead and close this PR for now. Thanks again for your time and interest in improving httpx! |
3 participants
This PR addresses issue #2218
go run cmd/httpx/httpx.go -hgo run cmd/httpx/httpx.go -u google.com -je test.jsonSummary by CodeRabbit