Skip to content

feat: implement JSON export functionality #2228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
khs-alt wants to merge 3 commits into from
Closed

feat: implement JSON export functionality #2228

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a748dc5
feat: implement JSON export functionality using json.Marshal
khs-alt Aug 4, 2025
04362bd
fix: resolve JSON export error handling issues
khs-alt Aug 11, 2025
78e184a
feat: enhance JSON exporter and add README
khs-alt Aug 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions runner/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@ type Options struct {
RespectHSTS bool
StoreResponse bool
JSONOutput bool
JSONExport string
CSVOutput bool
Comment on lines 223 to 226
Copy link

@coderabbitai coderabbitai bot Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add validation & conflict checks for the new JSONExport field

The option is parsed and stored but never validated.
Consider extending ValidateOptions() to:

  1. Fail fast when the provided path is not writable / its parent directory does not exist.
  2. Warn or error when -json-export is combined with streaming JSONL output (-json) – the two formats differ and may confuse users / downstream tooling.
  3. Guard against passing a directory path instead of a file.

Example patch:

@@ func (options *Options) ValidateOptions() error {
+	// sanity-check json-export
+	if options.JSONExport != "" {
+		// reject if used together with json lines
+		if options.JSONOutput {
+			return fmt.Errorf("flags -json and -json-export are mutually exclusive")
+		}
+		dir := filepath.Dir(options.JSONExport)
+		if dir != "." && !fileutil.DirExists(dir) {
+			return fmt.Errorf("directory for json-export (%s) does not exist", dir)
+		}
+		if strings.HasSuffix(options.JSONExport, string(os.PathSeparator)) {
+			return fmt.Errorf("json-export expects a file path, got directory '%s'", options.JSONExport)
+		}
+	}

This prevents late-runtime errors and clarifies expected usage.

🤖 Prompt for AI Agents 
In runner/options.go around lines 223 to 226, the new JSONExport field lacks
validation and conflict checks. Extend the ValidateOptions() function to verify
that the JSONExport path's parent directory exists and is writable, return an
error if not. Add a check to prevent using JSONExport together with JSONOutput,
returning an error or warning about the conflicting output formats. Also, ensure
the JSONExport path is not a directory by checking its file info and returning
an error if it is. These validations will prevent runtime errors and clarify
correct usage.

CSVOutputEncoding string
PdcpAuth string
Expand Down Expand Up @@ -475,6 +476,7 @@ func ParseOptions() *Options {
flagSet.BoolVarP(&options.StoreVisionReconClusters, "store-vision-recon-cluster", "svrc", false, "include visual recon clusters (-ss and -sr only)"),
flagSet.StringVarP(&options.Protocol, "protocol", "pr", "", "protocol to use (unknown, http11)"),
flagSet.StringVarP(&options.OutputFilterErrorPagePath, "filter-error-page-path", "fepp", "filtered_error_page.json", "path to store filtered error pages"),
flagSet.StringVarP(&options.JSONExport, "json-export", "je", "", "file to export results in JSON format"),
)

flagSet.CreateGroup("configs", "Configurations",
Expand Down
24 changes: 23 additions & 1 deletion runner/runner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -780,7 +780,8 @@
}
}()

var plainFile, jsonFile, csvFile, indexFile, indexScreenshotFile *os.File
var plainFile, jsonFile, csvFile, indexFile, indexScreenshotFile, jsonExportFile *os.File
var jsonExportResults []Result

if r.options.Output != "" && r.options.OutputAll {
plainFile = openOrCreateFile(r.options.Resume, r.options.Output)
Expand Down Expand Up @@ -1172,6 +1173,27 @@
}
}

if r.options.JSONExport != "" {
filename := r.options.JSONExport
Copy link
Member

@dogancanbakir dogancanbakir Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can pass r.options.JSONExport instead of introducing new var.

Copy link
Contributor Author

@khs-alt khs-alt Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback! I'll update it.


if jsonExportResults == nil {
jsonExportResults = make([]Result, 0)
}
jsonExportResults = append(jsonExportResults, resp)
Copy link
Member

@dogancanbakir dogancanbakir Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of copying this and adding batch support to be nice on memory and make it easier/flexible to play with the implementation details later on. Since the relatively large data could cause memory issues, and we will make sure most of the data is written to the file.

Copy link
Contributor Author

@khs-alt khs-alt Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your suggestion, it looks correct!
I have two questions:

  1. httpx repo doesn't have a pkg folder, so should I make the same as nuclei pkg/reporting/exporters/jsonexporter/jsonexporter.go and add to use this code, or any path to put jsonexporter.go code?
  2. Current jsonexporter is a little different to use httpx, should it be okay to change jsonexporter.go code for using httpx, or change httpx to use jsonexporter.go code format

Copy link
Member

@dogancanbakir dogancanbakir Aug 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re 1: let's not go with structural change for now, we can just introduce a new go file
re 2: we can adapt it for the httpx


if jsonExportFile == nil {
jsonExportFile = openOrCreateFile(r.options.Resume, filename)
defer func() {
if jsonExportFile != nil && len(jsonExportResults) > 0 {
if jsonData, err := json.Marshal(jsonExportResults); err == nil {
jsonExportFile.Write(jsonData)

Check failure on line 1189 in runner/runner.go

View workflow job for this annotation

GitHub Actions / Lint Test 

Error return value of `jsonExportFile.Write` is not checked (errcheck)
}
jsonExportFile.Close()
}
}()
}
}

if r.options.CSVOutput {
row := resp.CSVRow(&r.scanopts)

Expand Down
Loading