Skip to content

Address feedback on new delete-nonreduced-fuzz-inputs script in rust#270

Merged
maflcko merged 1 commit intobitcoin-core:mainfrom
ekzyis:rust-delete-nonreduced-fuzz-inputs-follow-ups
Mar 30, 2026
Merged

Address feedback on new delete-nonreduced-fuzz-inputs script in rust#270
maflcko merged 1 commit intobitcoin-core:mainfrom
ekzyis:rust-delete-nonreduced-fuzz-inputs-follow-ups

Conversation

@ekzyis
Copy link
Copy Markdown
Contributor

@ekzyis ekzyis commented Mar 30, 2026

Addressed feedback:

  • use relative link in README
  • refactor argument parsing to be easier to extend
    • drops 'too many arguments' error message
  • Use ?; consistently
  • avoid temporary copy when building arguments for Command::new
  • replace wget with curl
  • also use --depth=1 when cloning AFLplusplus
  • use git add + git commit consistently instead of git commit -a
  • don't require -Znext-lockfile-bump with cargo

Additionally:

  • fix git_clone error message did not use format! macro
  • add .gitignore to ignore cargo target/

All in one commit because I think this is easy enough to review, and not "mission-critical code" anyway. Lmk if you would prefer it differently.


This is currently running, but lgtm.

Btw, I'm considering improving the UX/DX: We could pipe the current output to a file and show a progress bar on stdout (without adding a dependency). The progress bar would indicate how many fuzz targets afl-cmin has already processed, at which sanitizer step we are, and maybe there's more useful progress information we can get by looking at the current output. This would help with knowing how long it might still take. I also always want to know how long it took to know what to expect next time, but currently, I have to remember to run the script with time or look at the commit timestamps to get an idea.

With #265, the script will take even longer; making improvements to its long-running nature more important.


let all_inputs_dir = move_fuzz_inputs()?;
git_commit_all(QA_ASSETS_PATH, "Delete fuzz inputs")?;
// git commit -a does not add untracked files so we use git add manually
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// git commit -a does not add untracked files so we use git add manually
// git commit does not commit untracked removed files

I think this can be removed, or reworded?

Copy link
Copy Markdown
Contributor Author

@ekzyis ekzyis Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn’t it important to specifically mention that this is about the staging behavior of -a? To me, this reads confusingly, since git commit does not commit anything unless you stage it beforehand.

Suggested change
// git commit -a does not add untracked files so we use git add manually
// git commit -a does not commit untracked removed files

edit: did this in b89fa64

@maflcko
Copy link
Copy Markdown
Contributor

maflcko commented Mar 30, 2026

Btw, I'm considering improving the UX/DX: We could pipe the current output to a file and show a progress bar on stdout (without adding a dependency). The progress bar would indicate how many fuzz targets afl-cmin has already processed, at which sanitizer step we are, and maybe there's more useful progress information we can get by looking at the current output. This would help with knowing how long it might still take. I also always want to know how long it took to know what to expect next time, but currently, I have to remember to run the script with time or look at the commit timestamps to get an idea.

Yeah, it is a bit long running. Though, the steps may not take equal time, as libfuzzer+sanitizers or AFL may take a longer time than the parallelized libfuzzer run without sanitizers. I usually also take a look at the current git commit (or the output of (h)top to guess the progress), or let it run over night and then check the next morning.

Addressed feedback:

* use relative link in README
* refactor argument parsing to be easier to extend
  * drops 'too many arguments' error message
* Use ?; consistently
* avoid temporary copy when building arguments for Command::new
* replace wget with curl
* also use --depth=1 when cloning AFLplusplus
* use git add + git commit consistently instead of git commit -a
* don't require -Znext-lockfile-bump with cargo

Additionally:

* fix git clone error message did not use format! macro
* add .gitignore to ignore cargo target/
@ekzyis ekzyis force-pushed the rust-delete-nonreduced-fuzz-inputs-follow-ups branch from 3f5ac2d to b89fa64 Compare March 30, 2026 14:28
@maflcko maflcko merged commit 22adc08 into bitcoin-core:main Mar 30, 2026
4 checks passed
@ekzyis
Copy link
Copy Markdown
Contributor Author

ekzyis commented Mar 30, 2026

Yeah, it is a bit long running. Though, the steps may not take equal time, as libfuzzer+sanitizers or AFL may take a longer time than the parallelized libfuzzer run without sanitizers. I usually also take a look at the current git commit (or the output of (h)top to guess the progress), or let it run over night and then check the next morning.

I will run the script many more times when working on #265. I will then look into a progress bar more, and see if it's worth it

This is currently running, but lgtm.

The script just finished. New data points wrt time and determinism:

commit c763081b3c52080340e24c2b901a57c826ae1126
Author: delete_nonreduced_inputs script <noreply@noreply.noreply>
Date:   Mon Mar 30 14:36:46 2026 +0000

    Reduced inputs for fuzzer,address,undefined,integer

commit b15f92121d032712a10ddc8bfd43ea4327c8c90e
Author: delete_nonreduced_inputs script <noreply@noreply.noreply>
Date:   Mon Mar 30 13:58:47 2026 +0000

    Reduced inputs for fuzzer

commit f9c63da036af837dfd3f389312dca28560561020
Author: delete_nonreduced_inputs script <noreply@noreply.noreply>
Date:   Mon Mar 30 13:44:29 2026 +0000

    Reduced inputs for afl-cmin

commit f52507dac479618168862d4d4a85c9e3328b711b
Author: delete_nonreduced_inputs script <noreply@noreply.noreply>
Date:   Mon Mar 30 13:29:40 2026 +0000

    Delete fuzz inputs

commit 26fe402afc38f1d4b8d61c6ff3a5849221017f1e
Author: maflcko <6399679+maflcko@users.noreply.github.com>
Date:   Mon Mar 30 10:45:44 2026 +0200

    Merge pull request #269 from ekzyis/rust-delete-nonreduced-fuzz-inputs

    Convert delete_nonreduced_fuzz_inputs.sh to rust
$ find all_inputs -type f | wc -l
122591
$ find qa-assets/fuzz_corpora -type f | wc -l
116405

Previously in #269 (comment):

$ find all_inputs -type f | wc -l
122591
$ find qa-assets/fuzz_corpora -type f | wc -l
116419

Only 14 files less now!

@ekzyis ekzyis deleted the rust-delete-nonreduced-fuzz-inputs-follow-ups branch March 30, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants