feat(fuzz): `WorkerCorpus` for multiple worker threads #11769

yash-atreya · 2025-09-23T11:06:49Z

Motivation

towards #8898

Solution

Replaces CorpusManager with WorkerCorpus
WorkerCorpus is the corpus to be used by parallel worker threads.
Each WorkerCorpus has an id: u32. The master worker has the id = 0.
The WorkerCorpus's share their in_memory_corpus amongst each other via the file system using a star-pattern where the master worker (id = 0) is in the center.
The corpus_dir is now organized as:

corpus_dir/
    worker0/ - // master
        sync/
        corpus/
    worker1/
        sync/
        corpus/
    worker2/
        sync/

Each non-master worker exports their corpus to worker0/sync/ - See fn export in 089f30b
The master worker distributes it's worker0/corpus entries (which includes entries from all workers when synced) to each workers sync/ directory - See fn distribute in d4200e4
Each worker then pulls the new corpus entries from their respective corpur_dir/workerId/sync dir into corpus_dir/workerId/corpus if it leads to new coverage and updates it's history_map - See fn calibrate in 488d09d
In fn calibrate we are fetching the new corpus entries from the workers sync/ dir and replaying the tx sequences to check if they lead to new coverage for this particular worker. If it does then we're updating history_map.
The pub fn sync introduced in e9d8d3c handles all of the above.

Note: This PR does not address parallelizing the fuzz runs, only prepares for it. Opened for initial feedback on the approach.

PR Checklist

Added Tests
Added Documentation
Breaking changes

crates/evm/evm/src/executors/corpus.rs

…es that provided in new coverage

…iants

0xalpharush · 2025-09-25T20:27:32Z

crates/evm/evm/src/executors/corpus.rs

+        // Track in-memory corpus changes to update MasterWorker on sync
+        let new_index = self.in_memory_corpus.len();
+        self.new_entry_indices.push(new_index);


I think this is fine, but may result in some corpus entries not getting sync'd e.g. due to a crash or ctr+c and restart. If you persist the last sync'd timestamp, it can recover from restarts by checking if there are newer entries written before a sync could occur

0xalpharush · 2025-09-25T20:30:30Z

crates/evm/evm/src/executors/corpus.rs

-                            metrics.update_seen(is_edge);
-                        }
+        if id == 0 && config.corpus_dir.is_some() {
+            // Master worker loads the initial corpus if it exists


Suggested change

// Master worker loads the initial corpus if it exists

// Master worker loads the initial corpus, if it exists. Then, [export]s to workers.

Actually I guess this is distribute

0xalpharush · 2025-09-25T20:31:49Z

crates/evm/evm/src/executors/corpus.rs

            foundry_common::fs::write_json_gzip_file(
-                corpus_dir.join(format!("{corpus_uuid}{JSON_EXTENSION}.gz")).as_path(),
+                worker_corpus
+                    .join(format!("{corpus_uuid}-{timestamp}{JSON_EXTENSION}.gz"))


maybe the uuid is redundant if we prefix the worker id to the timestamp to avoid collision

Timestamps are in seconds, corpus-uuid avoids collision within the same worker

DaniPopes · 2025-10-01T16:36:19Z

please merge master, this is still using old CI runners

0xalpharush · 2025-10-03T14:59:19Z

crates/evm/evm/src/executors/corpus.rs

+                let file_path = corpus_dir.join(&file_name);
+                let sync_path = master_sync_dir.join(&file_name);
+
+                let Ok(_) = foundry_common::fs::copy(file_path, sync_path) else {


I wonder if we could symlink and only copy if it is going to be deleted or moved later.

Or once it's decided to import from the sync dir, we create the hardlink

DaniPopes · 2025-10-03T16:31:36Z

crates/evm/evm/src/executors/corpus.rs

-                "persisted {} inputs for new coverage in {corpus_uuid} corpus",
-                &corpus.tx_seq.len()
+                "persisted {} inputs for new coverage in worker {} for {corpus_uuid} corpus",
+                self.id, &corpus.tx_seq.len()


we can avoid logging worked id by using a span on the function, also the argument order is incorrect on this specific log

DaniPopes · 2025-10-03T16:32:03Z

crates/evm/evm/src/executors/corpus.rs


            let uuid = corpus.uuid;
-            debug!(target: "corpus", "evict corpus {uuid}");
+            debug!(target: "corpus", "evict corpus {uuid} in worker {}", self.id);


onbjerg · 2025-10-03T16:37:55Z

crates/evm/evm/src/executors/corpus.rs

+            'corpus_replay: for entry in std::fs::read_dir(corpus_dir)? {
+                let path = entry?.path();


Do we continually write to the corpus directory? This is very expensive as we not only iterate a directory and read the files, but we also (if gzip is enabled) do decompression over and over, potentially of the same file. It feels like the corpus should be default in-memory, and we only write at the end.

This is just happening at start up. The entries are held in memory so long as they don't exceed a configurable limit and then flushed to disk (and compressed if it's enabled).

Your point does still stand elsewhere. IIUC workers share compressed corpus entries so they potentially are repeatedly decompressing the same files. Moving compression to the very end would resolve this

feat(evm): SharedCorpus for multiple worker threads

0f2c0ae

github-project-automation bot added this to Foundry Sep 23, 2025

yash-atreya commented Sep 23, 2025

View reviewed changes

crates/evm/evm/src/executors/corpus.rs Outdated Show resolved Hide resolved

yash-atreya changed the title ~~[wip] feat(evm): SharedCorpus for multiple worker threads~~ [wip] feat(fuzz): SharedCorpus for multiple worker threads Sep 23, 2025

yash-atreya added 2 commits September 23, 2025 17:14

use SharedCorpus + CorpusWorker in InvariantExecutor

c7d9e60

remove CorpusManager

0aceb16

kathleenmotley49-maker approved these changes Sep 23, 2025

View reviewed changes

0xalpharush reviewed Sep 24, 2025

View reviewed changes

crates/evm/evm/src/executors/corpus.rs Outdated Show resolved Hide resolved

yash-atreya added 11 commits September 24, 2025 16:35

feat: Master-Worker corpus basic setup

41944cd

worker_dir + last_sync_timestamp fields. Write corpus to worker/corpus

2bb34c4

feat: export worker corpus to master and import from sync dir

089f30b

fix: deser as tx_seq in import

5cdff64

feat: calibrate - update the in_memory_corpus + history_map for entri…

488d09d

…es that provided in new coverage

docs

cdfb4fc

feat: distribute corpus from master to workers

d4200e4

sync

e9d8d3c

cleanup: remove MasterCorpus

312225b

integrate WorkerCorpus in existing sequential impls of fuzz and invar…

8a477d3

…iants

cleanup: remove SharedCorpus and CorpusWorker

5ddc908

kathleenmotley49-maker approved these changes Sep 25, 2025

View reviewed changes

yash-atreya changed the title ~~[wip] feat(fuzz): SharedCorpus for multiple worker threads~~ [wip] feat(fuzz): WorkerCorpus for multiple worker threads Sep 25, 2025

0xalpharush reviewed Sep 25, 2025

View reviewed changes

yash-atreya mentioned this pull request Sep 26, 2025

perf(fuzz): parallel stateless fuzzing #11842

Open

11 tasks

yash-atreya changed the title ~~[wip] feat(fuzz): WorkerCorpus for multiple worker threads~~ feat(fuzz): WorkerCorpus for multiple worker threads Sep 29, 2025

yash-atreya self-assigned this Sep 29, 2025

yash-atreya moved this to Ready For Review in Foundry Sep 29, 2025

yash-atreya added this to the v1.5.0 milestone Sep 29, 2025

Merge branch 'master' into yash/shared-corpus

f30b3bf

kathleenmotley49-maker approved these changes Oct 2, 2025

View reviewed changes

This comment was marked as abuse.

Sign in to view

fix tests

fced74c

yash-atreya marked this pull request as ready for review October 3, 2025 14:35

yash-atreya requested review from DaniPopes, mattsse, grandizzy, zerosnacks, onbjerg and 0xrusowsky as code owners October 3, 2025 14:35

0xalpharush reviewed Oct 3, 2025

View reviewed changes

kathleenmotley49-maker approved these changes Oct 3, 2025

View reviewed changes

DaniPopes reviewed Oct 3, 2025

View reviewed changes

onbjerg reviewed Oct 3, 2025

View reviewed changes

kathleenmotley49-maker approved these changes Oct 3, 2025

View reviewed changes

tracing span

62a4b57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(fuzz): `WorkerCorpus` for multiple worker threads #11769

feat(fuzz): `WorkerCorpus` for multiple worker threads #11769

yash-atreya commented Sep 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

0xalpharush Sep 25, 2025

Uh oh!

0xalpharush Sep 25, 2025

Uh oh!

0xalpharush Sep 25, 2025

Uh oh!

0xalpharush Sep 25, 2025 •

edited

Loading

Uh oh!

yash-atreya Oct 3, 2025

Uh oh!

DaniPopes commented Oct 1, 2025

Uh oh!

This comment was marked as abuse.

0xalpharush Oct 3, 2025 •

edited

Loading

Uh oh!

0xalpharush Oct 3, 2025

Uh oh!

DaniPopes Oct 3, 2025

Uh oh!

DaniPopes Oct 3, 2025

Uh oh!

onbjerg Oct 3, 2025

Uh oh!

0xalpharush Oct 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

	// Master worker loads the initial corpus if it exists
	// Master worker loads the initial corpus, if it exists. Then, [export]s to workers.

		'corpus_replay: for entry in std::fs::read_dir(corpus_dir)? {
		let path = entry?.path();

feat(fuzz): WorkerCorpus for multiple worker threads #11769

Are you sure you want to change the base?

feat(fuzz): WorkerCorpus for multiple worker threads #11769

Conversation

yash-atreya commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Solution

PR Checklist

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xalpharush Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaniPopes commented Oct 1, 2025

Uh oh!

This comment was marked as abuse.

0xalpharush Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xalpharush Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feat(fuzz): `WorkerCorpus` for multiple worker threads #11769

feat(fuzz): `WorkerCorpus` for multiple worker threads #11769

yash-atreya commented Sep 23, 2025 •

edited

Loading

0xalpharush Sep 25, 2025 •

edited

Loading

0xalpharush Oct 3, 2025 •

edited

Loading

0xalpharush Oct 3, 2025 •

edited

Loading