Seeding rand does not make the fuzzing campaign deterministic

[addisoncrump]

(Submitted upon email request for preserving anonymity; I cannot give follow-up details on this issue)

Seeding the rand used in the state does not make the fuzzing campaign deterministic even when a single fuzzer is used with 100% stability. As such, there is a non-determinism source which is affecting runs that is unaccounted for in "standard" fuzzers.

We should identify the source of this non-determinism so that users seeking to do replayable fuzzer campaigns may do so reliably.

[Mrmaxmeier]

Just checking: Could time-based input scheduling be the culprit here?
I believe AFL and AFL++ measure input execution time in milliseconds (possibly masking some non-determinism) while LibAFL is using nanos in a few places?

[tokatoka]

no. we are doing the same as AFL, AFL++
#1885

and the values are, as dan said, generally 0 or 1

[domenukk]

In AFL++ we had to fix/disable the timing for scheduling, yes.
In any case, the moment the target has timeouts it'll break.

I don't think determinism is a high goal for fuzzers in general

[langston-barrett]

I don't think determinism is a high goal for fuzzers in general

I'd just like to add my perspective as a user building several fuzzers on top of LibAFL: Determinism is important to me in developing my fuzzers, because I generally have tests that show that my fuzzer can produce certain behaviors in certain targets in a reasonable amount of time. These tests use fixed seeds for all sources of randomness, and their determinism is important because it helps us accurately track regressions.

[domenukk]

The better way would be not to use fixed seeds here, but rather to rerun the test 5 times. Scheduling, mutators, etc., are free to change between versions, or we may just add a single new call to rand at any point in time, so for a CI szenario a fixed seed is not advisable, either (IMHO)

[langston-barrett]

I disagree.

A few of my fuzzers target system-level software (specifically UEFI EDK II) and require running the boot process for a while before taking a snapshot and beginning fuzzing. Running each such test five times would be quite expensive.

Running the tests without a fixed seed (even if they are each run multiple times) would still run the risk of spurious failure or success if seeeds are chosen unluckily or luckily, respectively. Erroneous results like this are frustrating to debug and erode developer confidence and investment in test suites and CI builds. They also make it harder to run git bisect to track down when problems were introduced.

I understand that LibAFL is free to add new calls to rand and that this might cause me to have to change the seeds in my test suite when upgrading LibAFL. This has been a relatively minor burden so far. I wouldn't advocate for determinism between versions, just within a particular version.

I don't want to distract from the purpose of this ticket so I'd be happy to continue this discussion elsewhere.

[addisoncrump]

Is it maybe because our hash map orderings are nondeterministic?