Move to a typed choice sequence mutator#53
Conversation
Liam-DeVoe
force-pushed
the
tcs-mutator
branch
from
2322b7f to
ae89938
Compare
|
yeah in fact adding |
Liam-DeVoe
force-pushed
the
tcs-mutator
branch
2 times, most recently
from
af84f1a to
5dc390c
Compare
|
OK, I'm much more confident in these changes now that we have some tests. Still have some type hints to take care of. |
Liam-DeVoe
force-pushed
the
tcs-mutator
branch
from
5dc390c to
67c0b33
Compare
| is_hypothesis_file = belongs_to(hypothesis) | ||
| is_hypofuzz_file = belongs_to(hypofuzz) | ||
| is_pytest_file = belongs_to(pytest) | ||
| is__pytest_file = belongs_to(_pytest) |
There was a problem hiding this comment.
Hmm, we should check that belongs_to() memoizes so that the returned functions share a cache.
There was a problem hiding this comment.
each belongs_to has its own cache, not shared. Would sharing help? Knowing that an fpath: str does or doesn't belong to pytest doesn't tell us anything about whether it belongs to _pytest, right?
There was a problem hiding this comment.
I mean we have is_hypothesis_file = belongs_to(hypothesis) defined in both Hypothesis and HypoFuzz, and iirc those are cached separately. Not a big deal either way, of course.
There was a problem hiding this comment.
Ah, right. Yeah, we probably should make belongs_to share a cache, and make it lru while we're at it.
| return getattr(cd.provider, f"draw_{choice_type}")(**kwargs) | ||
|
|
||
|
|
||
| class HypofuzzProvider(PrimitiveProvider): |
There was a problem hiding this comment.
Just noting that #36 will involve going notably further, in putting much of the corpus logic etc. into the provider, and setting lifetime = "test_function". That shouldn't be in this PR, just want to avoid any confusion for future readers.
There was a problem hiding this comment.
We'll see how much we can move into the provider...Hypofuzz wants control over e.g. intercepting interesting results, and how many examples are generated, so I'm not sure we can fully move to settings(backend="hypofuzz")(test_function)(**extra_kw). I think we either give up control over control flow from hypofuzz, or add provider hooks that alter control flow (on_failing_example_found which can raise a control flow exception "actually, I don't want you to shrink this"). I'd prefer not to do the latter; I think provider should be almost entirely read-only, not explicitly writing back to hypothesis control flow.
Fully moving to a provider could be ok if we accept the flow is "when a process fuzzes f, run a constant 1k examples and then defer back to our process pool, and any failures are shrunk immediately".
...except now that I write this, I don't know how we would run a distill phase with this proposed structure. We'd need to keep a reference to the underlying function/kwargs around, and avoiding that is kinda the point of the backend.
2 participants
Ditches the buffer mutator in favor of an (extremely simple) tcs mutator.
TBH I have relatively low confidence that the logic in
corpus.pyis correct after this change. I see more test writing in my future...