Reproduce testcase locally #4976

PauloVLB · 2025-10-08T12:52:01Z

This PR adds a new reproduce command to the butler script, enabling local reproduction of a ClusterFuzz testcase. This functionality is a foundational component that will be migrated to a new, standalone CLI tool in the future. The current implementation requires a local clone of the ClusterFuzz repository with all dependencies configured.

Changes Made

A new command was added to the butler script that reproduces a testcase. Given a testcase-id and a config-dir, the command performs the following actions:

Fetches the specified testcase.
Sets up the corresponding job and fuzzer environment.
Sets up the required build and checks if it is a "bad" build.
Runs test_for_crash_with_retries to check for a crash.
If a crash is found, it then runs test_for_reproducibility and displays the final output.

Note: Support for testcase exceptions (e.g., data bundles and launch scripts) is out of scope for this PR and will be addressed in follow-up work.

How to Test Manually

1. Authentication

First, you must authenticate using Google Cloud Application Default Credentials. Execute the following command in your terminal:

gcloud auth application-default login

2. Usage

Once authenticated, run the reproduce command with the following format:

python butler.py reproduce --config-dir=<path_to_config> --testcase-id=<testcase_id>

Important: Please ensure the testcase you are using does not require a launch script, as this is not yet supported.

Testing

Unit tests for this feature have been added in /src/clusterfuzz/_internal/tests/core/local/butler/reproduce_test.py.

Future Work

The following improvements are planned as follow-ups to this PR:

Validate the usage with fuzzers that have data bundles.
Prevent the fuzzer from being downloaded again if it already exists locally.
Add support for fuzzers with launch scripts.

butler.py

ViniciustCosta

Great job overall! Thanks for making the code very robust and thoroughly adding types and docstrings.

I'm still a bit worried about running untrusted code locally though, but it is something to discuss in further steps.

ViniciustCosta · 2025-10-09T12:41:34Z

src/local/butler/reproduce.py

+  os.environ['CONFIG_DIR_OVERRIDE'] = os.path.abspath(args.config_dir)
+  local_config.ProjectConfig().set_environment()
+  environment.set_bot_environment()
+  os.environ['LOG_TO_CONSOLE'] = 'True'


It'd be better to create a method in environment module to set these two env vars (e.g., set_local_log_only).

Since this is not done in any script currently, I think we should set it in butler.py (or run.py) possibly with an argument to let the logs go to GCP it if the user wants to.

We could pass an argument to

clusterfuzz/butler.py

Line 432 in fb94beb

def _setup():

And set it there, what do you think? Something like

def _setup(args): """Set up configs and import paths.""" # ... if args.local_logging: from clusterfuzz._internal.system import environment environment.set_local_log_only()

I've addressed this in #4988

ViniciustCosta · 2025-10-09T12:50:12Z

src/local/butler/reproduce.py

+  Args:
+    args: Parsed command-line arguments.
+  """
+  os.environ['CONFIG_DIR_OVERRIDE'] = os.path.abspath(args.config_dir)


Nit: it would be nice to have all these in a setup method (e.g., _setup_reproduce).

Agreed. Addressing in #4988

ViniciustCosta · 2025-10-09T12:55:59Z

src/local/butler/reproduce.py

+from clusterfuzz._internal.datastore import data_handler
+from clusterfuzz._internal.datastore import data_types
+from clusterfuzz._internal.datastore import ndb_init
+from clusterfuzz._internal.datastore.data_types import Fuzzer


We usually don't import classes, just modules. (from ...datastore import data_type).
go/pystyle#imports

Thanks! Addressing in #4988

ViniciustCosta · 2025-10-09T12:58:21Z

src/local/butler/reproduce.py

+  Returns:
+    True if setup was successful, False otherwise.
+  """
+  fuzzer: Optional[Fuzzer] = data_types.Fuzzer.query(


Thanks for typing the code! However, for internal variables, we try to type it only when it is very difficult to infer its type (here the type is in the query itself), as typing all vars would hinder readability and python's flexibility.

go/pystyle#typing-variables

Also, for Python 3.10+ it is recommended to use the union type | instead of the Optional syntax.
go/pystyle#none-type

Thanks! Addressing in #4988

ViniciustCosta · 2025-10-09T13:06:16Z

src/local/butler/reproduce.py

+_EXECUTABLE_PERMISSIONS = 0o750
+
+
+def _setup_fuzzer(fuzzer_name: str) -> bool:


I might be oversimplifying things, but I wonder if we could have used methods already available for this. It seems that we are "reinventing the wheel" having to create another setup_fuzzer method, which I imagine copies a lot of the work done in other setup methods.

Maybe, as this is a local thing, we could create a centralized setup_local_fuzzer in some other utils/fuzzer module?

You're right, there is significant overlap with the existing logic in setup.py.

The key reason for not reusing it directly is that its flow is designed for untrusted tasks and ultimately downloads things from a signed URL:

clusterfuzz/src/clusterfuzz/_internal/bot/tasks/setup.py

Line 624 in fb94beb

if not storage.download_signed_url_to_file(update_input.fuzzer_download_url,

which we don't want to do locally, right?

My approach instead follows the pattern used by trusted tasks (like unpack_task), which use the blobs module for direct downloads.

I agree that centralizing this logic into a setup_local_fuzzer is a great idea, and I can refactor it into a shared utils module.

As discussed offline, I will add setup_local_fuzzer and setup_local_testcase in the existing setup.py. Doing that in #4988

ViniciustCosta · 2025-10-09T13:12:16Z

src/local/butler/reproduce.py

+
+  try:
+    _, testcase_file_path = setup._get_testcase_file_and_path(testcase)
+    if not blobs.read_blob_to_disk(testcase.fuzzed_keys, testcase_file_path):


I might be missing something, but what about trying to use the minimized_keys instead of the fuzzed_keys? How does this work for other tasks, such as progression task?

It uses

clusterfuzz/src/clusterfuzz/_internal/bot/tasks/setup.py

Line 342 in fb94beb

def _get_testcase_key_and_archive_status(testcase):

which checks whether the testcase has minimized keys or not, if so, returns the minimized key. I will reproduce this behavior here. Thanks!

ViniciustCosta · 2025-10-09T13:15:09Z

src/local/butler/reproduce.py

+  Args:
+    args: Parsed command-line arguments.
+  """
+  testcase: Optional[Testcase] = data_handler.get_testcase_by_id(


ViniciustCosta · 2025-10-09T13:15:17Z

src/local/butler/reproduce.py

+    logs.error(f'Testcase with ID {args.testcase_id} not found.')
+    return
+
+  job: Optional[Job] = data_types.Job.query(


ViniciustCosta · 2025-10-09T13:15:51Z

src/local/butler/reproduce.py

+    logs.error(f'Failed to setup fuzzer {testcase.fuzzer_name}. Exiting.')
+    return
+
+  ok, testcase_file_path = _setup_testcase_locally(testcase)


I think ok is not a very insightful variable name.

Thanks! Addressing in #4988

ViniciustCosta · 2025-10-09T13:19:20Z

src/local/butler/reproduce.py

+    logs.error(f'Fuzzer {fuzzer_name} not found.')
+    return False
+
+  environment.set_value('UNTRUSTED_CONTENT', fuzzer.untrusted_content)


We should add a warning to users about running untrusted code on their machines.

Thanks! Addressing in #4988

vitaliset

Thanks for working on this feature! :)

+1 to @ViniciustCosta comments.

vitaliset · 2025-10-11T23:29:08Z

src/local/butler/reproduce.py

+    shell.clear_testcase_directories()
+  except Exception as e:
+    logs.error(f'Error clearing testcase directories: {e}')
+    return False, None


As discussed offline, using a separate ok boolean in a return, such as (ok, value), is generally unneeded. It complicates the return type hint and is redundant. Returning None to signal failure is idiomatic in Python. If value can be None to indicate an issue, the ok boolean is superfluous because value is None already conveys the failure state. For truly exceptional failure conditions, raising custom exceptions is often the most Pythonic approach, as this allows the caller to handle different failure types distinctly.

Also, we don't use this "status return pattern" in the codebase, so this would make it non-standard. Please fix this in your follow-up refactor. :)

Thanks! Fixed in #4988

vitaliset · 2025-10-11T23:30:17Z

src/clusterfuzz/_internal/tests/core/local/butler/reproduce_test.py

+    self.mock.open.return_value.__enter__.return_value = self.mock_archive_reader
+
+    # Common mock fuzzer object
+    self.mock_fuzzer = mock.MagicMock(spec=Fuzzer)


Always use mock.create_autospec with instance=True and spec_set=True. This makes mocks safer by ensuring they mimic the original object's API, catching errors if methods are misspelled or called with incorrect signatures. This applies generally in the file.

go/python-tips/049

Huge thanks for the tip! Applying that on #4988

vitaliset · 2025-10-11T23:38:57Z

src/clusterfuzz/_internal/tests/core/local/butler/reproduce_test.py

@@ -0,0 +1,391 @@
+# Copyright 2025 Google LLC


Replacing your code’s dependencies with mocks can make unit tests easier to write and faster to run. However, among other problems, using mocks can lead to tests that are less effective at catching bugs. go/tott/697

It's okay to use mocks, but always reflect on whether there is a better option: go/choose-test-double

Writing this comment as a general advice! :)

PauloVLB added 10 commits September 24, 2025 17:28

wip: create reproduce script in butler

a4c4ea7

wip: adds new option in butler to call reproduce script

cd63614

wip: sets up the build and starting to setup the testcase

28c7b39

wip: downloads the testcase locally and trys to run

805258d

wip: use crash app args and adds support for non-builtin fuzzers

3ddd933

wip: start refactoring reproduce

59f6ad0

wip: add tests and lint reproduce

14f6395

wip: finish testing reproduce

27820cb

refactor: lint and add docstrings to tests

7aef915

Merge branch 'master' into reproduce_testcase_locally

9364a11

PauloVLB requested review from ViniciustCosta, decoNR, javanlacerda and vitaliset October 8, 2025 12:52

PauloVLB marked this pull request as ready for review October 8, 2025 12:54

javanlacerda reviewed Oct 8, 2025

View reviewed changes

butler.py Show resolved Hide resolved

javanlacerda approved these changes Oct 8, 2025

View reviewed changes

butler.py Show resolved Hide resolved

javanlacerda merged commit 893e97e into master Oct 8, 2025
9 checks passed

javanlacerda deleted the reproduce_testcase_locally branch October 8, 2025 16:58

ViniciustCosta reviewed Oct 9, 2025

View reviewed changes

vitaliset reviewed Oct 11, 2025

View reviewed changes

PauloVLB mentioned this pull request Oct 15, 2025

Refactor reproduce #4988

Open

		_EXECUTABLE_PERMISSIONS = 0o750


		def _setup_fuzzer(fuzzer_name: str) -> bool:

Reproduce testcase locally #4976

Reproduce testcase locally #4976

Conversation

PauloVLB commented Oct 8, 2025

Changes Made

How to Test Manually

1. Authentication

2. Usage

Testing

Future Work

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ViniciustCosta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vitaliset left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants