diff --git a/infra/experimental/chronos/README.md b/infra/experimental/chronos/README.md index 71e05492496e..4a163960d504 100644 --- a/infra/experimental/chronos/README.md +++ b/infra/experimental/chronos/README.md @@ -5,9 +5,29 @@ and analysis of projects' testing infrastructure. This is used by projects, e.g. [OSS-Fuzz-gen](https://github.com/google/oss-fuzz-gen) to help speed up valuation processes during fuzzing harness generation. -Chronos is focused on two features, rebuilding projects fast and running the tests of a given project. +At the core, Chronos relies on caching containers after project build, in order +to enable fast rebuilding of a project following minor patches, and also enable +running of the tests in a given project. To support this, Chronos creates a snapshot +of a the docker container given project post build completion. This means that all `.o` files, generated +configuations etc. persist in the docker container. These artifacts are then +leverage for future "replay builds" where only a minor part of the project has changed, +e.g. due to some patching on the project. This patching could be e.g. minor adjustments +to the fuzzing harness source code e.g. by [oss-fuzz-gen](https://github.com/google/oss-fuzz-gen). -## Rebuilding projects fast +As such, at the core of Chronos are cached containers that are generated by taking a +snapshot of the container of a project post project build. + +Chronos is focused on two features, rebuilding projects fast and running the +tests of a given project. + + +## Chronos features: fast rebuilding and running the tests of a project + +### CLI interface for Chronos + +The default route to validating Chronos is using the CLI available in `infra/experimental/chronos/manager.py` + +### Chronos feature: Fast rebuilding Chronos enables rebuilding projects efficiently in contexts where only a small patch needs to be evalualted in the target. This is achieved by running a replay build script @@ -17,24 +37,28 @@ of the original `build.sh`. The replay build scripts are constructed in two ways: manually or automatically. -### Automated rebuilds +#### Option 1: Automated rebuilds -Chronos support automated rebuilding by: +Chronos support automated rebuilding. This is meant as a generic mechanism to enable Chronos support for projects by default. This is achieved by: 1. Calling into a `replay_build.sh` script during the building inside the container [here](https://github.com/google/oss-fuzz/blob/206656447b213fb04901d15122692d8dd4d45312/infra/base-images/base-builder/compile#L292-L296) 2. The `replay_build.sh` calls into `make_build_replayable.py`: [here](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/replay_build.sh) 3. `make_build_replayable.py` adjusts the build environment to wrap around common commands, to avoid performing a complete run of `build.sh`: [here](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/make_build_replayable.py). -### Manually provided replay builds +The automated rebuilding works in combination with [Ccache](https://ccache.dev/), in order to facilitate cachine of e.g. `.o` files. +This means that during rebuild mode as long as we have a cacche, we don't need to e.g. run `configure` again and will only have to +rebuild the changed source code. + +#### Option 2: Manually provided replay builds `replay_build.sh` above, is simply just a wrapper script around `build.sh` that aims to enable fast rebuilding of the project. This `replay_build.sh` can, however, be overwritten in the Dockerfile -of the project's builder image. Examples of this is [php](https://github.com/google/oss-fuzz/blob/206656447b213fb04901d15122692d8dd4d45312/projects/php/replay_build.sh#L1) and [ffmpeg](https://github.com/google/oss-fuzz/blob/master/projects/ffmpeg/replay_build.sh#L1). +of the project's builder image to support a custom approach to fast rebuilding. Two examples of this is [php](https://github.com/google/oss-fuzz/blob/206656447b213fb04901d15122692d8dd4d45312/projects/php/replay_build.sh#L1) and [ffmpeg](https://github.com/google/oss-fuzz/blob/master/projects/ffmpeg/replay_build.sh#L1). Providing a manual `replay_build.sh` is likely more efficient at build time and can help speed up the process. Automated replay build scripts can also be erroneous. -### Testing the validity of a replay build +#### Testing the validity of a replay build The Chronos manager can use the `manager.py` to validate the validity of a replay build for a given project: @@ -46,96 +70,51 @@ python3 infra/experimental/chronos/manager.py check-test tinyobjloader If the above command fails for the relevant project, then the replay build feature does not work for the given project. -## Running tests of a project +### Chronos feature: Running tests of a project The second part of Chronos is a feature to enable running the tests of a given project. This is done by way of a script `run_tests.sh`. Samples of this script include [jsonnet](https://github.com/google/oss-fuzz/blob/master/projects/jsonnet/run_tests.sh#L1) and [tinyobjloader](https://github.com/google/oss-fuzz/blob/master/projects/tinyobjloader/run_tests.sh#L1). -### Testing the validity of run_tests.sh +#### Run tests constraints + +1. The `run_tests.sh` main task is to run the tests of a project and return `0` upon success and non-null otherwise. +2. The `run_tests.sh` script must leave the main repository in the state as it was prior to the execution of `run_tests.sh` relative to `git diff` (or similar diff features of other version control systems). + +#### Testing the validity of run_tests.sh The Chronos manager can use the `manager.py` to validate the validity of a `run_tests.sh` script: ```sh -python3 infra/experimental/chronos/manager.py +python3 infra/experimental/chronos/manager.py check-tests json-c ``` -**Running tests of a project** - -## Pre-built images. - -Daily pre-built images are available at: - -- `us-central1-docker.pkg.dev/oss-fuzz/oss-fuzz-gen/-ofg-cached-address` -- `us-central1-docker.pkg.dev/oss-fuzz/oss-fuzz-gen/-ofg-cached-coverage` - -They can be used as drop-in replacements for the usual `gcr.io/oss-fuzz/` images. - -These images are generated in 2 ways: -- (Preferred) [Generate](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/bash_parser.py) - a replay build script that can be re-run alongside existing build artifacts, - leveraging existing build system mechanisms to avoid rebuilding (e.g. running - `make` twice should not actually rebuild everything). This is error-prone, so - we validate the script works by running it. -- (Fallback, if the replay build script didn't work). We leverage - [ccache](https://ccache.dev/), to provide a compiler cache. This is often not - as fast as the replay build script, because some project builds spend - significant time doing non-compiler tasks (e.g. checking out submodules, - running configure scripts). - -Note: this mechanism does not work for every single OSS-Fuzz project today. The -resulting image may either: -- Not provide much performance improvement compared with a normal image, or -- Not exist at all (if neither approach worked). - -Stats from a recent run: -(Feb 3 2025). - -## Usage locally - -**Example 1: htslib** +### Constraints imposed on replay_build.sh and run_tests.sh -From the OSS-Fuzz root +At the core of chronos are the two scripts `replay_build.sh` and `run_tests.sh`. We have a default +mechanism for `replay_build.sh` so it's not strictly necessary to have a custom one, although it will +likely improve speed and maybe correctness by providing one. -```sh -$ RUN_ALL=1 ./infra/experimental/chronos/build_cache_local.sh htslib c address -... -... -Vanilla compile time: -17 -Replay worked -Replay compile time: -2 -Ccache compile time: -9 -``` +There are three stages of the Chronos workflow: +1. The cached containers represent the state of a build container after a successful project build. +2. The `replay_build.sh` is able to rebuild a given project from the state of a cached container. +3. The `run_tests.sh` script is able to run the tests of a given project. This should be able to succeed following the running of a `replay_build.sh`. -## Check tests +The stages (2) and (3) must both support running without network connectivity. +Specifically, this means that the `replay_build.sh` must not do tasks e.g. fetch +dependencies, download corpus, or anything of this nature. Similarly, the `run_tests.sh` +must be able to operate completely in a closed network environment. -Another feature of Chronos is the ability to run tests in a replayed build. -This requires `run_tests.sh` to be available in the cached image at -`$SRC/run_tests.sh`. -Sample running: +## Pre-built images. -``` -$ git clone https://github.com/google/oss-fuzz -$ cd oss-fuzz -$ ./infra/experimental/chronos/check_tests.sh jsonnet -... -... -100% tests passed, 0 tests failed out of 10 - -Total Test time (real) = 119.80 sec -``` +Chronos cached images are build daily, and pre-built images are available at: -In order ot make the above work, the general approach is to have a -`run_tests.sh` script in the OSS-Fuzz project's folder, which is copied into -the main image. +- `us-central1-docker.pkg.dev/oss-fuzz/oss-fuzz-gen/-ofg-cached-address` +- `us-central1-docker.pkg.dev/oss-fuzz/oss-fuzz-gen/-ofg-cached-coverage` -Notice that the `run_tests.sh` is run from a cached image, meaning the -`run_tests.sh` is run after a run of building fuzzers. +They can be used as drop-in replacements for the usual `gcr.io/oss-fuzz/` images. \ No newline at end of file diff --git a/infra/experimental/chronos/manager.py b/infra/experimental/chronos/manager.py index b810b0ac733e..3d12dae2adbd 100644 --- a/infra/experimental/chronos/manager.py +++ b/infra/experimental/chronos/manager.py @@ -21,45 +21,11 @@ import json import subprocess -import requests - import bad_patch import logic_error_patch logger = logging.getLogger(__name__) -OSS_FUZZ_BUILD_HISTORY_URL = ( - 'https://oss-fuzz-build-logs.storage.googleapis.com/status.json') -OSS_FUZZ_BUILD_HISTORY = [] - -RUN_TEST_HEURISTIC_0 = 'make test' -RUN_TEST_HEURISTIC_1 = 'make tests' -RUN_TEST_HEURISTIC_2 = 'make check' - -RUN_TESTS_TO_TRY = [ - RUN_TEST_HEURISTIC_0, RUN_TEST_HEURISTIC_1, RUN_TEST_HEURISTIC_2 -] - - -def _get_oss_fuzz_build_status(project): - """Returns the build status of a project in OSS-Fuzz.""" - if not OSS_FUZZ_BUILD_HISTORY: - # Load the build history from a file or other source. - # This is a placeholder for actual implementation. - build_status = requests.get(OSS_FUZZ_BUILD_HISTORY_URL, timeout=30) - OSS_FUZZ_BUILD_HISTORY.extend( - json.loads(build_status.text).get('projects', [])) - - for project_data in OSS_FUZZ_BUILD_HISTORY: - if project_data['name'] == project: - logger.info('Found project %s in OSS-Fuzz build history.', project) - return project_data.get('history', [{ - 'success': False - }])[0].get('success', False) - - logger.info('Project %s not found in OSS-Fuzz build history.', project) - return False - def _get_project_cached_named(project, sanitizer='address'): """Gets the name of the cached project image.""" @@ -164,7 +130,7 @@ def build_cached_project(project, cleanup=True, sanitizer='address'): return True -def check_cached_replay(project, sanitizer='address', integrity_test=False): +def check_cached_replay(project, sanitizer='address', integrity_check=False): """Checks if a cache build succeeds and times is.""" build_project_image(project) if not build_cached_project(project, sanitizer=sanitizer): @@ -190,7 +156,7 @@ def check_cached_replay(project, sanitizer='address', integrity_test=False): '-c', ] - if integrity_test: + if integrity_check: # Use different bad patches to test the cached replay build failed = [] for bad_patch_name, bad_patch_map in bad_patch.BAD_PATCH_GENERATOR.items(): @@ -238,7 +204,7 @@ def check_cached_replay(project, sanitizer='address', integrity_test=False): def check_test(project, sanitizer='address', run_full_cache_replay=False, - integrity_test=False, + integrity_check=False, stop_on_failure=False): """Run the `run_tests.sh` script for a specific project. Will build a cached container first.""" @@ -276,7 +242,7 @@ def check_test(project, '/bin/bash', '-c', ] - if integrity_test: + if integrity_check: integrity_checks = [] # Patch the code with some logic error and see if build_test able to detect @@ -395,127 +361,6 @@ def check_run_tests_script(project, project) -def _get_project_language(project): - """Returns the language of the project.""" - project_path = os.path.join('projects', project) - if not os.path.isdir(project_path): - return '' - - # Check for a .lang file or similar to determine the language - project_yaml = os.path.join(project_path, 'project.yaml') - if os.path.exists(project_yaml): - with open(project_yaml, 'r', encoding='utf-8') as f: - for line in f: - if 'language' in line: - return line.split(':')[1].strip() - - # Default to C++ if no specific language file is found - return '' - - -def _autogenerate_run_tests_script(project, container_output): - """Autogenerate `run_tests.sh` for a project.""" - project_path = os.path.join('projects', project) - run_tests_script = os.path.join(project_path, 'run_tests.sh') - - for run_test_script in RUN_TESTS_TO_TRY: - with open(run_tests_script, 'w', encoding='utf-8') as f: - f.write('#!/bin/bash\n') - f.write('set -eux\n') - f.write(run_test_script + '\n') - f.write(f'echo "Running tests for project: {project}"\n') - # Add more commands as needed to run tests - os.chmod(run_tests_script, 0o755) - logger.info('Created run_tests.sh for %s', project) - - # Adjust the Dockerfile to copy it in - dockerfile_path = os.path.join(project_path, 'Dockerfile') - - add_run_tests = False - with open(dockerfile_path, 'r', encoding='utf-8') as f: - if 'COPY run_tests.sh' not in f.read(): - add_run_tests = True - if add_run_tests: - with open(dockerfile_path, 'a', encoding='utf-8') as f: - f.write('\n# Copy the autogenerated run_tests.sh script\n') - f.write('COPY run_tests.sh $SRC/run_tests.sh\n') - f.write('RUN chmod +x $SRC/run_tests.sh\n') - - succeeded = check_test(project, container_output=container_output) - - # If it succeeded initially, then make sure that it actually succeeds - # to generate a coverage report. if it does not, then it means the - # generation actually failed. - coverage_success = extract_test_coverage(project) - if not coverage_success: - logger.error('Coverage generation failed for %s', project) - succeeded = False - - success_file = os.path.join(project_path, 'run_tests.succeeded') - with open(success_file, 'a', encoding='utf-8') as f: - f.write(f'Auto-generation succeeded: {succeeded}\n') - if succeeded: - logger.info('Autogenerated run_tests.sh for %s successfully.', project) - break - - -def autogen_projects(apply_filtering=False, - max_projects_to_try=1, - container_output='stdout', - projects_to_target=[]): - """Autogenerate `run_tests.sh` for all projects.""" - if projects_to_target: - projects = projects_to_target - else: - projects = os.listdir('projects') - projects_tries = 0 - for project in projects: - if projects_tries >= max_projects_to_try: - logger.info('Reached maximum number of projects to try: %d', - max_projects_to_try) - break - - project_path = os.path.join('projects', project) - if not os.path.isdir(project_path): - continue - - # Ensure the project language is C or C++ - if _get_project_language(project).lower() not in ['c', 'c++']: - continue - - run_tests_script = os.path.join(project_path, 'run_tests.sh') - if os.path.exists(run_tests_script): - logger.info('Skipping %s, run_tests.sh already exists.', project) - continue - - if apply_filtering: - # Apply filtering logic to increase performance - build_script = os.path.join(project_path, 'build.sh') - if not os.path.exists(build_script): - logger.warning('Skipping %s, build.sh does not exist.', project) - continue - with open(build_script, 'r', encoding='utf-8') as f: - lines = f.readlines() - # Filter out lines that are not relevant for the test script - filtered_lines = [line for line in lines if 'make' in line] - if not filtered_lines: - logger.warning('Skipping %s, no relevant lines found in build.sh.', - project) - continue - - # It only makes sense to autogenerate if the project actually builds, so - # query OSS-Fuzz to make sure the most recent build was successful. - - if not _get_oss_fuzz_build_status(project): - logger.warning('Skipping %s, most recent build was not successful.', - project) - continue - - projects_tries += 1 - logger.info('Autogenerating run_tests.sh for %s', project) - _autogenerate_run_tests_script(project, container_output) - - def extract_test_coverage(project): """Extract code coverage report from run_tests.sh script.""" build_project_image(project) @@ -556,30 +401,19 @@ def extract_test_coverage(project): def _cmd_dispatcher_check_test(args): check_test(args.project, args.sanitizer, args.run_full_cache_replay, - args.check_patch_integrity, args.stop_on_failure) + args.integrity_check, args.stop_on_failure) def _cmd_dispatcher_check_replay(args): check_cached_replay(args.project, args.sanitizer, - integrity_test=args.integrity_test) + integrity_check=args.integrity_check) def _cmd_dispatcher_build_cached_image(args): build_cached_project(args.project, sanitizer=args.sanitizer) -def _cmd_dispatcher_autogen_tests(args): - autogen_projects(args.apply_filtering, args.max_projects_to_try, - args.projects) - - -def _cmd_dispatcher_build_many_caches(args): - for project in args.projects: - logger.info('Building cached project: %s', project) - check_cached_replay(project, sanitizer=args.sanitizer) - - def _cmd_dispatcher_extract_coverage(args): extract_test_coverage(args.project) @@ -625,24 +459,24 @@ def parse_args(): 'If set, will run the full cache replay instead of just checking the script.' ) check_test_parser.add_argument( - '--check-patch-integrity', + '--integrity-check', action='store_true', help= 'If set, will patch and test with logic errors to ensure build integrity.' ) - check_replay_script_parser = subparsers.add_parser( - 'check-replay-script', + check_replay_parser = subparsers.add_parser( + 'check-replay', help='Checks if the replay script works for a specific project.') - check_replay_script_parser.add_argument( - 'project', help='The name of the project to check.') - check_replay_script_parser.add_argument( + check_replay_parser.add_argument('project', + help='The name of the project to check.') + check_replay_parser.add_argument( '--sanitizer', default='address', help='The sanitizer to use for the cached build (default: address).') - check_replay_script_parser.add_argument( - '--integrity-test', + check_replay_parser.add_argument( + '--integrity-check', action='store_true', help='If set, will test the integrity of the replay script.') @@ -676,50 +510,6 @@ def parse_args(): default='stdout', help='How to handle output from the container. ') - autogen_tests_parser = subparsers.add_parser( - 'autogen-tests', - help='Tries to autogenerate `run_tests.sh` for projects.') - autogen_tests_parser.add_argument( - '--apply-filtering', - action='store_true', - help=('If set, applies filtering to increase performance but ' - 'tests on fewer projects that are more likely to succeed.')) - autogen_tests_parser.add_argument( - '--max-projects-to-try', - type=int, - default=1, - help='Maximum number of projects to try (default: 1).') - autogen_tests_parser.add_argument( - '--container-output', - choices=['silent', 'file', 'stdout'], - default='stdout', - help='How to handle output from the container. ') - autogen_tests_parser.add_argument( - '--projects', - default='', - nargs='+', - help=('The name of the projects to autogenerate tests for. ' - 'If not specified, all projects will be considered.')) - - build_many_caches = subparsers.add_parser( - 'build-many-caches', - help='Builds cached images for multiple projects in parallel.') - build_many_caches.add_argument( - '--projects', - nargs='+', - required=True, - help='List of projects to build cached images for.') - build_many_caches.add_argument( - '--sanitizer', - default='address', - help='The sanitizer to use for the cached build (default: address).') - build_many_caches.add_argument( - '--container-output', - choices=['silent', 'file', 'stdout'], - default='stdout', - help='How to handle output from the container. ') - build_many_caches.add_argument('--silent-replays', action='store_true') - extract_coverage_parser = subparsers.add_parser( 'extract-test-coverage', help='Extract code coverage reports from run_tests.sh script') @@ -737,10 +527,8 @@ def main(): dispatch_map = { 'check-test': _cmd_dispatcher_check_test, - 'check-replay-script': _cmd_dispatcher_check_replay, + 'check-replay': _cmd_dispatcher_check_replay, 'build-cached-image': _cmd_dispatcher_build_cached_image, - 'autogen-tests': _cmd_dispatcher_autogen_tests, - 'build-many-caches': _cmd_dispatcher_build_many_caches, 'extract-test-coverage': _cmd_dispatcher_extract_coverage, 'check-run-tests-script': _cmd_check_run_tests_script }