Skip to content

Commit 9c2b825

Browse files
Merge branch 'master' into poppler-fix-build
2 parents de37b20 + 2a8c8d3 commit 9c2b825

File tree

699 files changed

+20742
-5249
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

699 files changed

+20742
-5249
lines changed
Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
#
15+
################################################################################
16+
17+
name: 'Ubuntu Version Sync'
18+
19+
on:
20+
pull_request:
21+
types: [opened, synchronize, reopened]
22+
23+
jobs:
24+
check-sync:
25+
runs-on: ubuntu-latest
26+
steps:
27+
- name: 'Checkout code'
28+
uses: actions/checkout@v4
29+
with:
30+
# Fetch all history so we can diff against the base branch.
31+
fetch-depth: 0
32+
33+
- name: 'Run sync check'
34+
run: |
35+
set -e
36+
37+
MODIFIED_FILES=$(git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }})
38+
echo "Checking for synchronized file changes..."
39+
echo "Modified files in this PR:"
40+
echo "$MODIFIED_FILES"
41+
42+
ERRORS=""
43+
44+
# Define the mapping of legacy files to their versioned counterparts.
45+
# Format: "legacy_file;versioned_file_pattern"
46+
# The pattern uses {version} which will be replaced with "ubuntu-20-04" and "ubuntu-24-04".
47+
# For Dockerfiles, the pattern is different from scripts.
48+
declare -A LEGACY_DOCKERFILES
49+
LEGACY_DOCKERFILES["infra/base-images/base-builder-fuzzbench/Dockerfile"]="infra/base-images/base-builder-fuzzbench/{version}.Dockerfile"
50+
LEGACY_DOCKERFILES["infra/base-images/base-builder-swift/Dockerfile"]="infra/base-images/base-builder-swift/{version}.Dockerfile"
51+
LEGACY_DOCKERFILES["infra/base-images/base-builder/Dockerfile"]="infra/base-images/base-builder/{version}.Dockerfile"
52+
LEGACY_DOCKERFILES["infra/base-images/base-clang/Dockerfile"]="infra/base-images/base-clang/{version}.Dockerfile"
53+
LEGACY_DOCKERFILES["infra/base-images/base-runner/Dockerfile"]="infra/base-images/base-runner/{version}.Dockerfile"
54+
55+
declare -A LEGACY_SCRIPTS
56+
LEGACY_SCRIPTS["infra/base-images/base-builder-fuzzbench/fuzzbench_install_dependencies"]="infra/base-images/base-builder-fuzzbench/fuzzbench_install_dependencies_{version}"
57+
LEGACY_SCRIPTS["infra/base-images/base-builder/install_deps.sh"]="infra/base-images/base-builder/install_deps_{version}.sh"
58+
LEGACY_SCRIPTS["infra/base-images/base-builder/install_swift.sh"]="infra/base-images/base-builder/install_swift_{version}.sh"
59+
LEGACY_SCRIPTS["infra/base-images/base-builder/precompile_honggfuzz"]="infra/base-images/base-builder/precompile_honggfuzz_{version}"
60+
LEGACY_SCRIPTS["infra/base-images/base-clang/checkout_build_install_llvm.sh"]="infra/base-images/base-clang/checkout_build_install_llvm_{version}.sh"
61+
LEGACY_SCRIPTS["infra/base-images/base-runner/install_deps.sh"]="infra/base-images/base-runner/install_deps_{version}.sh"
62+
63+
VERSIONS=("ubuntu-20-04" "ubuntu-24-04")
64+
65+
# Check Dockerfiles
66+
for legacy_file in "${{!LEGACY_DOCKERFILES[@]}}"; do
67+
if echo "$MODIFIED_FILES" | grep -q "^${legacy_file}$"; then
68+
echo "Legacy file changed: $legacy_file. Verifying counterparts..."
69+
for version in "${{VERSIONS[@]}}"; do
70+
pattern=${{LEGACY_DOCKERFILES[$legacy_file]}}
71+
versioned_file="${{pattern/{{version}}/$version}}"
72+
if ! echo "$MODIFIED_FILES" | grep -q "^${{versioned_file}}$"; then
73+
ERRORS+="\n- Legacy file '${legacy_file}' was changed, but its counterpart '${versioned_file}' was not."
74+
fi
75+
done
76+
fi
77+
done
78+
79+
# Check Scripts
80+
for legacy_file in "${{!LEGACY_SCRIPTS[@]}}"; do
81+
if echo "$MODIFIED_FILES" | grep -q "^${legacy_file}$"; then
82+
echo "Legacy script changed: $legacy_file. Verifying counterparts..."
83+
for version in "${{VERSIONS[@]}}"; do
84+
pattern=${{LEGACY_SCRIPTS[$legacy_file]}}
85+
versioned_file="${{pattern/{{version}}/$version}}"
86+
if ! echo "$MODIFIED_FILES" | grep -q "^${{versioned_file}}$"; then
87+
ERRORS+="\n- Legacy script '${legacy_file}' was changed, but its counterpart '${versioned_file}' was not."
88+
fi
89+
done
90+
fi
91+
done
92+
93+
if [ -n "$ERRORS" ]; then
94+
echo -e "\n\e[31mError: Found synchronization issues between legacy and versioned files.\e[0m"
95+
echo -e "Please update the following files to match their legacy counterparts or ensure they are included in this PR:$ERRORS"
96+
exit 1
97+
else
98+
echo -e "\n\e[32mSuccess: All modified legacy files are synchronized with their versioned counterparts.\e[0m"
99+
fi

docs/advanced-topics/bug_fixing_guidance.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ model of your project.
4747
OSS-Fuzz will report some bugs that are labeled `Reliably reproduces: NO` and
4848
these can be tricky to deal with. A non-reproducible bug is an issue that
4949
OSS-Fuzz did indeed discover, however, OSS-Fuzz is unable to reproduce the bug
50-
with `python infra/helper.py reproduce`. In general, our suggestion is to do
50+
with `python3 infra/helper.py reproduce`. In general, our suggestion is to do
5151
analysis of the bug and determine whether there in fact is an issue.
5252

5353
The non-reproducible bugs can be of varying nature. Some of these bugs will be

docs/advanced-topics/code_coverage.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ configurations, scripts, and other changes. We recommend you pull the most
2727
recent images by running the following command:
2828

2929
```bash
30-
$ python infra/helper.py pull_images
30+
$ python3 infra/helper.py pull_images
3131
```
3232

3333
## Build fuzz targets
@@ -36,8 +36,8 @@ Code coverage report generation requires a special build configuration to be
3636
used. To create a code coverage build for your project, run these commands:
3737

3838
```bash
39-
$ python infra/helper.py build_image $PROJECT_NAME
40-
$ python infra/helper.py build_fuzzers --sanitizer=coverage $PROJECT_NAME
39+
$ python3 infra/helper.py build_image $PROJECT_NAME
40+
$ python3 infra/helper.py build_fuzzers --sanitizer=coverage $PROJECT_NAME
4141
```
4242

4343
## Establish access to GCS
@@ -70,7 +70,7 @@ If you want to generate a code coverage report using the corpus aggregated on
7070
OSS-Fuzz, run this command:
7171

7272
```bash
73-
$ python infra/helper.py coverage $PROJECT_NAME
73+
$ python3 infra/helper.py coverage $PROJECT_NAME
7474
```
7575

7676
If you want to generate a code coverage report using the corpus you have
@@ -79,7 +79,7 @@ locally, copy the corpus into the
7979
target, then run this command:
8080

8181
```bash
82-
$ python infra/helper.py coverage --no-corpus-download $PROJECT_NAME
82+
$ python3 infra/helper.py coverage --no-corpus-download $PROJECT_NAME
8383
```
8484

8585
### Single fuzz target
@@ -88,14 +88,14 @@ You can generate a code coverage report for a particular fuzz target by using
8888
the `--fuzz-target` argument:
8989

9090
```bash
91-
$ python infra/helper.py coverage --fuzz-target=<fuzz_target_name> $PROJECT_NAME
91+
$ python3 infra/helper.py coverage --fuzz-target=<fuzz_target_name> $PROJECT_NAME
9292
```
9393

9494
In this mode, you can specify an arbitrary corpus location for the fuzz target
9595
(instead of the corpus downloaded from OSS-Fuzz) by using `--corpus-dir`:
9696

9797
```bash
98-
$ python infra/helper.py coverage --fuzz-target=<fuzz_target_name> \
98+
$ python3 infra/helper.py coverage --fuzz-target=<fuzz_target_name> \
9999
--corpus-dir=<my_local_corpus_dir> $PROJECT_NAME
100100
```
101101

@@ -105,15 +105,15 @@ You may want to use some of the options provided by the [llvm-cov tool], like
105105
`-ignore-filename-regex=`. You can pass these to the helper script after `--`:
106106

107107
```bash
108-
$ python infra/helper.py coverage $PROJECT_NAME -- \
108+
$ python3 infra/helper.py coverage $PROJECT_NAME -- \
109109
-ignore-filename-regex=.*code/to/be/ignored/.* <other_extra_args>
110110
```
111111

112112
If you want to specify particular source files or directories to show in the
113113
report, list their paths at the end of the extra arguments sequence:
114114

115115
```bash
116-
$ python infra/helper.py coverage zlib -- \
116+
$ python3 infra/helper.py coverage zlib -- \
117117
<other_extra_args> /src/zlib/inftrees.c /src/zlib_uncompress_fuzzer.cc /src/zlib/zutil.c
118118
```
119119

docs/advanced-topics/debugging.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ While developing your build script, it may be useful to run bash within the
1919
container:
2020

2121
```bash
22-
$ python infra/helper.py shell $PROJECT_NAME # runs /bin/bash within container
22+
$ python3 infra/helper.py shell $PROJECT_NAME # runs /bin/bash within container
2323
$ compile # runs compilation manually
2424
```
2525

@@ -34,7 +34,7 @@ image:
3434
$ cp /path/to/testcase build/out/$PROJECT_NAME
3535

3636
# Run the Docker image containing GDB.
37-
$ python infra/helper.py shell base-runner-debug
37+
$ python3 infra/helper.py shell base-runner-debug
3838
$ gdb --args /out/$PROJECT_NAME/$FUZZ_TARGET_NAME /out/$PROJECT_NAME/testcase
3939
```
4040

docs/advanced-topics/reproducing.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -72,16 +72,16 @@ can be reproduced only with a fresh image being used. Pull the latest images
7272
by running the following command:
7373

7474
```bash
75-
$ python infra/helper.py pull_images
75+
$ python3 infra/helper.py pull_images
7676
```
7777

7878
### Build the image and the fuzzers
7979

8080
Run the following commands:
8181

8282
```bash
83-
$ python infra/helper.py build_image $PROJECT_NAME
84-
$ python infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> \
83+
$ python3 infra/helper.py build_image $PROJECT_NAME
84+
$ python3 infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> \
8585
--architecture <x86_64/i386> $PROJECT_NAME
8686
```
8787

@@ -102,7 +102,7 @@ If you can't reproduce a particular bug building for x86_64, try building for i3
102102
After you build an image and a fuzzer, you can reproduce a bug by running the following command:
103103

104104
```bash
105-
$ python infra/helper.py reproduce $PROJECT_NAME <fuzz_target_name> <testcase_path>
105+
$ python3 infra/helper.py reproduce $PROJECT_NAME <fuzz_target_name> <testcase_path>
106106
```
107107

108108
For example, to build the [libxml2](https://github.com/google/oss-fuzz/tree/master/projects/libxml2)
@@ -111,19 +111,19 @@ reproduce a crash testcase for a fuzzer named `libxml2_xml_read_memory_fuzzer`,
111111
you would run:
112112

113113
```bash
114-
$ python infra/helper.py build_image libxml2
115-
$ python infra/helper.py build_fuzzers --sanitizer undefined libxml2
116-
$ python infra/helper.py reproduce libxml2 libxml2_xml_read_memory_fuzzer ~/Downloads/testcase
114+
$ python3 infra/helper.py build_image libxml2
115+
$ python3 infra/helper.py build_fuzzers --sanitizer undefined libxml2
116+
$ python3 infra/helper.py reproduce libxml2 libxml2_xml_read_memory_fuzzer ~/Downloads/testcase
117117
```
118118

119119
## Reproduce using local source checkout
120120

121121
You can also mount local sources into the running container by using these commands:
122122

123123
```bash
124-
$ python infra/helper.py build_fuzzers \
124+
$ python3 infra/helper.py build_fuzzers \
125125
--sanitizer <address/memory/undefined> $PROJECT_NAME <source_path>
126-
$ python infra/helper.py reproduce $PROJECT_NAME <fuzz_target_name> <testcase_path>
126+
$ python3 infra/helper.py reproduce $PROJECT_NAME <fuzz_target_name> <testcase_path>
127127
```
128128

129129
Once you reproduce the bug, you can do the following:
@@ -144,10 +144,10 @@ Our infrastructure runs some sanity tests to make sure that your build was
144144
correctly configured, even if it succeeded. To reproduce these locally, run these commands:
145145

146146
```bash
147-
$ python infra/helper.py build_image $PROJECT_NAME
148-
$ python infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> \
147+
$ python3 infra/helper.py build_image $PROJECT_NAME
148+
$ python3 infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> \
149149
--engine <libfuzzer/afl/honggfuzz/centipede> --architecture <x86_64/i386> $PROJECT_NAME
150-
$ python infra/helper.py check_build --sanitizer <address/memory/undefined> \
150+
$ python3 infra/helper.py check_build --sanitizer <address/memory/undefined> \
151151
--engine <libfuzzer/afl/honggfuzz/centipede> --architecture <x86_64/i386> $PROJECT_NAME \
152152
<fuzz_target_name>
153153
```

docs/faq.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,7 +185,7 @@ This may happen if the Docker images fetched locally become out of sync. Make
185185
sure you run the following command to pull the most recent images:
186186

187187
```bash
188-
$ python infra/helper.py pull_images
188+
$ python3 infra/helper.py pull_images
189189
```
190190

191191
Please refer to

docs/getting-started/new-project-guide/python_lang.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ Atheris + Hypothesis harnesses.
4040
The `language` attribute must be specified.
4141

4242
```yaml
43-
language: python
43+
language: python3
4444
```
4545
4646
The only supported fuzzing engine is libFuzzer (`libfuzzer`). The supported

docs/getting-started/new_project_guide.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ by running the following commands:
6464
$ cd /path/to/oss-fuzz
6565
$ export PROJECT_NAME=<project_name>
6666
$ export LANGUAGE=<project_language>
67-
$ python infra/helper.py generate $PROJECT_NAME --language=$LANGUAGE
67+
$ python3 infra/helper.py generate $PROJECT_NAME --language=$LANGUAGE
6868
```
6969

7070
Once the template configuration files are created, you can modify them to fit your project.
@@ -365,8 +365,8 @@ You can build your docker image and fuzz targets locally, so you can test them b
365365

366366
```bash
367367
$ cd /path/to/oss-fuzz
368-
$ python infra/helper.py build_image $PROJECT_NAME
369-
$ python infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> $PROJECT_NAME
368+
$ python3 infra/helper.py build_image $PROJECT_NAME
369+
$ python3 infra/helper.py build_fuzzers --sanitizer <address/memory/undefined> $PROJECT_NAME
370370
```
371371

372372
The built binaries appear in the `/path/to/oss-fuzz/build/out/$PROJECT_NAME`
@@ -378,25 +378,25 @@ You can build your docker image and fuzz targets locally, so you can test them b
378378
2. Find failures to fix by running the `check_build` command:
379379

380380
```bash
381-
$ python infra/helper.py check_build $PROJECT_NAME
381+
$ python3 infra/helper.py check_build $PROJECT_NAME
382382
```
383383

384384
3. If you want to test changes against a particular fuzz target, run the following command:
385385

386386
```bash
387-
$ python infra/helper.py run_fuzzer --corpus-dir=<path-to-temp-corpus-dir> $PROJECT_NAME <fuzz_target>
387+
$ python3 infra/helper.py run_fuzzer --corpus-dir=<path-to-temp-corpus-dir> $PROJECT_NAME <fuzz_target>
388388
```
389389

390390
4. We recommend taking a look at your code coverage as a test to ensure that
391391
your fuzz targets get to the code you expect. This would use the corpus
392392
generated from the previous `run_fuzzer` step in your local corpus directory.
393393

394394
```bash
395-
$ python infra/helper.py build_fuzzers --sanitizer coverage $PROJECT_NAME
396-
$ python infra/helper.py coverage $PROJECT_NAME --fuzz-target=<fuzz_target> --corpus-dir=<path-to-temp-corpus-dir>
395+
$ python3 infra/helper.py build_fuzzers --sanitizer coverage $PROJECT_NAME
396+
$ python3 infra/helper.py coverage $PROJECT_NAME --fuzz-target=<fuzz_target> --corpus-dir=<path-to-temp-corpus-dir>
397397
```
398398

399-
You may need to run `python infra/helper.py pull_images` to use the latest
399+
You may need to run `python3 infra/helper.py pull_images` to use the latest
400400
coverage tools. Please refer to
401401
[code coverage]({{ site.baseurl }}/advanced-topics/code-coverage/) for detailed
402402
information on code coverage generation.

docs/reference/glossary.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ A [test input] that causes a specific bug to reproduce.
7474
Fuzzers are usually built with one or more [sanitizer](https://github.com/google/sanitizers) enabled.
7575

7676
```bash
77-
$ python infra/helper.py build_fuzzers --sanitizer undefined json
77+
$ python3 infra/helper.py build_fuzzers --sanitizer undefined json
7878
```
7979

8080
Supported sanitizers:
@@ -95,5 +95,5 @@ You can choose which configurations to automatically run your fuzzers with in `p
9595
ClusterFuzz supports fuzzing on x86_64 (aka x64) by default. However you can also fuzz using AddressSanitizer and libFuzzer on i386 (aka x86, or 32 bit) by specifying the `$ARCHITECTURE` build environment variable using the `--architecture` option:
9696

9797
```bash
98-
python infra/helper.py build_fuzzers --architecture i386 json
98+
python3 infra/helper.py build_fuzzers --architecture i386 json
9999
```

0 commit comments

Comments
 (0)