Improve thrift fuzzing support #13874
Open
+812
−60
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
mhlakhani is integrating a new project: |
mhlakhani
commented
Aug 26, 2025
| LD_LIBRARY_PATH=\"$JVM_LD_LIBRARY_PATH\":\$this_dir \ | ||
| \$this_dir/jazzer_driver --agent_path=\$this_dir/jazzer_agent_deploy.jar \ | ||
| --cp=$RUNTIME_CLASSPATH \ | ||
| --target_class=org.apache.thrift.test.fuzz.$fuzzer_basename \ |
There was a problem hiding this comment.
there's something wrong with these fuzzers and I am not sure if I'm doing something stupid (probably am) or if there's an oss-fuzz bug.
Concretely, I can build and run these fuzzers fine locally if I do e.g.
python3 infra/helper.py build_image --cache thrift-java && python3 infra/helper.py build_fuzzers thrift-java
python3 infra/helper.py run_fuzzer thrift-java ParseJSONFuzzer
this outputs something like
➜ oss-fuzz python3 infra/helper.py run_fuzzer thrift-java ParseJSONFuzzer
INFO:__main__:Running: docker run --privileged --shm-size=2g --platform linux/amd64 --rm -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e RUN_FUZZER_MODE=interactive -e HELPER=True -v /home/mhl/projects/oss-fuzz/build/out/thrift-java:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer ParseJSONFuzzer.
vm.mmap_rnd_bits = 28
/out/ParseJSONFuzzer -rss_limit_mb=2560 -timeout=25 /tmp/ParseJSONFuzzer_corpus < /dev/null
OpenJDK 64-Bit Server VM warning: Option CriticalJNINatives was deprecated in version 16.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
INFO: Loaded 268 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.sanitizers.ClojureLangHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
INFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
INFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
INFO: Loaded 12 hooks from com.code_intelligence.jazzer.sanitizers.ScriptEngineInjection
INFO: Loaded 3 hooks from com.code_intelligence.jazzer.sanitizers.ServerSideRequestForgery
INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
INFO: Loaded 6 hooks from com.code_intelligence.jazzer.sanitizers.XPathInjection
INFO: Instrumented org.apache.thrift.test.fuzz.ParseJSONFuzzer (took 55 ms, size +13%)
INFO: Instrumented org.apache.thrift.protocol.TProtocolFactory (took 1 ms, size +0%)
INFO: using inputs from: /tmp/ParseJSONFuzzer_corpus
INFO: found LLVMFuzzerCustomMutator (0x7f8ace11da00). Disabling -len_control by default.
INFO: libFuzzer ignores flags that start with '--'
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 234785997
INFO: Loaded 1 modules (512 inline 8-bit counters): 512 [0x56104cbd44a0, 0x56104cbd46a0),
INFO: Loaded 1 PC tables (512 PCs): 512 [0x56104cbaea60,0x56104cbb0a60),
INFO: 0 files found in /tmp/ParseJSONFuzzer_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol$Factory (took 3 ms, size +11%)
INFO: Instrumented org.apache.thrift.protocol.TProtocol (took 30 ms, size +5%)
INFO: Instrumented org.apache.thrift.protocol.TWriteProtocol (took 0 ms, size +0%)
INFO: Instrumented org.apache.thrift.protocol.TReadProtocol (took 0 ms, size +0%)
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol (took 34 ms, size +16%)
INFO: Instrumented org.apache.thrift.test.fuzz.FuzzTestUtils (took 3 ms, size +16%)
INFO: Instrumented org.apache.thrift.TException (took 2 ms, size +14%)
INFO: Instrumented org.apache.thrift.transport.TTransport (took 4 ms, size +14%)
INFO: New number of coverage counters: 1024
INFO: Instrumented org.apache.thrift.transport.TMemoryInputTransport (took 4 ms, size +14%)
INFO: Instrumented org.apache.thrift.transport.TEndpointTransport (took 3 ms, size +20%)
INFO: Instrumented org.apache.thrift.transport.TMemoryBuffer (took 2 ms, size +16%)
INFO: Instrumented org.apache.thrift.TConfiguration (took 2 ms, size +9%)
INFO: Instrumented org.apache.thrift.TConfiguration$Builder (took 2 ms, size +11%)
INFO: Instrumented org.apache.thrift.transport.TTransportException (took 2 ms, size +8%)
INFO: Instrumented org.apache.thrift.protocol.TProtocolException (took 4 ms, size +8%)
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol$JSONBaseContext (took 1 ms, size +16%)
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol$JSONPairContext (took 1 ms, size +25%)
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol$JSONListContext (took 0 ms, size +22%)
INFO: Instrumented org.apache.thrift.protocol.TStruct (took 0 ms, size +18%)
INFO: Instrumented org.apache.thrift.protocol.TJSONProtocol$LookaheadReader (took 1 ms, size +24%)
INFO: New number of coverage counters: 2048
Notably, you can see org.apache.thrift.test.fuzz.ParseJSONFuzzer was instrumented!
However, if I run python3 infra/helper.py check_build thrift-java it fails, with an error like the below, complaining about the exact same class
('/tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer', CompletedProcess(args=['bad_build_check', '/tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer'], returncode=1, stdout=b"BAD BUILD: /tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer seems to have either startup crash or exit:\nvm.mmap_rnd_bits = 28\n/tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null\nOpenJDK 64-Bit Server VM warning: Option CriticalJNINatives was deprecated in version 16.0 and will likely be removed in a future release.\nOpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended\nINFO: Loaded 265 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks\nINFO: Loaded 5 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks\nINFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks\nINFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks\nINFO: Loaded 2 hooks from com.code_intelligence.jazzer.sanitizers.ClojureLangHooks\nINFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization\nINFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection\nINFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection\nINFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup\nINFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection\nINFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall\nINFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection\nINFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks\nINFO: Loaded 12 hooks from com.code_intelligence.jazzer.sanitizers.ScriptEngineInjection\nINFO: Loaded 3 hooks from com.code_intelligence.jazzer.sanitizers.ServerSideRequestForgery\nINFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection\nINFO: Loaded 6 hooks from com.code_intelligence.jazzer.sanitizers.XPathInjection\nERROR: 'org.apache.thrift.test.fuzz.ParseJSONFuzzer' not found on classpath:\n\n/out/libthrift-0.23.0.jar:/out/libthrift-0.23.0-test.jar::/usr/local/lib/jazzer_api_deploy.jar:/tmp/not-out/tmpkrpgta8q/jazzer_agent_deploy.jar\n\nAll required classes must be on the classpath specified via --cp.\n", stderr=b''))
BAD BUILD: /tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer seems to have either startup crash or exit:
vm.mmap_rnd_bits = 28
/tmp/not-out/tmpkrpgta8q/ParseJSONFuzzer -rss_limit_mb=2560 -timeout=25 -seed=1337 -runs=4 < /dev/null
OpenJDK 64-Bit Server VM warning: Option CriticalJNINatives was deprecated in version 16.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
INFO: Loaded 265 hooks from com.code_intelligence.jazzer.runtime.TraceCmpHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.runtime.TraceDivHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.runtime.TraceIndirHooks
INFO: Loaded 4 hooks from com.code_intelligence.jazzer.runtime.NativeLibHooks
INFO: Loaded 2 hooks from com.code_intelligence.jazzer.sanitizers.ClojureLangHooks
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.Deserialization
INFO: Loaded 5 hooks from com.code_intelligence.jazzer.sanitizers.ExpressionLanguageInjection
INFO: Loaded 70 hooks from com.code_intelligence.jazzer.sanitizers.LdapInjection
INFO: Loaded 46 hooks from com.code_intelligence.jazzer.sanitizers.NamingContextLookup
INFO: Loaded 1 hooks from com.code_intelligence.jazzer.sanitizers.OsCommandInjection
INFO: Loaded 48 hooks from com.code_intelligence.jazzer.sanitizers.ReflectiveCall
INFO: Loaded 8 hooks from com.code_intelligence.jazzer.sanitizers.RegexInjection
INFO: Loaded 16 hooks from com.code_intelligence.jazzer.sanitizers.RegexRoadblocks
INFO: Loaded 12 hooks from com.code_intelligence.jazzer.sanitizers.ScriptEngineInjection
INFO: Loaded 3 hooks from com.code_intelligence.jazzer.sanitizers.ServerSideRequestForgery
INFO: Loaded 19 hooks from com.code_intelligence.jazzer.sanitizers.SqlInjection
INFO: Loaded 6 hooks from com.code_intelligence.jazzer.sanitizers.XPathInjection
ERROR: 'org.apache.thrift.test.fuzz.ParseJSONFuzzer' not found on classpath:
/out/libthrift-0.23.0.jar:/out/libthrift-0.23.0-test.jar::/usr/local/lib/jazzer_api_deploy.jar:/tmp/not-out/tmpkrpgta8q/jazzer_agent_deploy.jar
All required classes must be on the classpath specified via --cp.
ERROR: 100.0% of fuzz targets seem to be broken. See the list above for a detailed information.
5 tasks
|
The UBSan test failure is being fixed upstream, will look at the rest. |
mhlakhani
added a commit
to apache/thrift
that referenced
this pull request
Aug 27, 2025
Fix a Python test issue exposed by OSS-fuzz integration in google/oss-fuzz#13874 I can reproduce this with the oss-fuzz build (by setting UBSan flags locally). I do not know why the issue only shows up then, but this fix is correct - the method signatures are the same as the Compact protocol factory. The factory is supposed to not be passed any argument, or the arguments that are passed should be the limits. This manifested as the below failure: ``` 2025-08-26T18:20:18.9632418Z /usr/local/bin/python test/thrift_TSerializer.py 2025-08-26T18:20:19.0144330Z .E.. 2025-08-26T18:20:19.0145197Z ====================================================================== 2025-08-26T18:20:19.0146549Z ERROR: test_TBinaryProtocolAccelerated (__main__.TestSerializer.test_TBinaryProtocolAccelerated) 2025-08-26T18:20:19.0148344Z ---------------------------------------------------------------------- 2025-08-26T18:20:19.0149187Z Traceback (most recent call last): 2025-08-26T18:20:19.0158543Z File "/src/thrift/lib/py/test/thrift_TSerializer.py", line 68, in test_TBinaryProtocolAccelerated 2025-08-26T18:20:19.0159474Z self.verify(self.binary_serialized, factory) 2025-08-26T18:20:19.0160254Z File "/src/thrift/lib/py/test/thrift_TSerializer.py", line 50, in verify 2025-08-26T18:20:19.0160914Z deserialize(Message(), serialized, factory).body, 2025-08-26T18:20:19.0161739Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-08-26T18:20:19.0162428Z File "/src/thrift/lib/py/build/lib.linux-x86_64-cpython-311/thrift/TSerialization.py", line 37, in deserialize 2025-08-26T18:20:19.0163082Z base.read(protocol) 2025-08-26T18:20:19.0163513Z File "/src/thrift/lib/py/gen-py/TestServer/ttypes.py", line 45, in read 2025-08-26T18:20:19.0164259Z self.body = iprot.readString().decode('utf-8', errors='replace') if sys.version_info[0] == 2 else iprot.readString() 2025-08-26T18:20:19.0164897Z ^^^^^^^^^^^^^^^^^^ 2025-08-26T18:20:19.0165620Z File "/src/thrift/lib/py/build/lib.linux-x86_64-cpython-311/thrift/protocol/TProtocol.py", line 179, in readString 2025-08-26T18:20:19.0166301Z return self.readBinary().decode('utf-8') 2025-08-26T18:20:19.0166624Z ^^^^^^^^^^^^^^^^^ 2025-08-26T18:20:19.0167302Z File "/src/thrift/lib/py/build/lib.linux-x86_64-cpython-311/thrift/protocol/TBinaryProtocol.py", line 234, in readBinary 2025-08-26T18:20:19.0167990Z self._check_string_length(size) 2025-08-26T18:20:19.0168745Z File "/src/thrift/lib/py/build/lib.linux-x86_64-cpython-311/thrift/protocol/TBinaryProtocol.py", line 48, in _check_string_length 2025-08-26T18:20:19.0169531Z self._check_length(self.string_length_limit, length) 2025-08-26T18:20:19.0170268Z File "/src/thrift/lib/py/build/lib.linux-x86_64-cpython-311/thrift/protocol/TProtocol.py", line 57, in _check_length 2025-08-26T18:20:19.0170906Z if limit is not None and length > limit: 2025-08-26T18:20:19.0171360Z ^^^^^^^^^^^^^^ 2025-08-26T18:20:19.0171764Z TypeError: '>' not supported between instances of 'int' and 'TBufferedTransport' 2025-08-26T18:20:19.0172088Z 2025-08-26T18:20:19.0172231Z ---------------------------------------------------------------------- ```
mhlakhani
commented
Aug 28, 2025
| npm install --save-dev @jazzer.js/core | ||
|
|
||
| # Copy source code into the $OUT directory | ||
| if [ ! -d $OUT/thrift ]; then |
There was a problem hiding this comment.
I'm trying to reproduce the coverage build failure:
2025-08-27T00:22:17.7826712Z Run `npm audit` for details.
2025-08-27T00:22:17.8047024Z + '[' '!' -d /out/thrift ']'
2025-08-27T00:22:17.8047471Z + cp -r lib/thrift /out
2025-08-27T00:22:17.8074600Z + cp -r test/fuzz /out
2025-08-27T00:22:17.8092575Z + cp -r /src/node_modules /out
2025-08-27T00:22:17.8104074Z cp: cannot stat '/src/node_modules': No such file or directory
2025-08-27T00:22:18.5909350Z ERROR:__main__:Building fuzzers failed.
This doesn't reproduce locally with the following commands, as /out/thrift does not exist. In the CI environment, are the project builds in the same container (which may explain why there's some lingering state)?
python3 infra/helper.py build_image --cache thrift-js && python3 infra/helper.py build_fuzzers thrift-js --sanitizer coverage
python3 infra/helper.py coverage thrift-js --fuzz-target=fuzz_roundtrip_TBinaryProtocol --corpus-dir=/tmp/oss_fuzz_corpora_1
I also tried python3 infra/helper.py shell thrift-js --sanitizer coverage and then compile and it worked fine.
|
following the guidelines on the contributions page (sorry if this is a bit much) -- @jonathanmetzman could you help with some of the questions inline? I'm happy to make the fixes, I can't seem to figure out how to approach them (or in some cases, even repro locally). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR rewrites the thrift fuzzing support (which had bitrotted and was not compiling).
Fuzzers have been committed upstream as part of https://issues.apache.org/jira/browse/THRIFT-5855 and they now compile as part of regular CI to avoid bitrot. A number of bugs have been found and fixed as part of the development of these fuzzers already.
This is the "final" step to get them running on oss-fuzz (after which, I'll submit corpora, look at coverage reports, fix whatever bugs are found, add CI fuzz integration, etc).
I deleted the existing
thriftproject as it's now a misnomer (per a suggestion on Discord, someone suggested we should have one project per language). Let me know if that is a breaking change, in which case I can rename the 'thrift-go' in this PR to be 'thrift' again.This PR adds fuzzing for the following projects:
For each language (except c_glib), we have fuzzers that just parse input, and those that ensure that data round-trips properly, for each supported protocol in the language (binary/compact/json). Additionally, for Rust, we have a corpus generator checked-in to ensure corpora are up to date as things evolve. I will see how that can be extended in the future to support all languages.
I've verified that all the fuzzers build, run, and pass check_build (modulo the java one, which I'll comment inline for).