Skip to content

Commit a77e84e

Browse files
jasocrow0vercl0k
jasocrow
and
0vercl0k
authored
Add support for Linux userland ELF snapshots and fuzzing (#192)
Co-authored-by: 0vercl0k <1476421+0vercl0k@users.noreply.github.com>
1 parent 6049f79 commit a77e84e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+2016
-106
lines changed

‎.github/workflows/wtf.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,6 @@ name: Builds
22

33
on: [push, pull_request]
44

5-
permissions:
6-
actions: read
7-
contents: read
8-
security-events: write
9-
105
jobs:
116
Windows:
127
runs-on: windows-2019
@@ -71,11 +66,15 @@ jobs:
7166
- name: Checkout
7267
uses: actions/checkout@v4
7368

69+
# Revert to the below when https://github.com/llvm/llvm-project/issues/84271 is fixed:
70+
# sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
7471
- name: Installing dependencies
7572
run: |
7673
sudo apt-get -y update
7774
sudo apt install -y g++-10 ninja-build
78-
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
75+
curl -O https://apt.llvm.org/llvm.sh
76+
chmod u+x ./llvm.sh
77+
sudo ./llvm.sh 17
7978
8079
- name: Build with gcc
8180
if: matrix.compiler == 'gcc'
@@ -87,11 +86,13 @@ jobs:
8786
chmod u+x ./build-release.sh
8887
./build-release.sh
8988
89+
# Revert to the below when https://github.com/llvm/llvm-project/issues/84271 is fixed:
90+
# clang-18 / clang++-18
9091
- name: Build with clang
9192
if: matrix.compiler == 'clang'
9293
env:
93-
CC: clang-18
94-
CXX: clang++-18
94+
CC: clang-17
95+
CXX: clang++-17
9596
run: |
9697
cd src/build
9798
chmod u+x ./build-release.sh

‎README.md‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<div align='center'>
22
<h1><code>what the fuzz</code></h1>
33
<p>
4-
<strong>A distributed, code-coverage guided, cross-platform snapshot-based fuzzer designed for attacking user and or kernel-mode targets running on Microsoft Windows.</strong>
4+
<strong>A distributed, code-coverage guided, cross-platform snapshot-based fuzzer designed for attacking user and or kernel-mode targets running on Microsoft Windows and Linux user-mode (experimental!).</strong>
55
</p>
66
<p>
77
<img src='https://github.com/0vercl0k/wtf/workflows/Builds/badge.svg'/>
@@ -13,7 +13,7 @@
1313

1414
## Overview
1515

16-
**what the fuzz** or **wtf** is a distributed, code-coverage guided, customizable, cross-platform snapshot-based fuzzer designed for attacking user and or kernel-mode targets running on Microsoft Windows. Execution of the target can be done inside an emulator with [bochscpu](https://github.com/yrp604/bochscpu) (slowest, most precise), inside a Windows VM with the [Windows Hypervisor Platform APIs](https://docs.microsoft.com/en-us/virtualization/api/hypervisor-platform/hypervisor-platform) or inside a Linux VM with the [KVM APIs](https://www.kernel.org/doc/html/latest/virt/kvm/api.html) (fastest).
16+
**what the fuzz** or **wtf** is a distributed, code-coverage guided, customizable, cross-platform snapshot-based fuzzer designed for attacking user and or kernel-mode targets running on Microsoft Windows or Linux (**experimental**, see [linux_mode](linux_mode/)). Execution of the target can be done inside an emulator with [bochscpu](https://github.com/yrp604/bochscpu) (slowest, most precise), inside a Windows VM with the [Windows Hypervisor Platform APIs](https://docs.microsoft.com/en-us/virtualization/api/hypervisor-platform/hypervisor-platform) or inside a Linux VM with the [KVM APIs](https://www.kernel.org/doc/html/latest/virt/kvm/api.html) (fastest).
1717

1818
It uncovered memory corruption vulnerabilities in a wide range of softwares: [IDA Pro](https://github.com/0vercl0k/fuzzing-ida75), a popular [AAA game](https://blog.ret2.io/2021/07/21/wtf-snapshot-fuzzing/), the [Windows kernel](https://microsoft.fandom.com/wiki/Architecture_of_Windows_NT), the [Microsoft RDP client](https://www.hexacon.fr/slides/Hexacon2022-Fuzzing_RDPEGFX_with_wtf.pdf), [NVIDIA GPU Display driver](https://nvidia.custhelp.com/app/answers/detail/a_id/5383), etc.
1919

@@ -25,8 +25,6 @@ If you would like to read more about its history or how to use it on a real targ
2525
- [Fuzzing RDPEGFX with "what the fuzz"](https://thalium.github.io/blog/posts/rdpegfx/) by [Colas Le Guernic](https://github.com/clslgrnc), Jérémy Rubert, and Anonymous
2626
- [A Journey to Network Protocol Fuzzing – Dissecting Microsoft IMAP Client Protocol](https://www.fortinet.com/blog/threat-research/analyzing-microsoft-imap-client-protocol) by [Wayne Chin Yick Low](https://www.fortinet.com/blog/search?author=Wayne+Chin+Yick+Low)
2727

28-
Special thanks to [@yrp604](https://github.com/yrp604) for providing valuable inputs throughout the project and [@masthoon](https://github.com/masthoon) for suggesting to write a demo targeting [HEVD](https://github.com/hacksysteam/HackSysExtremeVulnerableDriver) secure mode.
29-
3028
## Usage
3129

3230
The best way to try the features out is to work with the [fuzzer_hevd](src/wtf/fuzzer_hevd.cc) / [fuzzer_tlv_server](src/wtf/fuzzer_tlv_server.cc) modules. You can grab the [target-hevd.7z](https://github.com/0vercl0k/wtf/releases) / [target-tlv_server.7z](https://github.com/0vercl0k/wtf/releases) archives and extract them into the `targets/` directory. The archives contain the directory trees that are expected for every targets:
@@ -316,3 +314,17 @@ To build it yourself you need to start a *Visual Studio Developper Command Promp
316314
## Authors
317315

318316
* Axel '[0vercl0k](https://twitter.com/0vercl0k)' Souchet
317+
318+
## Contributors
319+
320+
Special thanks to:
321+
- [@yrp604](https://github.com/yrp604) for providing valuable inputs throughout the project,
322+
- [@masthoon](https://github.com/masthoon) for suggesting to write a demo targeting [HEVD](https://github.com/hacksysteam/HackSysExtremeVulnerableDriver) secure mode,
323+
- [Markus Gaasedelen](https://github.com/0vercl0k/wtf/pull/12/) for adding Tenet support,
324+
- [@y0ny0ns0n](https://github.com/y0ny0ns0n) for contributing the [multi-input fuzzing example](https://github.com/0vercl0k/wtf/pull/67),
325+
- [Colas Le Guernic](https://github.com/clslgrnc) / Jérémy Rubert / Anonymous for implementing [edge coverage for bochscpu](https://github.com/0vercl0k/wtf/pull/137),
326+
- [@1ndahous3](https://github.com/1ndahous3) for contributing the [generic ioctl fuzzer module](https://github.com/0vercl0k/wtf/pull/155),
327+
- Jason Crowder / [Kyle Ossinger](https://k0ss.net/) from Cisco ASIG for [the Linux mode](https://github.com/0vercl0k/wtf/pull/192),
328+
- and all the other contributors 🙏
329+
330+
[ ![contributors-img](https://contrib.rocks/image?repo=0vercl0k/wtf) ](https://github.com/0vercl0k/wtf/graphs/contributors)

‎linux_mode/.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
gdb.txt
2+
3+
__pycache__/
4+
5+
# Snapshot files
6+
mem.dmp
7+
raw
8+
symbol-store.json
9+
vm.log
10+
vm.pid
11+
regs.json

‎linux_mode/README.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
<div align='center'>
2+
<h1><code>what the fuzz: Linux mode</code></h1>
3+
<p>
4+
<strong>Experimental user-mode Linux mode by Jason Crowder and <a href="https://k0ss.net/">Kyle Ossinger</a> from Cisco ASIG</strong>
5+
</p>
6+
<p>
7+
<img src='https://github.com/0vercl0k/wtf/workflows/Builds/badge.svg'/>
8+
</p>
9+
<p>
10+
<img src='../pics/wtf-linux.gif'/>
11+
</p>
12+
</div>
13+
14+
## Overview
15+
16+
This provides experimental Linux ELF userland snapshotting support based on previous work by [Kasimir](https://github.com/0vercl0k/wtf/pull/102) and scripts from [Snapchange](https://github.com/awslabs/snapchange/tree/main/qemu_snapshot).
17+
18+
<p align='center'>
19+
<img src='../pics/wtf-linux-snapshot.webp'>
20+
</p>
21+
22+
## Setting up the environment
23+
24+
Move into the `linux_mode/qemu_snapshot` directory and run `setup.sh`:
25+
26+
```console
27+
user@pc:/wtf/linux_mode/qemu_snapshot$ ./setup.sh
28+
```
29+
30+
This script installs all pre-requisite tools, compiles qemu, and builds a target virtual machine consisting of a Linux kernel and disk image.
31+
32+
## Taking a snapshot
33+
34+
Create a subdirectory in `linux_mode` for your snapshot and create a `bkpt.py` file, like [linux_mode/crash_test/bkpt.py](crash_test/bkpt.py):
35+
36+
```py
37+
# imports
38+
import sys, os
39+
40+
# import fuzzing breakpoint
41+
from gdb_fuzzbkpt import *
42+
43+
target_dir = 'linux_crash_test'
44+
45+
# address to break on, found using gdb
46+
break_address = 'do_crash_test'
47+
48+
# name of the file in which to break
49+
file_name = 'a.out'
50+
51+
# create the breakpoint for the executable specified
52+
FuzzBkpt(target_dir, break_address, file_name, sym_path=file_name)
53+
```
54+
55+
* `target_dir` - subdirectory in `targets` to save the snapshot data
56+
* `break_address` - address to break on. This can be a hardcoded address or a symbol if `sym_path` is provided
57+
* `file_name` - name of the file in the target VM associated with the breakpoint
58+
* `sym_path` - optional argument if you'd like symbols to be loaded
59+
60+
Start the virtual machine in one tab while in the snapshot subdirectory by running `../qemu_snapshot/gdb_server.sh`:
61+
62+
```console
63+
user@pc:/wtf/linux_mode/crash_test$ ../qemu_snapshot/gdb_server.sh
64+
```
65+
66+
In a separate tab, scp the target file to the target VM. With `crash_test` this can be done by first compiling the target file:
67+
68+
```console
69+
user@pc:/wtf/linux_mode/crash_test$ gcc test.c
70+
```
71+
72+
Then transfer the target file to the VM:
73+
74+
```console
75+
user@pc:/wtf/linux_mode/crash_test$ pushd ../qemu_snapshot/target_vm
76+
user@pc:/wtf/linux_mode/qemu_snapshot/target_vm$ ./scp.sh ../../crash_test/a.out
77+
a.out 100% 16KB 1.2MB/s 00:00
78+
```
79+
80+
Go back to the `crash_test` directory.
81+
82+
```console
83+
user@pc:/wtf/linux_mode/qemu_snapshot/target_vm$ popd
84+
/wtf/linux_mode/crash_test
85+
user@pc:/wtf/linux_mode/crash_test$
86+
```
87+
88+
Now, run `../qemu_snapshot/gdb_client.sh`:
89+
90+
```console
91+
user@pc:/wtf/linux_mode/crash_test$ ../qemu_snapshot/gdb_client.sh
92+
```
93+
94+
In the first tab, log in to the Linux machine (user `root`) and run the target file:
95+
96+
```console
97+
linux login: root
98+
Linux linux 6.7.0-rc3 #1 SMP PREEMPT_DYNAMIC Thu Nov 30 18:38:29 UTC 2023 x86_64
99+
100+
The programs included with the Debian GNU/Linux system are free software;
101+
the exact distribution terms for each program are described in the
102+
individual files in /usr/share/doc/*/copyright.
103+
104+
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
105+
permitted by applicable law.
106+
A valid context for root could not be obtained.
107+
Last login: Fri Dec 1 21:21:22 UTC 2023 on ttyS0
108+
root@linux:~# ./a.out
109+
110+
Enter some input.
111+
d
112+
```
113+
114+
Once the breakpoint is hit, the second tab will start the snapshotting process:
115+
116+
```console
117+
Continuing.
118+
In right process? True
119+
Calling mlockall
120+
Saving 67 bytes at 555555555146
121+
In right process? True
122+
Restoring 67 bytes at 0x555555555146
123+
Restored
124+
In the Qemu tab, press Ctrl+C, run the `cpu` command
125+
```
126+
127+
Once the second tab indicates to run the `cpu` command, press Ctrl+C and run the `cpu` command from the first tab:
128+
129+
```console
130+
Thread 1 "qemu-system-x86" received signal SIGINT, Interrupt.
131+
0x00007ffff77a4ebe in __ppoll (fds=0x5555568337d0, nfds=8, timeout=<optimized out>, timeout@entry=0x7fffffffdea0, sigmask=si
132+
42 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
133+
(gdb) cpu
134+
cpu_state: 0x55555681e240
135+
done...continuing debuggee
136+
```
137+
138+
The second tab will detect once the first tab has finished executing the `cpu` command and continue creating a snapshot for the target VM.
139+
140+
Once the second tab indicates that snapshotting is complete, the target VM can be terminated.
141+
142+
```console
143+
In the Qemu tab, press Ctrl+C, run the `cpu` command
144+
Detected cpu registers dumped to regs.json
145+
Connecting to Qemu monitor at localhost:55555
146+
Connected
147+
Instructing Qemu to dump physical memory to file raw
148+
Done
149+
Converting raw file raw to dump file /wtf/targets/linux_crash_test/state/mem.dmp
150+
Done
151+
mv regs.json /wtf/targets/linux_crash_test/state/regs.json
152+
mv symbol-store.json /wtf/targets/linux_crash_test/state/symbol-store.json
153+
Snapshotting complete
154+
155+
Breakpoint 1, 0x0000555555555189 in do_crash_test ()
156+
(gdb)
157+
```
158+
159+
## Harnessing and Fuzzing
160+
161+
Writing harnesses is the same process as writing harnesses for Windows executables. Example harnesses for crash_test and page_fault_test are present in [src/wtf/fuzzer_linux_crash_test.cc](../src/wtf/fuzzer_linux_crash_test.cc) and [src/wtf/fuzzer_linux_page_fault_test.cc](../src/wtf/fuzzer_linux_page_fault_test.cc).
162+
163+
Now that we have everything set up we can start our server and fuzzer:
164+
165+
Provide a seed input:
166+
167+
```console
168+
user@pc:/wtf/targets/linux_crash_test$ echo a>inputs/a
169+
```
170+
171+
Run the master:
172+
173+
```console
174+
user@pc:/wtf/targets/linux_crash_test$ ../../src/build/wtf master --name linux_crash_test --max_len=10
175+
```
176+
177+
Run the fuzzee and note that crashes are found quickly.
178+
179+
```console
180+
user@pc:/wtf/targets/linux_crash_test$ ../../src/build/wtf fuzz --backend=bochscpu --name linux_crash_test
181+
Setting @fptw to 0xff'ff.
182+
The debugger instance is loaded with 16 items
183+
Setting debug register status to zero.
184+
Setting debug register status to zero.
185+
Setting mxcsr_mask to 0xffbf.
186+
Dialing to tcp://localhost:31337/..
187+
#113174 cov: 47 exec/s: 11.3k lastcov: 2.0s crash: 1782 timeout: 0 cr3: 0 uptime: 10.0s
188+
```
189+
190+
To fuzz with KVM, create a coverage breakpoints file by loading the target file in IDA and running [scripts/gen_linux_coveragefile_ida.py](../scripts/gen_linux_coveragefile_ida.py). Transfer the coverage breakpoints file to the `coverage` subfolder in the target's directory. For example, for `linux_crash_test` transfer the coverage breakpoint file to `targets/linux_crash_test/coverage/a.cov`. Once transferred, KVM can be used for fuzzing:
191+
192+
```console
193+
user@pc:/wtf/targets/linux_crash_test$ sudo ../../src/build/wtf fuzz --backend=kvm --name linux_crash_test
194+
Setting @fptw to 0xff'ff.
195+
The debugger instance is loaded with 16 items
196+
Parsing coverage/a.cov..
197+
Applied 44 code coverage breakpoints
198+
Setting debug register status to zero.
199+
Setting debug register status to zero.
200+
Setting mxcsr_mask to 0xffbf.
201+
Resolved breakpoint 0xffffffff82001240 at GPA 0x2001240 aka HVA 0x564428d2afe0
202+
Resolved breakpoint 0xffffffff82000ff0 at GPA 0x2000ff0 aka HVA 0x564428d2cda0
203+
Resolved breakpoint 0xffffffff81099dc0 at GPA 0x1099dc0 aka HVA 0x564428d2db80
204+
Resolved breakpoint 0xffffffff810708e0 at GPA 0x10708e0 aka HVA 0x564428d2e6b0
205+
Resolved breakpoint 0x5555555551e7 at GPA 0x972c1e7 aka HVA 0x564428d32117
206+
Dialing to tcp://localhost:31337/..
207+
#24348 cov: 8 exec/s: 2.4k lastcov: 3.0s crash: 871 timeout: 0 cr3: 0 uptime: 10.0s
208+
```
209+
210+
## Symbolizing
211+
212+
The only current way to symbolize and debug your testcases is to use the bochscpu backend and generate a Tenet traces as per [Generating Tenet traces](https://github.com/0vercl0k/wtf?tab=readme-ov-file#generating-tenet-traces).

‎linux_mode/crash_test/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Test program and breakpoint script to verify Linux snapshotting and fuzzing work
2+
correctly.

‎linux_mode/crash_test/bkpt.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Jason Crowder - February 2024
2+
# imports
3+
import sys, os
4+
5+
# import fuzzing breakpoint
6+
from gdb_fuzzbkpt import *
7+
8+
target_dir = "linux_crash_test"
9+
10+
# address to break on, found using gdb
11+
break_address = "do_crash_test"
12+
13+
# name of the file in which to break
14+
file_name = "a.out"
15+
16+
# create the breakpoint for the executable specified
17+
FuzzBkpt(target_dir, break_address, file_name, sym_path=file_name)

‎linux_mode/crash_test/test.c‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
// Jason Crowder - February 2024
2+
#include <stdio.h>
3+
#include <stdlib.h>
4+
5+
void do_crash_test(char* input) {
6+
if (input[0] == 'C' && input[1] == 'R' && input[2] == 'A' &&
7+
input[3] == 'S' && input[4] == 'H') {
8+
*(char*)NULL = '\0';
9+
}
10+
}
11+
12+
void end_crash_test() { printf("End crash test.\n"); }
13+
14+
int main(int argc, char* argv[]) {
15+
char* buf = NULL;
16+
size_t cbBuf = 10;
17+
ssize_t cbRead = 0;
18+
19+
buf = (char*)calloc(1, cbBuf);
20+
if (!buf) {
21+
printf("calloc failed.\n");
22+
goto END;
23+
}
24+
25+
printf("Enter some input.\n");
26+
cbRead = getline(&buf, &cbBuf, stdin);
27+
if (-1 == cbRead) {
28+
perror("getline failure: ");
29+
goto END;
30+
}
31+
32+
do_crash_test(buf);
33+
34+
end_crash_test();
35+
36+
END:
37+
if (buf) {
38+
free(buf);
39+
}
40+
}

‎linux_mode/page_fault_test/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Test page fault program for making sure memory locking instructions and code
2+
work correctly.

‎linux_mode/page_fault_test/bkpt.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Jason Crowder - February 2024
2+
# imports
3+
import sys, os
4+
5+
# import fuzzing breakpoint
6+
from gdb_fuzzbkpt import *
7+
8+
target_dir = "linux_page_fault_test"
9+
10+
break_address = "page_fault_test"
11+
12+
# name of the file in which to break
13+
file_name = "a.out"
14+
15+
# create the breakpoint for the executable specified
16+
FuzzBkpt(target_dir, break_address, file_name, bp_hits_required=1, sym_path=file_name)

0 commit comments

Comments
 (0)