Fix Unicorn Engine 1GB limit that calls exit: raise OSError instead (Fixes Gallopsled#2343) (Gallopsled#2347)

* Fix Unicorn Engine 1GB limit exit(): raise OSError instead (Fixes Gallopsled#2343)

This commit fixes Gallopsled#2343: an issue where `pwn checksec <binary>` would fail with a bogus error of:

```
$ pwn checksec /root/x/bin/main
Could not allocate dynamic translator buffer
```

This actually comes from Unicorn Engine which tries to allocate a 1GB RWX mapping:
```
root@pwndbg:~# strace -e openat,mmap pwn checksec /root/x/bin/main 2>&1 | tail -n 10
openat(AT_FDCWD, "/usr/lib/python3/dist-packages/mpmath-0.0.0.egg-info/PKG-INFO", O_RDONLY|O_CLOEXEC) = 7
openat(AT_FDCWD, "/usr/local/lib/python3.10/dist-packages/unicorn/lib/libunicorn.so.2", O_RDONLY|O_CLOEXEC) = 7
mmap(NULL, 22447520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 7, 0) = 0x7f2604f9d000
mmap(0x7f2605339000, 13496320, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x39c000) = 0x7f2605339000
mmap(0x7f2606018000, 3039232, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x107b000) = 0x7f2606018000
mmap(0x7f26062fe000, 1601536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 7, 0x1360000) = 0x7f26062fe000
mmap(0x7f2606485000, 525728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f2606485000
mmap(NULL, 1073741824, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
Could not allocate dynamic translator buffer
+++ exited with 1 +++
```

...and if it fails, it calls `exit()`. This can be seen in Unicorn Engine code: https://github.com/unicorn-engine/unicorn/blob/56f3bdedb42d26bee1532cc01baf5eaf44a9aa23/qemu/accel/tcg/translate-all.c#L960-L963

This issue has been reported to Unicorn Engine in unicorn-engine/unicorn#1766 but since it hasn't been fixed, this commit applies a workaround for it.

* CI: add test for pwn checksec with 500MB limit for UE 1GB limit

* Add changelog entry

* Update .github/workflows/ci.yml

---------

Co-authored-by: peace-maker <[email protected]>

Author: disconnect3d

.github/workflows/ci.yml

@@ -164,6 +164,7 @@ jobs:
         pwn phd -l 0x3d --color=always /etc/os-release
 
         pwn checksec /bin/bash
+        (ulimit -v 500000 && pwn checksec /bin/bash)
 
         pwn errno 2
         pwn errno -1

CHANGELOG.md

@@ -96,6 +96,7 @@ The table below shows which release corresponds to each branch, and what date th
 - [#2338][2338] Fix: follow symlink for libs on ssh connection
 - [#2341][2341] Launch GDB correctly in iTerm on Mac
 - [#2268][2268] Add a `flatten` argument to `ssh.libs`
+- [#2347][2347] Fix/workaround Unicorn Engine 1GB limit that calls exit()
 
 [2242]: https://github.com/Gallopsled/pwntools/pull/2242
 [2277]: https://github.com/Gallopsled/pwntools/pull/2277
@@ -120,6 +121,7 @@ The table below shows which release corresponds to each branch, and what date th
 [2338]: https://github.com/Gallopsled/pwntools/pull/2338
 [2341]: https://github.com/Gallopsled/pwntools/pull/2341
 [2268]: https://github.com/Gallopsled/pwntools/pull/2268
+[2347]: https://github.com/Gallopsled/pwntools/pull/2347
 
 ## 4.12.0 (`beta`)
 

pwnlib/elf/plt.py

@@ -53,9 +53,30 @@ def emulate_plt_instructions(elf, got, address, data, targets):
 
     return rv
 
+
+def __ensure_memory_to_run_unicorn():
+    """
+    Check if there is enough memory to run Unicorn Engine.
+    Unicorn Engine requires 1GB of memory to run, if there isn't enough memory it calls exit(1).
+
+    This is a bug in Unicorn Engine, see: https://github.com/unicorn-engine/unicorn/issues/1766
+    """
+    try:
+        from mmap import mmap, MAP_ANON, MAP_PRIVATE, PROT_EXEC, PROT_READ, PROT_WRITE
+
+        mm = mmap(
+            -1, 1024 * 1024 * 1024, MAP_PRIVATE | MAP_ANON, PROT_WRITE | PROT_READ | PROT_EXEC
+        )
+        mm.close()
+    except OSError:
+        raise OSError("Cannot allocate 1GB memory to run Unicorn Engine")
+
+
 def prepare_unicorn_and_context(elf, got, address, data):
     import unicorn as U
 
+    __ensure_memory_to_run_unicorn()
+
     # Instantiate the emulator with the correct arguments for the current
     # architecutre.
     arch = {