Hook bypassing #41
Description
Problem
There are two kinds of global hooks in LuaJIT.
- Debug hooks (
debug.sethook) - VM events (
jit.attach)
Debug hooks deal with interpreter level actions, like executing a line, making calls, returning, etc. VM hooks deal with VM-level actions (duh) like generating bytecode, starting traces, exiting traces, recording functions, etc.
Both risk Autorun for detection, but I have good news for an exhaustive solution for both, although they're not pretty.
Solution for debug hooks
Debug hooks are called as a function executes, meaning no matter what, level 2 of the stack ALWAYS points to the function running. debug.gethook is global, so we can easily spoof the return to the real one. Then, we can detour debug.sethook to set a wrapper which checks the function if it's an Autorun function, and drops it if so.
Solution for VMEVENTS
We can basically spoof FindMetaTable and build a fake _VMEVENTS table with each jit.attach call adding/removing entries according to the VM hash for their respective events, and return that. Meanwhile, we would detour the actual callbacks with a Lua-level check if the received VM event for a function is an Autorun function, and if so, drop it. Otherwise, we can call the callback.
There are native-level bypasses for each that I've used, which work, but they're quite limited and I figure Autorun.detour is a good tool for this job.