Land rapid7#3013, Metasm update

Author: wvu

lib/metasm.rb

@@ -1,7 +1,2 @@
 # Load a slightly tweaked METASM stub
 require 'metasm/metasm'
-
-# Manually load the classes we need from METASM
-require 'metasm/ia32'
-require 'metasm/mips'
-require 'metasm/exe_format/shellcode'

lib/metasm/.hg_archival.txt

@@ -21,6 +21,10 @@ Ready-to-use scripts can be found in the samples/ subdirectory, check the
 comments in the scripts headers. You can also try the --help argument if
 you're feeling lucky.
 
+For more information, check the doc/ subdirectory. The text files can be
+compiled to html using the misc/txt2html.rb script.
+
+
 
 Here is a short overview of the Metasm internals.
 
@@ -167,8 +171,8 @@ You can encode/decode an ExeFormat (ie decode sections, imports, headers etc)
 Constructor: ExeFormat.decode_file(str), ExeFormat.decode_file_header(str)
 Methods: ExeFormat#encode_file(filename), ExeFormat#encode_string
 
-PE and ELF files have a LoadedPE/LoadedELF counterpart, that is able to work
-with memory-mmaped versions of those formats (e.g. to debugging running
+PE and ELF files have a LoadedPE/LoadedELF counterpart, that are able to work
+with memory-mmaped versions of those formats (e.g. to debug running
 processes)
 
 
@@ -198,27 +202,31 @@ disassembly/patching easily (using LoadedPE/LoadedELF as ExeFormat)
 
 Debugging:
 
-Metasm includes a few interfaces to allow live debugging.
+Metasm includes a few interfaces to handle debugging.
 The WinOS and LinOS classes offer access to the underlying OS processes (e.g.
 OS.current.find_process('foobar') will retrieve a running process with foobar
 in its filename ; then process.mem can be used to access its memory.)
 
-The Windows and Linux debugging APIs (x86 only) have a basic ruby interface
-(PTrace32, extended in samples/rubstop.rb ; and WinDBG, a simple mapping of the
-windows debugging API) ; those will be more worked on/integrated in the future.
+The Windows and Linux low-level debugging APIs have a basic ruby interface
+(PTrace and WinAPI) ; which are used by the unified high-end Debugger class.
+Remote debugging is supported through the GDB server wire protocol.
 
-A linux console debugging interface is available in samples/lindebug.rb ; it
-uses a SoftICE-like look and feel.
-This interface can talk to a gdb-server through samples/gdbclient.rb ; use
-[udp:]<host:port> as target.
+High-level debuggers can be created with the following ruby line:
+Metasm::OS.current.create_debugger('foo')
 
-The disassembler scripts allow live process interaction by using as target
-'live:<pid or part of filename>'.
+Only one kind of host debugger class can exist at a time ; to debug multiple
+processes, attach to other processes using the existing class. This is due
+to the way the OS debugging API works on Windows and Linux.
 
-A generic debugging interface is available, it is defined in metasm/os/main.rb
-It may be accessed using the Metasm::OS.current.create_debugger('foo')
+The low-level backends are defined in the os/ subdirectory, the front-end is
+defined in debug.rb.
 
-It can be viewed in action using the GUI and 'open live' target.
+A linux console debugging interface is available in samples/lindebug.rb ; it
+uses a (simplified) SoftICE-like look and feel.
+It can talk to a gdb-server socket ; use a [udp:]<host:port> target.
+
+The disassembler-gui sample allow live process interaction when using as
+target 'live:<pid or part of program name>'.
 
 
 C Parser:
@@ -236,7 +244,11 @@ It handles all the constructs i am aware of, except hex floats:
  - __int8 etc native types
  - Label addresses (&&label)
 Also note that all those things are parsed, but most of them will fail to
-compile on the Ia32 backend (the only one implemented so far.)
+compile on the Ia32/X64 backend (the only one implemented so far.)
+
+Parsing C files should be done using an existing ExeFormat, with the
+parse_c_file method. This ensures that format-specific macros/ABI are correctly
+defined (ex: size of the 'long' type, ABI to pass parameters to functions, etc)
 
 When you parse a C String using C::Parser.parse(text), you receive a Parser
 object. It holds a #toplevel field, which is a C::Block, which holds #structs,
@@ -249,15 +261,11 @@ CExpressions...)
 
 A C::Parser may be #precompiled to transform it into a simplified version that
 is easier to compile: typedefs are removed, control sequences are transformed
-in if () goto ; etc.
+into 'if (XX) goto YY;' etc.
 
 To compile a C program, use PE/ELF.compile_c, that will create a C::Parser with
 exe-specific macros defined (eg __PE__ or __ELF__).
 
-The prefered way to create a C::Parser is to initialize it with a CPU and the
-desired ExeFormat, so that it is
-correctly initialized (eg type sizes: is long 4 or 8 bytes? etc) ; and
-may define preprocessor macros needed to correctly parse standard headers.
 Vendor-specific headers may need to use either #pragma prepare_visualstudio
 (to parse the Microsoft Visual Studio headers) or prepare_gcc (for gcc), the
 latter may be auto-detected (or may not).

lib/metasm/README

@@ -2,13 +2,14 @@ List of TODO items, by section, in random order
 
 Ia32
  emu fpu
- add all sse2 instrs
+ AVX support
  realmode
 
 X86_64
  decompiler
 
 CPU
+ Arm
  Sparc
  Cell
 
@@ -26,14 +27,14 @@ Assembler
 Disasm
  DecodedData
  Exe decoding generate decodeddata ?
- Function-local namespace (esp+12 -> esp+var_42)
+ Function variable names using stack analysis + ExpressionString
  Fix thunk detection (thunk: mov ecx, 42  jmp [iat_thiscall] is not a thunk)
  Test with ET_REL style exe
  Store stuff out of mem (to handle big binaries)
  Better :default usage
   good on call eax, but not on <600k instrs> ret
   use binary personality ? (uses call vs uses pushret..)
- Improve backtrace -> patch di.instr.args exprs
+ Improve 'backtrace => patch di.instr.args'
  path-specific backtracking ( foo: call a ; a: jmp retloc ; bar: call b ; b: jmp retloc ; retloc: ret ; call foo ; ret : last ret trackback should only reach a:)
  Decode pseudo/macro-instrs (mips 'li')
  Deoptimizer (instr reordering for readability)
@@ -69,6 +70,7 @@ Decompiler
  Handle/hide compiler-generated stuff (getip, stack cookie setup/check..)
  Handle call 1f ; 1: pop eax
  More user control (force/forbid register arg, return type, etc)
+ Preserve C decompiled line association to range of asm decoded addrs
 
 Debugger
  OSX
@@ -81,7 +83,6 @@ Debugger
  Remote debugging (small standalone C client)
  Support dbghelp.dll (ms symbol server info)
  Support debugee function call (gdb 'call')
- Manipulate memory through C struct casts
 
 ExeFormat
  Handle minor editing without decode/reencode (eg patch ELF entrypoint)
@@ -105,10 +106,9 @@ GUI
    show breakpoints
    show jump direction from current flag values
  have a console frontend
- better graph positionning fallback
  zoom font when zooming graph
- copy/paste, selection
+ text selection
  map (part of) the binary & debug it (map a PE on a linux host & run it)
 
 Ruby
- compile ruby AST to native optimized code
+ write a fast ruby-like interpreter