Automerge: [lldb] Add utility to create Mach-O corefile from YAML desc (#153911)

I've wanted a utility to create a corefile for test purposes given a bit
of memory and regsters, for a while. I've written a few API tests over
the years that needed exactly this capability -- we have several one-off
Mach-O corefile creator utility in the API testsuite to do this. But
it's a lot of boilerplate when you only want to specify some register
contents and memory contents, to create an API test.

This adds yaml2mach-core, a tool that should build on any system, takes
a yaml description of register values for one or more threads,
optionally memory values for one or more memory regions, and can take a
list of UUIDs that will be added as LC_NOTE "load binary" metadata to
the corefile so binaries can be loaded into virtual address space in a
test scenario.

The format of the yaml file looks like

```
cpu: armv7m

# optionally specify the number of bits used for addressing
# (this line is from a different, 64-bit, yaml file)
addressable-bits:
    num-bits: 39

# optionally specify one or more binary UUID and slide/virtual address to be added as an LC_NOTE
# (this line is from a different, 64-bit, yaml file)
binaries:
  - name: debug-binary.development
    uuid: 67942352-5857-3D3D-90CB-A3F80BA67B04
    virtual-address: 0xfffffff01840c000

threads:
  - regsets:
     - flavor: gpr
        registers: [{name: sp, value: 0x2000fe70}, {name: r7, value: 0x2000fe80},
                    {name: pc, value: 0x0020392c}, {name: lr, value: 0x0020392d}]

memory-regions:
  # stack memory
  - addr: 0x2000fe70
    UInt32: [ 0x0000002a, 0x20010e58, 0x00203923,
              0x00000001, 0x2000fe88, 0x00203911,
              0x2000ffdc, 0xfffffff9 ]
  # instructions of a function
  - addr: 0x203910
     UInt8: [ 0xf8, 0xb5, 0x04, 0xaf, 0x06, 0x4c, 0x07, 0x49,
              0x74, 0xf0, 0x2e, 0xf8, 0x01, 0xac, 0x74, 0xf0 ]
```

and that's all that is needed to specify a corefile where four register
values are specified (the others will be set to 0), and two memory
regions will be emitted.

The memory can be specified as an array of UInt8, UInt32, or UInt64, I
anticipate that some of these corefiles may have stack values
constructed manually and it may be simpler for a human to write them in
a particular grouping of values.

I needed this utility for an upcoming patch for ARM Cortex-M processors,
to create a test for the change. I took the opportunity to remove two of
the "trivial mach-o corefile" creator utilities I've written in the
past, which also restricted the tests to only run on Darwin systems
because I was using the system headers for Mach-O constant values.

rdar://110663219

Author: jasonmolenda

lldb/packages/Python/lldbsuite/test/configuration.py

@@ -64,6 +64,9 @@
 # Path to the yaml2obj tool. Not optional.
 yaml2obj = None
 
+# Path to the yaml2macho-core tool. Not optional.
+yaml2macho_core = None
+
 # The arch might dictate some specific CFLAGS to be passed to the toolchain to build
 # the inferior programs.  The global variable cflags_extras provides a hook to do
 # just that.
@@ -174,3 +177,11 @@ def get_yaml2obj_path():
     """
     if yaml2obj and os.path.lexists(yaml2obj):
         return yaml2obj
+
+
+def get_yaml2macho_core_path():
+    """
+    Get the path to the yaml2macho-core tool.
+    """
+    if yaml2macho_core and os.path.lexists(yaml2macho_core):
+        return yaml2macho_core

lldb/packages/Python/lldbsuite/test/dotest.py

@@ -280,6 +280,9 @@ def parseOptionsAndInitTestdirs():
         configuration.llvm_tools_dir = args.llvm_tools_dir
         configuration.filecheck = shutil.which("FileCheck", path=args.llvm_tools_dir)
         configuration.yaml2obj = shutil.which("yaml2obj", path=args.llvm_tools_dir)
+        configuration.yaml2macho_core = shutil.which(
+            "yaml2macho-core", path=args.llvm_tools_dir
+        )
 
     if not configuration.get_filecheck_path():
         logging.warning("No valid FileCheck executable; some tests may fail...")

lldb/packages/Python/lldbsuite/test/lldbtest.py

@@ -1702,6 +1702,29 @@ def yaml2obj(self, yaml_path, obj_path, max_size=None):
             command += ["--max-size=%d" % max_size]
         self.runBuildCommand(command)
 
+    def yaml2macho_core(self, yaml_path, obj_path, uuids=None):
+        """
+        Create a Mach-O corefile at the given path from a yaml file.
+
+        Throws subprocess.CalledProcessError if the object could not be created.
+        """
+        yaml2macho_core_bin = configuration.get_yaml2macho_core_path()
+        if not yaml2macho_core_bin:
+            self.assertTrue(False, "No valid yaml2macho-core executable specified")
+        if uuids != None:
+            command = [
+                yaml2macho_core_bin,
+                "-i",
+                yaml_path,
+                "-o",
+                obj_path,
+                "-u",
+                uuids,
+            ]
+        else:
+            command = [yaml2macho_core_bin, "-i", yaml_path, "-o", obj_path]
+        self.runBuildCommand(command)
+
     def cleanup(self, dictionary=None):
         """Platform specific way to do cleanup after build."""
         module = builder_module()
@@ -2264,7 +2287,9 @@ def completions_match(self, command, completions, max_completions=-1):
         given list of completions"""
         interp = self.dbg.GetCommandInterpreter()
         match_strings = lldb.SBStringList()
-        interp.HandleCompletion(command, len(command), 0, max_completions, match_strings)
+        interp.HandleCompletion(
+            command, len(command), 0, max_completions, match_strings
+        )
         # match_strings is a 1-indexed list, so we have to slice...
         self.assertCountEqual(
             completions, list(match_strings)[1:], "List of returned completion is wrong"

lldb/test/API/macosx/arm-corefile-regctx/Makefile

@@ -13,20 +13,14 @@
 class TestArmMachoCorefileRegctx(TestBase):
     NO_DEBUG_INFO_TESTCASE = True
 
-    @skipUnlessDarwin
-    def setUp(self):
-        TestBase.setUp(self)
-        self.build()
-        self.create_corefile = self.getBuildArtifact("a.out")
-        self.corefile = self.getBuildArtifact("core")
-
     def test_armv7_corefile(self):
         ### Create corefile
-        retcode = call(self.create_corefile + " armv7 " + self.corefile, shell=True)
+        corefile = self.getBuildArtifact("core")
+        self.yaml2macho_core("armv7m.yaml", corefile)
 
         target = self.dbg.CreateTarget("")
         err = lldb.SBError()
-        process = target.LoadCore(self.corefile)
+        process = target.LoadCore(corefile)
         self.assertTrue(process.IsValid())
         thread = process.GetSelectedThread()
         frame = thread.GetSelectedFrame()
@@ -50,11 +44,12 @@ def test_armv7_corefile(self):
 
     def test_arm64_corefile(self):
         ### Create corefile
-        retcode = call(self.create_corefile + " arm64 " + self.corefile, shell=True)
+        corefile = self.getBuildArtifact("core")
+        self.yaml2macho_core("arm64.yaml", corefile)
 
         target = self.dbg.CreateTarget("")
         err = lldb.SBError()
-        process = target.LoadCore(self.corefile)
+        process = target.LoadCore(corefile)
         self.assertTrue(process.IsValid())
         thread = process.GetSelectedThread()
         frame = thread.GetSelectedFrame()

lldb/test/API/macosx/arm-corefile-regctx/TestArmMachoCorefileRegctx.py

@@ -0,0 +1,30 @@
+cpu: arm64
+threads:
+  # (lldb) reg read
+  # % pbpaste | grep = | sed 's, ,,g' | awk -F= '{print "{name: " $1 ", value: " $2 "},"}'
+  - regsets:
+      - flavor: gpr
+        registers: [
+           {name: x0, value: 0x0000000000000001}, {name: x1, value: 0x000000016fdff3c0},
+           {name: x2, value: 0x000000016fdff3d0}, {name: x3, value: 0x000000016fdff510},
+           {name: x4, value: 0x0000000000000000}, {name: x5, value: 0x0000000000000000},
+           {name: x6, value: 0x0000000000000000}, {name: x7, value: 0x0000000000000000},
+           {name: x8, value: 0x000000010000d910}, {name: x9, value: 0x0000000000000001},
+           {name: x10, value: 0xe1e88de000000000}, {name: x11, value: 0x0000000000000003},
+           {name: x12, value: 0x0000000000000148}, {name: x13, value: 0x0000000000004000},
+           {name: x14, value: 0x0000000000000008}, {name: x15, value: 0x0000000000000000},
+           {name: x16, value: 0x0000000000000000}, {name: x17, value: 0x0000000100003f5c},
+           {name: x18, value: 0x0000000000000000}, {name: x19, value: 0x0000000100003f5c},
+           {name: x20, value: 0x000000010000c000}, {name: x21, value: 0x000000010000d910},
+           {name: x22, value: 0x000000016fdff250}, {name: x23, value: 0x000000018ce12366},
+           {name: x24, value: 0x000000016fdff1d0}, {name: x25, value: 0x0000000000000001},
+           {name: x26, value: 0x0000000000000000}, {name: x27, value: 0x0000000000000000},
+           {name: x28, value: 0x0000000000000000}, {name: fp, value: 0x000000016fdff3a0},
+           {name: lr, value: 0x000000018cd97f28}, {name: sp, value: 0x000000016fdff140},
+           {name: pc, value: 0x0000000100003f5c}, {name: cpsr, value: 0x80001000}
+        ]
+      - flavor: exc
+        registers: [ {name: far, value: 0x0000000100003f5c}, 
+                     {name: esr, value: 0xf2000000}, 
+                     {name: exception, value: 0x00000000}
+                   ]

lldb/test/API/macosx/arm-corefile-regctx/arm64.yaml

@@ -0,0 +1,36 @@
+cpu: armv7m
+threads:
+  # (lldb) reg read
+  # % pbpaste | grep = | sed 's, ,,g' | awk -F= '{print "{name: " $1 ", value: " $2 "},"}'
+  - regsets:
+      - flavor: gpr
+        registers: [
+          {name: r0, value: 0x00010000}, {name: r1, value: 0x00020000},
+          {name: r2, value: 0x00030000}, {name: r3, value: 0x00040000},
+          {name: r4, value: 0x00050000}, {name: r5, value: 0x00060000},
+          {name: r6, value: 0x00070000}, {name: r7, value: 0x00080000},
+          {name: r8, value: 0x00090000}, {name: r9, value: 0x000a0000},
+          {name: r10, value: 0x000b0000}, {name: r11, value: 0x000c0000},
+          {name: r12, value: 0x000d0000}, {name: sp, value: 0x000e0000},
+          {name: lr, value: 0x000f0000}, {name: pc, value: 0x00100000},
+          {name: cpsr, value: 0x00110000}
+        ]
+      - flavor: exc
+        registers: [ {name: far, value: 0x00003f5c},
+                     {name: esr, value: 0xf2000000},
+                     {name: exception, value: 0x00000000}
+                   ]
+
+memory-regions:
+  # $sp is 0x000e0000, have bytes surrounding that address
+  - addr: 0x000dffe0
+    UInt8: [
+            0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+            0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11,
+            0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a,
+            0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x21, 0x22, 0x23,
+            0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c,
+            0x2d, 0x2e, 0x2f, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35,
+            0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 
+            0x3f
+           ]