[lldb] Add utility to create Mach-O corefile from YAML desc

I've wanted a utility to create a corefile for test purposes given
a bit of memory and regsters, for a while.  I've written a few API
tests over the years that needed exactly this capability -- we have
several one-off Mach-O corefile creator utility in the API testsuite
to do this.  But it's a lot of boilerplate when you only want to
specify some register contents and memory contents, to create an
API test.

This adds yaml2mach-core, a tool that should build on any system,
takes a yaml description of register values for one or more threads,
optionally memory values for one or more memory regions, and can
take a list of UUIDs that will be added as LC_NOTE "load binary"
metadata to the corefile so binaries can be loaded into virtual
address space in a test scenario.

The format of the yaml file looks like

cpu: armv7m
endian: little
threads:
  - regsets:
      - flavor: gpr
        registers: [{name: sp, value: 0x2000fe70}, {name: r7, value: 0x2000fe80},
                    {name: pc, value: 0x0020392c}, {name: lr, value: 0x0020392d}]

memory-regions:
  - addr: 0x2000fe70
    UInt32: [
      0x0000002a, 0x20010e58, 0x00203923, 0x00000001,
      0x2000fe88, 0x00203911, 0x2000ffdc, 0xfffffff9
    ]
  - addr: 0x203910
    UInt8: [
      0xf8, 0xb5, 0x04, 0xaf, 0x06, 0x4c, 0x07, 0x49,
      0x74, 0xf0, 0x2e, 0xf8, 0x01, 0xac, 0x74, 0xf0
    ]

and that's all that is needed to specify a corefile where four register
values are specified (the others will be set to 0), and two memory
regions will be emitted.

The memory can be specified as an array of UInt8, UInt32, or UInt64,
I anticipate that some of these corefiles may have stack values
constructed manually and it may be simpler for a human to write
them in a particular grouping of values.

Accepting "endian" is probably a boondoggle that won't ever come
to any use, and honestly I don't 100% know what the correct byte
layout would be for a big endian Mach-O file any more.  In a RISC-V
discussion a month ago, it was noted that register byte layout will
be little endian even when there is a big endian defined format for
RV, so memory would be byteswapped but registers would not.  It may
have been better not to pretend to support this, but on the other
hand it might be neat to be able to generate a big endian test case
simply.

I needed this utility for an upcoming patch for ARM Cortex-M processors,
to create a test for the change.  I took the opportunity to remove two
of the "trivial mach-o corefile" creator utilities I've written in the
past, which also restricted the tests to only run on Darwin systems
because I was using the system headers for Mach-O constant values.

rdar://110663219

Author: jasonmolenda

lldb/packages/Python/lldbsuite/test/configuration.py

@@ -64,6 +64,9 @@
 # Path to the yaml2obj tool. Not optional.
 yaml2obj = None
 
+# Path to the yaml2macho-core tool. Not optional.
+yaml2macho_core = None
+
 # The arch might dictate some specific CFLAGS to be passed to the toolchain to build
 # the inferior programs.  The global variable cflags_extras provides a hook to do
 # just that.
@@ -174,3 +177,10 @@ def get_yaml2obj_path():
     """
     if yaml2obj and os.path.lexists(yaml2obj):
         return yaml2obj
+
+def get_yaml2macho_core_path():
+    """
+    Get the path to the yaml2macho-core tool.
+    """
+    if yaml2macho_core and os.path.lexists(yaml2macho_core):
+        return yaml2macho_core

lldb/packages/Python/lldbsuite/test/dotest.py

@@ -280,6 +280,7 @@ def parseOptionsAndInitTestdirs():
         configuration.llvm_tools_dir = args.llvm_tools_dir
         configuration.filecheck = shutil.which("FileCheck", path=args.llvm_tools_dir)
         configuration.yaml2obj = shutil.which("yaml2obj", path=args.llvm_tools_dir)
+        configuration.yaml2macho_core = shutil.which("yaml2macho-core", path=args.llvm_tools_dir)
 
     if not configuration.get_filecheck_path():
         logging.warning("No valid FileCheck executable; some tests may fail...")

lldb/packages/Python/lldbsuite/test/lldbtest.py

@@ -1702,6 +1702,21 @@ def yaml2obj(self, yaml_path, obj_path, max_size=None):
             command += ["--max-size=%d" % max_size]
         self.runBuildCommand(command)
 
+    def yaml2macho_core(self, yaml_path, obj_path, uuids=None):
+        """
+        Create a Mach-O corefile at the given path from a yaml file.
+
+        Throws subprocess.CalledProcessError if the object could not be created.
+        """
+        yaml2macho_core_bin = configuration.get_yaml2macho_core_path()
+        if not yaml2macho_core_bin:
+            self.assertTrue(False, "No valid yaml2macho-core executable specified")
+        if uuids != None:
+          command = [yaml2macho_core_bin, "-i", yaml_path, "-o", obj_path, "-u", uuids]
+        else:
+          command = [yaml2macho_core_bin, "-i", yaml_path, "-o", obj_path]
+        self.runBuildCommand(command)
+
     def cleanup(self, dictionary=None):
         """Platform specific way to do cleanup after build."""
         module = builder_module()

lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.cpp

@@ -12,6 +12,7 @@
 #include "lldb/Core/PluginManager.h"
 #include "lldb/Core/Section.h"
 #include "lldb/Symbol/Symbol.h"
+#include "lldb/Target/Target.h"
 #include "lldb/Utility/LLDBLog.h"
 #include "lldb/Utility/Log.h"
 #include "llvm/ADT/DenseSet.h"
@@ -233,6 +234,41 @@ void ObjectFileJSON::CreateSections(SectionList &unified_section_list) {
   }
 }
 
+bool ObjectFileJSON::SetLoadAddress(Target &target, lldb::addr_t value,
+                                    bool value_is_offset) {
+  Log *log(GetLog(LLDBLog::DynamicLoader));
+  if (!m_sections_up)
+    return true;
+
+  const bool warn_multiple = true;
+
+  addr_t slide = value;
+  if (!value_is_offset) {
+    addr_t lowest_addr = LLDB_INVALID_ADDRESS;
+    for (const SectionSP &section_sp : *m_sections_up) {
+      addr_t section_load_addr = section_sp->GetFileAddress();
+      lowest_addr = std::min(lowest_addr, section_load_addr);
+    }
+    if (lowest_addr == LLDB_INVALID_ADDRESS)
+      return false;
+    slide = value - lowest_addr;
+  }
+
+  // Apply slide to each section's file address.
+  for (const SectionSP &section_sp : *m_sections_up) {
+    addr_t section_load_addr = section_sp->GetFileAddress();
+    if (section_load_addr != LLDB_INVALID_ADDRESS) {
+      LLDB_LOGF(
+          log,
+          "ObjectFileJSON::SetLoadAddress section %s to load addr 0x%" PRIx64,
+          section_sp->GetName().AsCString(), section_load_addr + slide);
+      target.SetSectionLoadAddress(section_sp, section_load_addr + slide,
+                                   warn_multiple);
+    }
+  }
+  return true;
+}
+
 bool ObjectFileJSON::MagicBytesMatch(DataBufferSP data_sp,
                                      lldb::addr_t data_offset,
                                      lldb::addr_t data_length) {

lldb/source/Plugins/ObjectFile/JSON/ObjectFileJSON.h

@@ -86,6 +86,9 @@ class ObjectFileJSON : public ObjectFile {
 
   Strata CalculateStrata() override { return eStrataUser; }
 
+  bool SetLoadAddress(Target &target, lldb::addr_t value,
+                      bool value_is_offset) override;
+
   static bool MagicBytesMatch(lldb::DataBufferSP data_sp, lldb::addr_t offset,
                               lldb::addr_t length);
 

lldb/test/API/macosx/arm-corefile-regctx/Makefile

@@ -13,20 +13,14 @@
 class TestArmMachoCorefileRegctx(TestBase):
     NO_DEBUG_INFO_TESTCASE = True
 
-    @skipUnlessDarwin
-    def setUp(self):
-        TestBase.setUp(self)
-        self.build()
-        self.create_corefile = self.getBuildArtifact("a.out")
-        self.corefile = self.getBuildArtifact("core")
-
     def test_armv7_corefile(self):
         ### Create corefile
-        retcode = call(self.create_corefile + " armv7 " + self.corefile, shell=True)
+        corefile = self.getBuildArtifact("core")
+        self.yaml2macho_core("armv7m.yaml", corefile)
 
         target = self.dbg.CreateTarget("")
         err = lldb.SBError()
-        process = target.LoadCore(self.corefile)
+        process = target.LoadCore(corefile)
         self.assertTrue(process.IsValid())
         thread = process.GetSelectedThread()
         frame = thread.GetSelectedFrame()
@@ -50,11 +44,12 @@ def test_armv7_corefile(self):
 
     def test_arm64_corefile(self):
         ### Create corefile
-        retcode = call(self.create_corefile + " arm64 " + self.corefile, shell=True)
+        corefile = self.getBuildArtifact("core")
+        self.yaml2macho_core("arm64.yaml", corefile)
 
         target = self.dbg.CreateTarget("")
         err = lldb.SBError()
-        process = target.LoadCore(self.corefile)
+        process = target.LoadCore(corefile)
         self.assertTrue(process.IsValid())
         thread = process.GetSelectedThread()
         frame = thread.GetSelectedFrame()

lldb/test/API/macosx/arm-corefile-regctx/TestArmMachoCorefileRegctx.py

@@ -0,0 +1,31 @@
+cpu: arm64
+endian: little
+threads:
+  # (lldb) reg read
+  # % pbpaste | grep = | sed 's, ,,g' | awk -F= '{print "{name: " $1 ", value: " $2 "},"}'
+  - regsets:
+      - flavor: gpr
+        registers: [
+           {name: x0, value: 0x0000000000000001}, {name: x1, value: 0x000000016fdff3c0},
+           {name: x2, value: 0x000000016fdff3d0}, {name: x3, value: 0x000000016fdff510},
+           {name: x4, value: 0x0000000000000000}, {name: x5, value: 0x0000000000000000},
+           {name: x6, value: 0x0000000000000000}, {name: x7, value: 0x0000000000000000},
+           {name: x8, value: 0x000000010000d910}, {name: x9, value: 0x0000000000000001},
+           {name: x10, value: 0xe1e88de000000000}, {name: x11, value: 0x0000000000000003},
+           {name: x12, value: 0x0000000000000148}, {name: x13, value: 0x0000000000004000},
+           {name: x14, value: 0x0000000000000008}, {name: x15, value: 0x0000000000000000},
+           {name: x16, value: 0x0000000000000000}, {name: x17, value: 0x0000000100003f5c},
+           {name: x18, value: 0x0000000000000000}, {name: x19, value: 0x0000000100003f5c},
+           {name: x20, value: 0x000000010000c000}, {name: x21, value: 0x000000010000d910},
+           {name: x22, value: 0x000000016fdff250}, {name: x23, value: 0x000000018ce12366},
+           {name: x24, value: 0x000000016fdff1d0}, {name: x25, value: 0x0000000000000001},
+           {name: x26, value: 0x0000000000000000}, {name: x27, value: 0x0000000000000000},
+           {name: x28, value: 0x0000000000000000}, {name: fp, value: 0x000000016fdff3a0},
+           {name: lr, value: 0x000000018cd97f28}, {name: sp, value: 0x000000016fdff140},
+           {name: pc, value: 0x0000000100003f5c}, {name: cpsr, value: 0x80001000}
+        ]
+      - flavor: exc
+        registers: [ {name: far, value: 0x0000000100003f5c}, 
+                     {name: esr, value: 0xf2000000}, 
+                     {name: exception, value: 0x00000000}
+                   ]

lldb/test/API/macosx/arm-corefile-regctx/arm64.yaml

@@ -0,0 +1,37 @@
+cpu: armv7m
+endian: little
+threads:
+  # (lldb) reg read
+  # % pbpaste | grep = | sed 's, ,,g' | awk -F= '{print "{name: " $1 ", value: " $2 "},"}'
+  - regsets:
+      - flavor: gpr
+        registers: [
+          {name: r0, value: 0x00010000}, {name: r1, value: 0x00020000},
+          {name: r2, value: 0x00030000}, {name: r3, value: 0x00040000},
+          {name: r4, value: 0x00050000}, {name: r5, value: 0x00060000},
+          {name: r6, value: 0x00070000}, {name: r7, value: 0x00080000},
+          {name: r8, value: 0x00090000}, {name: r9, value: 0x000a0000},
+          {name: r10, value: 0x000b0000}, {name: r11, value: 0x000c0000},
+          {name: r12, value: 0x000d0000}, {name: sp, value: 0x000e0000},
+          {name: lr, value: 0x000f0000}, {name: pc, value: 0x00100000},
+          {name: cpsr, value: 0x00110000}
+        ]
+      - flavor: exc
+        registers: [ {name: far, value: 0x00003f5c},
+                     {name: esr, value: 0xf2000000},
+                     {name: exception, value: 0x00000000}
+                   ]
+
+memory-regions:
+  # $sp is 0x000e0000, have bytes surrounding that address
+  - addr: 0x000dffe0
+    UInt8: [
+            0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+            0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11,
+            0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a,
+            0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x21, 0x22, 0x23,
+            0x24, 0x25, 0x26, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c,
+            0x2d, 0x2e, 0x2f, 0x30, 0x31, 0x32, 0x33, 0x34, 0x35,
+            0x36, 0x37, 0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 
+            0x3f
+           ]