Assembly macros update (#466)

* Change the default for the labels

* Update macro.inc and include_asm.h each time splat is run

* Option to customize generated macro.inc file

* use endlabels in incbins

* changelog and docs

* black

* update docs

* Update tests

* review changes and some fixes

* path fixes

* nmlabel -> nonmatching

* Use configured labels instead of hardcoded ones

* fix python 3.9 not having newline argument on write_text

* Fix windows?

Author: AngheloAlf

CHANGELOG.md

@@ -2,9 +2,25 @@
 
 ### 0.35.0
 
-* Add `pair_segment` option to segments:
+* Add `pair_segment` option to segments.
   * Allows pairing the sections of two different segments together, making cross-segment rodata migration possible.
 * Now `create_config` for N64 games can create basic `symbol_addrs.txt` and `reloc_addrs.txt` files from the information inferred from its analysis.
+* splat will now start generating and regenerating files related to assembly macros, allowing less painful updates to newer splat versions.
+  * This includes generating files like `include/include_asm.h`, `include/macro.inc`, `include/labels.inc` and `include/gte_macros.inc`.
+  * This makes easier for splat to introduce new kinds of labels or update the current definitions with minimal pain for the end-user.
+  * This behaviour can be slightly customized with `generated_macro_inc_content` or completely turned off with `generate_asm_macros_files`.
+  * Both `splat` and `create_config` have this behaviour.
+* New yaml options:
+  * `asm_nonmatching_label_macro`: Set the label to be used to signal an assembly symbol haven't been matched yet. Defaults to `nonmatching`.
+  * `asm_data_end_label`: Set the label used at the end of a data symbol. Defaults to `enddlabel`.
+  * `generated_macro_inc_content`: A string defining custom contents to be emitted at the generated `include/macro.inc` file.
+  * `generate_asm_macros_files`: Controls if the assembly macros files should be regenerated by splat or not.
+* Change the default value for multiple assembly macro labels.
+  * `asm_function_alt_macro`: Changed from `glabel` to `alabel`.
+  * `asm_jtbl_label_macro`: Changed from `glabel` to `jlabel`.
+  * `asm_data_macro`: Changed from `glabel` to `dlabel`.
+  * `asm_end_label`: Changed from empty to `endlabel`.
+* `spimdisasm` 1.36.0 or above is now required.
 
 ### 0.34.3
 

docs/Configuration.md

@@ -122,6 +122,21 @@ String that is placed before the contents of newly-generated assembly (`.s`) fil
 generated_s_preamble: .set fp=64
 ```
 
+### generated_macro_inc_content
+
+String that is placed after the contents of the splat-generated `macro.inc` file.
+
+### generate_asm_macros_files
+
+Tells splat to regenerate files containing assembly macros and C macros each time splat is run.
+
+Specifically splat generates `include/include_asm.h`, `include/macro.inc`, `include/labels.inc` and `include/gte_macros.inc`, which contain the proper C and assembly macro definitions expected for building the generated assembly correctly. This allows splat to update the macro definitions with minimal user intervention and headaches.
+
+Turning this off can be useful in case the user wants to control exactly the contents of those files, but it is not recommended, since the user definitions may get outdated. Before turning this option off consider using the [`generated_macro_inc_content`](#generated_macro_inc_content) option to customize the contents of the generated `macro.inc` file.
+
+Some files may not be generated depending on the selected platform and compiler, because those setups don't require them. For example `include_asm.h` won't be generated if the compiler is set to `IDO` or `MWCCPS2`. `include/gte_macros.inc` is only generated on psx projects.
+
+Defaults to `True`.
 
 ### o_as_suffix
 
@@ -653,26 +668,48 @@ Determines the macro used to declare functions in asm files
 
 ### asm_function_alt_macro
 
-Determines the macro used to declare symbols in the middle of functions in asm files (which may be alternative entries)
+Determines the macro used to declare symbols in the middle of functions in asm files (which may be alternative entries).
+
+Defaults to `alabel`.
 
 ### asm_jtbl_label_macro
 
-Determines the macro used to declare jumptable labels in asm files
+Determines the macro used to declare jumptable labels in asm files.
+
+Defaults to `jlabel`.
 
 ### asm_data_macro
 
-Determines the macro used to declare data symbols in asm files
+Determines the macro used to declare data symbols in asm files.
+
+Defaults to `dlabel`.
 
 ### asm_end_label
 
-Determines the macro used at the end of a function, such as endlabel or .end
+Determines the macro used at the end of a function, such as `endlabel` or `.end`.
+
+Defaults to `endlabel`.
+
+### asm_data_end_label
+
+Determines the macro used at the end of a data symbol.
+
+Defaults to `enddlabel`.
 
 ### asm_ehtable_label_macro
 
 Determines the macro used to declare ehtable labels in asm files.
 
 Defaults to `ehlabel`
 
+### asm_nonmatching_label_macro
+
+Determines the macro used to declare the given symbol is a non matching one.
+
+Explicitly specifying that a symbol haven't been matched yet in the generated assembly is useful for other tools that consume the build artifacts of the project. This information can be used by those tools for stuff like progress reporting.
+
+Defaults to `nonmatching`
+
 ### asm_emit_size_directive
 
 Toggles the .size directive emitted by the disassembler

docs/General-Workflow.md

@@ -118,7 +118,7 @@ The macros to include text/rodata assembly are different for GCC vs IDO compiler
 **GCC**: `INCLUDE_ASM` & `INCLUDE_RODATA` (text/rodata respectively)
 **IDO**: `GLOBAL_ASM`
 
-These macros must be defined in an included header, which splat currently does not produce.
+These macros must be defined in an included header, which splat generates and updates by default for GCC-based projects.
 
 For a GCC example, see the [include.h](https://github.com/AngheloAlf/drmario64/blob/master/include/include_asm.h) from the Dr. Mario project.
 
@@ -130,27 +130,75 @@ For MWCC, you will need [mwccgap](https://github.com/mkst/mwccgap) to include as
 
 splat relies on some assembly macros for the asm generation. They usually live on the `include/macro.inc` file. Without these macros then an assembler would not be able to build our disassemblies.
 
+By default splat will generate files with the required assembly macros.
+
 Those macros usually look like this:
 
 ```mips
-.macro glabel label
-    .global \label
+# A function symbol.
+.macro glabel label, visibility=global
+    .\visibility \label
     .type \label, @function
     \label:
+        .ent \label
 .endm
 
-.macro dlabel label
-    .global \label
+# The end of a function symbol.
+.macro endlabel label
+    .size \label, . - \label
+    .end \label
+.endm
+
+# An alternative entry to a function.
+.macro alabel label, visibility=global
+    .\visibility \label
+    .type \label, @function
     \label:
+        .aent \label
 .endm
 
+# A label referenced by an error handler table.
+.macro ehlabel label, visibility=global
+    .\visibility \label
+    \label:
+.endm
+
+
+# A label referenced by a jumptable.
 .macro jlabel label
     .global \label
     \label:
 .endm
+
+
+# A data symbol.
+.macro dlabel label, visibility=global
+    .\visibility \label
+    .type \label, @object
+    \label:
+.endm
+
+# End of a data symbol.
+.macro enddlabel label
+    .size \label, . - \label
+.endm
+
+
+# Label to signal the symbol haven't been matched yet.
+.macro nonmatching label, size=1
+    .global \label\().NON_MATCHING
+    .type \label\().NON_MATCHING, @object
+    .size \label\().NON_MATCHING, \size
+    \label\().NON_MATCHING:
+.endm
 ```
 
-Where `glabel` is used for functions, `dlabel` is used for data, rodata and bss variables and `jlabel` is used for branch labels used by jumptables.
+The most commonly used labels are:
+
+- `glabel` and `endlabel` which are used to define function symbols.
+- `dlabel` and `enddlabel` which are used to defined data, rodata and bss symbols.
+- `jlabel` is used for defining branch labels used by jumptables.
+- `nonmatching` is used to define the symbol haven't been matched yet.
 
 Asm differ tools can sometimes struggle to show diffs with `jlabel`s when combined with certain compilers. A workaround for this issue is to mark the `jlabel` as a function, like this:
 

requirements.txt

@@ -4,7 +4,7 @@ tqdm
 intervaltree
 colorama
 # This value should be keep in sync with the version listed on disassembler/spimdisasm_disassembler.py and pyproject.toml
-spimdisasm>=1.33.0
+spimdisasm>=1.36.0
 rabbitizer>=1.10.0
 pygfxd
 n64img>=0.1.4

src/splat/disassembler/spimdisasm_disassembler.py

@@ -7,7 +7,7 @@
 
 class SpimdisasmDisassembler(disassembler.Disassembler):
     # This value should be kept in sync with the version listed on requirements.txt and pyproject.toml
-    SPIMDISASM_MIN = (1, 33, 0)
+    SPIMDISASM_MIN = (1, 36, 0)
 
     def configure(self):
         # Configure spimdisasm
@@ -93,9 +93,15 @@ def configure(self):
         )
         spimdisasm.common.GlobalConfig.ASM_DATA_LABEL = options.opts.asm_data_macro
         spimdisasm.common.GlobalConfig.ASM_TEXT_END_LABEL = options.opts.asm_end_label
+        spimdisasm.common.GlobalConfig.ASM_DATA_END_LABEL = (
+            options.opts.asm_data_end_label
+        )
         spimdisasm.common.GlobalConfig.ASM_EHTBL_LABEL = (
             options.opts.asm_ehtable_label_macro
         )
+        spimdisasm.common.GlobalConfig.ASM_NM_LABEL = (
+            options.opts.asm_nonmatching_label_macro
+        )
 
         if options.opts.asm_emit_size_directive is not None:
             spimdisasm.common.GlobalConfig.ASM_EMIT_SIZE_DIRECTIVE = (

src/splat/scripts/create_config.py

@@ -6,7 +6,7 @@
 
 from ..util.n64 import find_code_length, rominfo
 from ..util.psx import psxexeinfo
-from ..util import log
+from ..util import log, file_presets, conf
 
 
 def main(file_path: Path):
@@ -70,10 +70,6 @@ def create_n64_config(rom_path: Path):
   use_legacy_include_asm: False
   mips_abi_float_regs: o32
 
-  asm_function_macro: glabel
-  asm_jtbl_label_macro: jlabel
-  asm_data_macro: dlabel
-
   # section_order: [".text", ".data", ".rodata", ".bss"]
   # auto_link_sections: [".data", ".rodata", ".bss"]
 
@@ -176,12 +172,17 @@ def create_n64_config(rom_path: Path):
   - [0x{rom.size:X}]
 """
 
-    out_file = f"{cleaned_basename}.yaml"
-    with open(out_file, "w", newline="\n") as f:
+    out_file = Path(f"{cleaned_basename}.yaml")
+    with out_file.open("w", newline="\n") as f:
         print(f"Writing config to {out_file}")
         f.write(header)
         f.write(segments)
 
+    # `file_presets` requires an initialized `opts`.
+    # A simple way to do that is to simply load the yaml we just generated.
+    conf.load([out_file])
+    file_presets.write_all_files()
+
     # Write reloc_addrs.txt file
     reloc_addrs = []
     if rom.entrypoint_info.bss_start_address is not None:
@@ -289,10 +290,6 @@ def create_psx_config(exe_path: Path, exe_bytes: bytes):
   o_as_suffix: True
   use_legacy_include_asm: False
 
-  asm_function_macro: glabel
-  asm_jtbl_label_macro: jlabel
-  asm_data_macro: dlabel
-
   section_order: [".rodata", ".text", ".data", ".bss"]
   # auto_link_sections: [".data", ".rodata", ".bss"]
 
@@ -329,7 +326,7 @@ def create_psx_config(exe_path: Path, exe_bytes: bytes):
 """
     text_offset = exe.text_offset
     if text_offset != 0x800:
-        segments += f"""\
+        segments += """\
       - [0x800, rodata, 800]
 """
     segments += f"""\
@@ -346,12 +343,17 @@ def create_psx_config(exe_path: Path, exe_bytes: bytes):
   - [0x{exe.size:X}]
 """
 
-    out_file = f"{cleaned_basename}.yaml"
-    with open(out_file, "w", newline="\n") as f:
+    out_file = Path(f"{cleaned_basename}.yaml")
+    with out_file.open("w", newline="\n") as f:
         print(f"Writing config to {out_file}")
         f.write(header)
         f.write(segments)
 
+    # `file_presets` requires an initialized `opts`.
+    # A simple way to do that is to simply load the yaml we just generated.
+    conf.load([out_file])
+    file_presets.write_all_files()
+
 
 def add_arguments_to_parser(parser: argparse.ArgumentParser):
     parser.add_argument(

src/splat/scripts/split.py

@@ -8,9 +8,9 @@
 
 from .. import __package_name__, __version__
 from ..disassembler import disassembler_instance
-from ..util import cache_handler, progress_bar, vram_classes, statistics
+from ..util import cache_handler, progress_bar, vram_classes, statistics, file_presets
 
-from colorama import Fore, Style
+from colorama import Style
 from intervaltree import Interval, IntervalTree
 import sys
 
@@ -571,6 +571,8 @@ def main(
     if options.opts.is_mode_active("code"):
         dump_symbols()
 
+        file_presets.write_all_files()
+
 
 def add_arguments_to_parser(parser: argparse.ArgumentParser):
     parser.add_argument(

src/splat/segtypes/common/textbin.py

@@ -97,6 +97,8 @@ def write_asm_contents(self, rom_bytes, f: TextIO):
 
             if self.is_text() and options.opts.asm_end_label != "":
                 f.write(f"{options.opts.asm_end_label} {sym.name}\n")
+            elif self.is_data() and options.opts.asm_data_end_label != "":
+                f.write(f"{options.opts.asm_data_end_label} {sym.name}\n")
 
             if sym.given_name_end is not None:
                 if (
@@ -106,6 +108,12 @@ def write_asm_contents(self, rom_bytes, f: TextIO):
                     f.write(f"{asm_label} {sym.given_name_end}\n")
                     if asm_label == ".globl":
                         f.write(f"{sym.given_name_end}:\n")
+                    if self.is_text() and options.opts.asm_end_label != "":
+                        f.write(f"{options.opts.asm_end_label} {sym.given_name_end}\n")
+                    elif self.is_data() and options.opts.asm_data_end_label != "":
+                        f.write(
+                            f"{options.opts.asm_data_end_label} {sym.given_name_end}\n"
+                        )
 
     def split(self, rom_bytes):
         if self.rom_end is None:

src/splat/segtypes/linker_entry.py

@@ -522,7 +522,7 @@ def save_symbol_header(self):
             )
 
     def save_dependencies_file(self, output_path: Path, target_elf_path: Path):
-        output = f"{target_elf_path.as_posix()}:"
+        output = f"{clean_up_path(target_elf_path).as_posix()}:"
 
         for entry in self.dependencies_entries:
             if entry.object_path is None:

src/splat/util/__init__.py

@@ -1,6 +1,7 @@
 from . import cache_handler as cache_handler
 from . import color as color
 from . import compiler as compiler
+from . import file_presets as file_presets
 from . import log as log
 from . import n64 as n64
 from . import options as options