ABI 'o32' is incompatible with target ABI 'n64'

[command-tab]

I'm attempting to link in a Nintendo 64 SDK static library (libnusys.a), but the Rust linker errors out:

rust-lld: error: lto.tmp: ABI 'o32' is incompatible with target ABI 'n64'

Do you think I ought be producing o32 ABI binaries instead of n64 ones to match the static library? I'm curious what you make of this.

Here's my build.rs:

fn main() {
    println!("cargo:rustc-link-lib=static=nusys");
    println!("cargo:rustc-link-search=native=/path/to/n64kit/nusys/lib");
}

Thanks!

[CrashOveride95]

At least on other linkers such as GNU Linker, I've always had to compile the code with o32 ABI otherwise linker will whine about the libraries being made for o32

Does it fail on building with o32?

[command-tab]

Hmm, I'm not sure how to get Rust to build o32, or if it even supports it 🤔

In PR #20, although this is not likely to be merged, I was able to link o32 newlib in without any issues. So it's definitely worked in the past.

[parasyte]

@command-tab You can build with cargo n64 build -vv to get verbose output, this will print the linker command (the last command)... You can then copy-paste that huge command ad append --verbose so that lld will print the object files that it is adding to lto.tmp. This won't solve the issue, but it could help with troubleshooting.

FWIW I had this same error from lld before I added the -C linker-plugin-lto arg for rustc.

The other workaround is just building with lto disabled. :(

[command-tab]

I did manage to get that command, locate rust-lld, and run it verbosely:

$ /Users/me/.rustup/toolchains/nightly-2020-05-15-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin/rust-lld "-flavor" "gnu" "--script=/var/folders/j3/7pf540td3xncn7jdqtb62_ph00058y/T/n64-build/linker.ld" "-plugin-opt=O3" "-plugin-opt=mcpu=mips3" "-L" "/Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/nu0-7f08bd2693117a76.nu0.allkffuq-cgu.0.rcgu.o" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/nu0-7f08bd2693117a76.nu0.allkffuq-cgu.1.rcgu.o" "-o" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/nu0-7f08bd2693117a76" "--gc-sections" "-L" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps" "-L" "/Users/me/src/cargo-n64/target/release/deps" "-L" "/Users/me/src/cargo-n64/examples/nu0/lib" "-L" "/Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib" "-Bstatic" "--whole-archive" "-lnusys" "--no-whole-archive" "--start-group" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/librrt0-c96d70ae75dc02b2.rlib" "/Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/liblibm-67173c6c13da7e27.rlib" "/Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/librustc_std_workspace_core-29be43fe69db4d9c.rlib" "/Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/libcore-071b07f42a980aa2.rlib" "--end-group" "/Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/libcompiler_builtins-ac0e6e2cd53c5dca.rlib" "-Bdynamic" --verbose
rust-lld: /var/folders/j3/7pf540td3xncn7jdqtb62_ph00058y/T/n64-build/linker.ld
rust-lld: /Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/nu0-7f08bd2693117a76.nu0.allkffuq-cgu.0.rcgu.o
rust-lld: /Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/nu0-7f08bd2693117a76.nu0.allkffuq-cgu.1.rcgu.o
rust-lld: /Users/me/src/cargo-n64/examples/nu0/lib/libnusys.a
rust-lld: /Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/librrt0-c96d70ae75dc02b2.rlib
rust-lld: /Users/me/src/cargo-n64/target/mips-nintendo64-none/release/deps/liblibm-67173c6c13da7e27.rlib
rust-lld: /Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/librustc_std_workspace_core-29be43fe69db4d9c.rlib
rust-lld: /Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/libcore-071b07f42a980aa2.rlib
rust-lld: /Users/me/src/cargo-n64/target/sysroot/lib/rustlib/mips-nintendo64-none/lib/libcompiler_builtins-ac0e6e2cd53c5dca.rlib
rust-lld: error: lto.tmp: ABI 'o32' is incompatible with target ABI 'n64'
rust-lld: error: lto.tmp: ABI 'o32' is incompatible with target ABI 'n64'
rust-lld: error: lto.tmp: ABI 'o32' is incompatible with target ABI 'n64'
...

I tried building with lto off in my example, but that didn't seem to have any effect. I had this in my Cargo.toml at /Users/me/src/cargo-n64/examples/nu0/Cargo.toml:

[profile.release]
lto = "off"

Should I be installing and running cargo-n64 with lto off, not my example? It also strikes me that I may not even be using the release profile. I should probably step back and get more familiar with Cargo.toml before jumping into the deep end 😂

Thanks for the help so far!

[parasyte]

You are linking against nusys. Personally I've never tried this! Don't know what to expect.

But I believe that rust-lang/cargo#8066 has given us some trouble recently, and you might be seeing similar issues. For more context on that change, see rust-lang/rust#66961

LTO on the cargo-n64 build won't change the behavior here. And don't worry about whether you're using the release profile; cargo-n64 enforces it. (We can't build debug profile due to lack of some intrinsics, IIRC. So I just always build in release mode to workaround that issue.)

Now that you have the list of library archive files, you could extract each one with ar (it should be noted that I had trouble extracting on macOS) and then inspect each individual object for ABI info.

[parasyte]

@command-tab I found today that cargo-n64 builds with the o32 ABI because of the LLVM features selected. Switch to n64 ABI with the following patch:

diff --git a/cargo-n64/src/templates/mips-nintendo64-none.fmt b/cargo-n64/src/templates/mips-nintendo64-none.fmt
index 77a7ab5..a01b497 100644
--- a/cargo-n64/src/templates/mips-nintendo64-none.fmt
+++ b/cargo-n64/src/templates/mips-nintendo64-none.fmt
@@ -1,14 +1,15 @@
 {{
   "arch": "mips",
   "cpu": "mips3",
-  "data-layout": "E-m:m-p:32:32-i8:8:32-i16:16:32-i64:64-n32:64-S64",
+  "data-layout": "E-m:m-p:64:64-i8:8:32-i16:16:32-i64:64-n64-S64",
   "disable-redzone": true,
   "env": "unknown",
   "executables": true,
-  "features": "+mips3,+gp64,+fpxx,+nooddspreg",
+  "features": "+mips3,+gp64",
   "linker": "rust-lld",
   "linker-flavor": "ld.lld",
   "llvm-target": "mips-unknown-unknown",
+  "llvm-abiname": "n64",
   "os": "none",
   "panic-strategy": "abort",
   "pre-link-args": {{
@@ -17,8 +18,8 @@
     ]
   }},
   "relocation-model": "static",
-  "target-c-int-width": "32",
+  "target-c-int-width": "64",
   "target-endian": "big",
-  "target-pointer-width": "32",
+  "target-pointer-width": "64",
   "vendor": "nintendo64"
 }}

I didn't notice any differences in a disassembler, and the hello-ipl3font example still works with this target configuration. I did not attempt to link it with nusys.

Strange that there is no discernible difference. Even the o32 binaries pass more than 4 arguments as registers. I've tested up to 10 args, and all are passed as registers (a0-a3 and t0-t5) which is beyond the n32 and n64 ABI capabilities. So I'm kind of at a loss, here.

[command-tab]

Cool, thanks! I'll check it out 👀

[moparisthebest]

I ran into this same problem trying to write some rust functions to pull into an existing C project using libdragon, I used the patched mips-nintendo64-none.json it compiles and I can link it, and it seems to run, but only until I try to do real work with it.

I can pass parameters like size_t/usize back and forth, but not pointers like *mut u8, so I created functions sending pointers on both sides and disassembled it, and it looks wrong to me:

void screen_text(u8 *msg) {
    printText(msg, 3, -1, global_disp_ctx);
    display_show(global_disp_ctx);
    sleep(1000);
}

void call_test(u8 *msg) {
    screen_text(msg);
}
/*
00006614 <call_test>:
    6614:       27bdffd8        addiu   sp,sp,-40
    6618:       ffbf0020        sd      ra,32(sp)
    661c:       0c000000        jal     0 <ttULONG>
    6620:       00000000        nop
    6624:       dfbf0020        ld      ra,32(sp)
    6628:       03e00008        jr      ra
    662c:       27bd0028        addiu   sp,sp,40
*/

extern "C" {
    fn screen_text(msg: *const u8);
}

#[no_mangle]
pub extern "C" fn rust_call_test(msg: *const u8) {
    unsafe { screen_text(msg); }
}
/*
00000000 <rust_call_test>:
   0:   67bdfff0        daddiu  sp,sp,-16
   4:   ffbf0008        sd      ra,8(sp)
   8:   0c000000        jal     0 <rust_call_test>
   c:   00000000        nop
  10:   dfbf0008        ld      ra,8(sp)
  14:   03e00008        jr      ra
  18:   67bd0010        daddiu  sp,sp,16
*/

No matter how I change mips-nintendo64-none.json, rust still calls this (seemingly wrong) way. I tried building myself a 32-bit C toolchain and using the original mips-nintendo64-none.json from this project where the calling convention matches, but libdragon won't build successfully that way.

At this point I'm at the end of my rope here trying to figure this out and hoping someone else will be able to help. Thanks for any pointers (pun intended) you can give me.

[parasyte]

The disassembly for both C and Rust functions looks fine. C just allocates more stack space (unnecessarily) but they are otherwise identical. Part of the problem here is that you are blindly passing a pointer argument from the caller, so these don't show how the address is produced. Compiling something like these examples might be more useful:

void call_test() {
    screen_text("Hello, world!");
}

extern "C" {
    fn screen_text(msg: *const u8);
}

#[no_mangle]
pub extern "C" fn rust_call_test() {
    let msg = "Hello, world!\0".as_bytes().as_ptr();
    unsafe { screen_text(msg); }
}

I tried to get the godbolt compiler to output something resembling the custom target. But I don't think rustc codegen options has everything needed for all cases. This code is simple enough that it should be very close: https://godbolt.org/z/nWvfMPYY9

rust_call_test:
        addiu   $sp, $sp, -24
        sw      $ra, 20($sp)
        lui     $1, %hi($__unnamed_1)
        jal     screen_text
        addiu   $4, $1, %lo($__unnamed_1)
        lw      $ra, 20($sp)
        jr      $ra
        addiu   $sp, $sp, 24

$__unnamed_1:
        .asciz  "Hello, world!"

No surprises here, $4 (aka $a0) is set to a 32-bit pointer for the function call.

[parasyte]

00000000 <rust_call_test>:
   0:   67bdfff0        daddiu  sp,sp,-16
   4:   ffbf0008        sd      ra,8(sp)
   8:   0c000000        jal     0 <rust_call_test>
   c:   00000000        nop
  10:   dfbf0008        ld      ra,8(sp)
  14:   03e00008        jr      ra
  18:   67bd0010        daddiu  sp,sp,16

The use of 64-bit instruction is interesting. I think the sd/ld pair must be from "target-pointer-width": "64", and the daddiu might be from the change to the data-layout. Both settings that I can't seem to change on godbolt.

[moparisthebest]

Compiled the new examples:

void call_test() {
    screen_text("Hello, world!");
}
/*
00006660 <call_test>:
    6660:       27bdffd8        addiu   sp,sp,-40
    6664:       ffbf0020        sd      ra,32(sp)
    6668:       3c040000        lui     a0,0x0
    666c:       0c000000        jal     0 <ttULONG>
    6670:       24840000        addiu   a0,a0,0
    6674:       dfbf0020        ld      ra,32(sp)
    6678:       03e00008        jr      ra
    667c:       27bd0028        addiu   sp,sp,40
*/

#[no_mangle]
pub extern "C" fn rust_call_test() {
    let msg = "Hello, world!\0".as_bytes().as_ptr();
    unsafe { screen_text(msg); }
}
/*
00000000 <rust_call_test>:
   0:   67bdfff0        daddiu  sp,sp,-16
   4:   ffbf0008        sd      ra,8(sp)
   8:   3c010000        lui     at,0x0
   c:   64210000        daddiu  at,at,0
  10:   00010c38        dsll    at,at,0x10
  14:   64210000        daddiu  at,at,0
  18:   00010c38        dsll    at,at,0x10
  1c:   0c000000        jal     0 <rust_call_test>
  20:   64240000        daddiu  a0,at,0
  24:   dfbf0008        ld      ra,8(sp)
  28:   03e00008        jr      ra
  2c:   67bd0010        daddiu  sp,sp,16
*/

I tried changing data-layout a bit but rust still wants to compile to these daddiu instead of addiu, well, when I didn't panic rustc :)