Prevent FFI from deallocating data before returning it

[magistau]

This effectively makes &ffi leak all allocated memory. It should probably be possible to only leak the necessary data, but the existing FFI code is a bit too convoluted to comprehend.

[magistau]

For completeness, here's a helpful GDB script:

break src/ffi.rs:630
  commands
    disable $_hit_bpnum
  end
set $drop_break = $bpnum
disable $drop_break
tcatch load ffi_lib
  commands
    silent
    break dummy_md5 if out != 0
      commands
        set $msg=&(*m@len)
        set $out=&(*out@len)
        printf "Updated $msg and $out\n"
        enable $drop_break
        display
      end
    continue
  end
display $out

Run with rust-gdb -x ffi-brp.gdb --args target/debug/uiua run tests_ffi/test.ua target/debug/libffi_lib.so after building both uiua and tests_ffi

[alex-s168]

I think you forgot to remove Debug after debugging

[magistau]

Well, it doesn't hurt to have a derived Debug I think? I mean, I can remove it, but I don't see a problem either way.

[Omnikar]

Yeah I don't see a reason not to derive Debug if it's possible to.

[alex-s168]

resolve this todo

[magistau]

I didn't change how this code behaves, only rewrote it to make the intent clearer. I don't know if this code is sound because I didn't write the original code.

[alex-s168]

resolve this TODO too

[magistau]

I think this is impossible without input from Kai — I have no idea about why a null pointer is used there, and I'm not sure where the code dependent on this could be.

[Omnikar]

I definitely think this PR (and #742) need input from Kai before we can think about merging them. I personally very much don't know enough about what's going on in this PR to be able to comment on it at the moment unfortunately.

[kaikalii]

How does this fix tests?

[magistau]

1. On macOS, a different file extension is used for dynamic libraries
2. &ffi doesn't deallocate data before returning it anymore. Tests were failing because &ffi implementation actually contained a bug causing UB — namely, an output buffer would be deallocated before &ffi even returns anything.

[kaikalii]

This should just be fixed in &ffi then. Not by making it always leak.

[kaikalii]

What's the test case that was causing issues?

[magistau]

This should just be fixed in &ffi then. Not by making it always leak.

Well, the better solution would be leaking just the needed data, but I do not understand Uiua code well enough for that. As noted in the PR description:

It should probably be possible to only leak the necessary data, but the existing FFI code is a bit too convoluted to comprehend.

To be honest, after spending >24h studying the code, running gdb over and over, digging into libffi and others, I still have little idea about what exactly is FfiBindings for, especially the other_data field. I rewrote a part of it to be as clear as possible, and there are two things I fail to understand in that part (see the TODOs). There's also some value.meta.pointer stuff and five macro declarations in a single 400SLoC do_ffi function, which is too intimidating to understand.

[magistau]

What's the test case that was causing issues?

Seemingly the only line testing output buffers:

uiua/tests_ffi/test.ua

Line 41 in 8a9de2d

⍤⟜≍: {.utf₈"hello"} ⍚(&memcpy "unsigned char":5) MDF {"hello" ▽5 0}

[kaikalii]

FfiBindings stores data that is being passed to foreign functions. libffi requires you to pass the data as pointers, and that data may be pointers, so it has to be allocated on the Rust side somewhere. Those allocations are meant to last until the foreign function call is complete.

[magistau]

Those allocations are meant to last until the foreign function call is complete.

Well, this is not the cause for output buffers, since those are supposed to live after &ffi is done.

[kaikalii]

Would this make you have to use &free all the time?

[magistau]

Well, it's not like you have to, but yeah, that's what happens.

[Marcos-cat]

This is solved by out parameter pointers in #783