Wrapper can't find the library symbol

[ghyatzo]

Hello, I'm trying to pick up the torch for an IDL-Julia interop.
I'm using Clang to wrap the API header (it's a macro madness, and yet the codegen is almost flawless, kudos)
In the generator.toml I've specified the library name to be libidl but now I am a bit uncertain how I can define the library symbol such that the wrapper knows what libidl is.

Unfortunately I can't really build IDL and I can't seem to wrap my head around _jlls.
For now I've opted for a local lookup of the idl dynamic library which then I save in a const.
So maybe a bit naively I have the following structure:

module IDL

include("LibIDL.jl")  # generated by Clang.jl
using .LibIDL

libidl = # ... lookup the lib in the system ... # 

Libdl.dlopen(libidl) # this call is successful

LibIDL.IDL_Init(#... args ... #) # this fails since it doesn't (rightfully) find a libidl symbol in LibIDL
end

Looking at another package like CimGui.jl I see that the libcimgui symbol arrives from the _jll package but I am not really sure how the internals of _JLLs work.

How can I let LibIDL know the value of libidl?
Should I simply do the local lookup inside the LibIDL module directly? I didn't really want to touch too much the generated output.

EDIT: Also I have a bit of a side question: How does one initialize datastructures that have been wrapped as a

struct wrappedStruct
    data::NTuple{16, Uint8}
end

[JamesWrigley]

What exactly are libidl and LibIDL.libidl set to? I think you only need to dlopen() the full path and then you should be able to @ccall by the library name. For example:

julia> Libc.Libdl.dlopen("build/libcimgui") # Path to a local build of libcimgui.so (excluding the extension)
Ptr{Nothing} @0x00000000028db830

julia> libcimgui = "libcimgui"
"libcimgui"

julia> unsafe_string(@ccall libcimgui.igGetVersion()::Cstring)
"1.91.5"

But @ccall'ing by the full path works for me too, I guess Julia will automatically dlopen() it the first time:

julia> unsafe_string(@ccall "build/libcimgui".igGetVersion()::Cstring)
"1.91.5"

EDIT: Also I have a bit of a side question: How does one initialize datastructures that have been wrapped as a

struct wrappedStruct
    data::NTuple{16, Uint8}
end

I'm not sure you can initialize those from Julia, typically the idea is that one would call the equivalent of@ccall create_wrappedStruct()::Ptr{wrappedStruct} and get back a pointer to the struct that you could interact with. If a struct only has a single data::NTuple field it's likely because it has union or bitfield members, which can't easily be represented in a regular Julia struct (see: https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/#Struct-Type-Correspondences).

[ghyatzo]

Hi,

libidl is just the absolute path to the library obtained from where idl or which idl commands together with some path manipulations, and I assume that one has it in the $PATH.

Your example does indeed work but I am not @ccalling the lib directly, instead I have all the wrapper functions generated by Clang.jl inside a LibIDL module like:

module LibIDL
function IDL_Cleanup(just_cleanup)
    # this libidl was unknown to this function since it is defined in the parent module really
    ccall((:IDL_Cleanup, libidl), Cint, (Cint,), just_cleanup) 
end
end

Now, I've manged to work around this, by making Clang NOT generate a module but a plain file lib_idl.jl
which then I've wrapped in the original LibIDL.jl like so:

module LibIDL

libidl = #= ... code to get the library path ... =# 

include("lib_idl.jl") # now the `libidl` symbol is available to the wrappers since they are no longer in a module.

#= ... exporting all symbols ... =#

end

But I am not sure I like this approach, Is there a better way?

Regarding the construction of NTuple opaque structures, I've found an interesting closed issue #304 with matching merged PR #305.
I enabled the add_record_constructors = true option in the generator.toml file but It didn't generate any constructor, so I've written the one I needed manually:

function IDL_INIT_DATA(init_options::IDL_INIT_DATA_OPTIONS_T)
    r = Ref{IDL_INIT_DATA}()
    x = Base.unsafe_convert(Ptr{IDL_INIT_DATA}, r)
    x.options = init_options # the magic is here with the getproperty/setproperty indirections
    GC.@preserve r unsafe_load(x)
end

# where
struct IDL_INIT_DATA
  data::NTuple{56, UInt8}
end

Still not exactly sure if I did something wrong and it didn't generate the constructors even though the logic should be there, or if Clang choked somewhere.

Anyhow, this seems to be working 🥳

init_data = IDL_INIT_DATA(init_options)
res = IDL_Initialize(Ref(init_data))
@assert res == 1

[JamesWrigley]

Sweet 🎉 I think the way you're doing it is fine, that's also what CImGui.jl does (except with the JLL).

Regarding the construction of NTuple opaque structures, I've found an interesting closed issue #304 with matching merged PR #305. I enabled the add_record_constructors = true option in the generator.toml file but It didn't generate any constructor, so I've written the one I needed manually:

Ah cool, I didn't realize we had that. Dumb question, but did you put add_record_constructors = true under the codegen option like in this example from the tests? https://github.com/JuliaInterop/Clang.jl/blob/ce61d3e177a0550b57a3f9f723fa4579d532fd52/test/bitfield/generate.toml

At least for that test case the constructor seems to be generated.

[ghyatzo]

The question is not dumb at all, I am.
Yes, By putting it in the codegen section it works. Oh well!

Sadly though, this god forsaken library uses too many nested unions of unnamed structures, so I have to deal with var"unnamed structure with comically long name and I even tell you where it is in the code" kind of things. So some constructors I'll have to write manually anyway.

Unless there is a way to avoid that?

[JamesWrigley]

I haven't tried them myself, but maybe a combination of use_deterministic_symbol and smart_de_anonymize?

[ghyatzo]

Thanks! I'll give it a try.