C# native dependencies not found in a packaged build

[tonetechnician]

Hey there!

Thanks so much for the work on this plugin. It has turned out to be super useful for an aspect of a project I'm working on.

I've been running into a hard to track error and was wondering if the author had any ideas on how to solve this, or has had this same issue happen to them.

I'm using this Confluent.SchemaRegistry.Serdes.Avro which relies on a native dependencies librdkafka.dll and librdkafkacpp.dll. When building my C# library, these are placed in the Managed\Build\runtimes\win-x64\native directory. PIE doesn't load these referenced binaries work unless I copy them to the same directory as the Binaries/Win64 folder.

When I build and package the project, my C# .dll and native libraries are created and placed correctly in the Managed directory but the native libraries are unable to be loaded by the UE executable and thus unable to be used by my C# library.

I've tried various places to add them:

1. Same directory as the UE binary
2. Same directory as the C# .dll (although this directory also has the Managed\Build\runtimes\win-x64\native link)
3. Same directory as the ref directory with a ref to the C# .dll.

I've also tried to load this explicitly into the Module.target using the .Build.cs file as follows:

        RuntimeDependencies.Add("$(TargetOutputDir)/librdkafka.dll", NugetKafkaModule + "/librdkafka.dll");
        RuntimeDependencies.Add("$(TargetOutputDir)/librdkafkacpp.dll", NugetKafkaModule + "/librdkafkacpp.dll");
        RuntimeDependencies.Add("$(TargetOutputDir)/zlib.dll", NugetKafkaModule + "/zlib.dll");
        RuntimeDependencies.Add("$(TargetOutputDir)/libzstd.dll", NugetKafkaModule + "/libzstd.dll");
        RuntimeDependencies.Add("$(TargetOutputDir)/msvcr120.dll", NugetKafkaModule + "/msvcr120.dll");
        RuntimeDependencies.Add("$(TargetOutputDir)/msvcp120.dll", NugetKafkaModule + "/msvcp120.dll");

The only way I've found to overcome this is to place the native dependencies in the System32 folder. Looking at resource below, it would seem that having the UnrealCLR plugin loaded at runtime means it's missing the first step of dynamic library linking, i.e "The directory from which the application loaded."

https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

Because our system is heavily automated and managed via process managers, it would definitely be more ideal to be able to load the native libraries by placing them alongside the UE executable. Or being able to explicitly load a reference to these native binaries. Is there a recommended approach to do this? Or maybe something I'm missing with how to structure the project?

Any thoughts or guidance would be extremely appreciated! Thank you!

[nxrighthere]

A while ago I was testing my NuGet package and it worked just fine (it also references native libraries). Give it a try and compare it to the package which you are using. Compare files in a published build, maybe you will spot something.

[nxrighthere]

I tried to debug this problem and it seems that the plugin which is used to dynamically load libraries doesn't copy native dependencies of librdkafka.dll to temporary storage for further loading. I'll report this problem to the maintainer, and let you know if we manage it.

[nxrighthere]

You can track the issue here natemcmaster/DotNetCorePlugins#213.

[tonetechnician]

Hey @nxrighthere

Thanks so much for digging into this a bit. Really appreciate that. Will be sure to follow the bug report. Great if this can be solved