Using DNNE to generate proxy libraries

[DaZombieKiller]

I am interested in experimenting with using DNNE to generate proxy libraries in C# using source generators, but I believe there is additional functionality necessary in DNNE to do so. I'm intending for this issue to serve as a discussion into how this functionality could be exposed (or if it's even considered to be within scope for the project.)

As a simple example: imagine that we want to log API calls to a COM library, which has the following exports:

DllGetClassObject
DllCanUnloadNow

A proxy library for this can be implemented today with code similar to the example below:

// The Get* methods will perform NativeLibrary.Load/GetExport to retrieve the exports from the real library.
static readonly delegate* unmanaged<Guid*, Guid*, void**, uint> DllGetClassObject = GetDllGetClassObject();
static readonly delegate* unmanaged<uint> DllCanUnloadNow = GetDllCanUnloadNow();

[UnmanagedCallersOnly(EntryPoint = "DllGetClassObject")]
static uint ProxyDllGetClassObject(Guid* rclsid, Guid* riid, void** ppv)
{
    // Perform some logging
    return DllGetClassObject(rclsid, riid, ppv);
}

[UnmanagedCallersOnly(EntryPoint = "DllCanUnloadNow")]
static uint ProxyDllCanUnloadNow()
{
    // Perform some logging
    return DllCanUnloadNow();
}

The signatures for these functions are well-known, but for the sake of the example let's imagine we either don't know what they are, or that they aren't relevant for what we want to do (in this example, logging API calls.)

Let's say that we only want to log calls to DllGetClassObject, and we don't care about DllCanUnloadNow (or the potentially many other exports in the library). We would need to know the signatures of all of these exports in order to produce a proxy library with the above technique.

The solution typically employed for this is for the body of the proxy method to simply jmp to the address of the real method, but as far as I'm aware there is currently no way to replicate this technique from DNNE.

[AaronRobinsonMSFT]

@DaZombieKiller Thanks for starting this conversation.

We would need to know the signatures of all of these exports in order to produce a proxy library with the above technique.

That is true. I don't see any way around that though. The biggest reason is the heavy impact of also transitioning from unmanaged to managed code. If a simple trampoline was all that was happening the transition would likely become far more costly to jump into managed code only to immediately jump to unmanaged code. I don't see a huge benefit in that.

there is currently no way to replicate this technique from DNNE.

From managed code, I think that is true. However, I just added support for supplying your own .c file during the native entry compilation process. Perhaps that could be used in some manner. See #99.

/cc @jkotas @jkoritzinsky @elinor-fung

[DaZombieKiller]

From managed code, I think that is true. However, I just added support for supplying your own .c file during the native entry compilation process. Perhaps that could be used in some manner. See #99.

This is certainly an option for 32-bit, though on 64-bit it would be problematic when MSVC is the compiler being used (since it doesn't allow inline assembly). Maybe an additional feature to support .asm files could work?

[AaronRobinsonMSFT]

Maybe an additional feature to support .asm files could work?

I'm not sure that is the best way. The issue here is now DNNE would need to trigger an assembler – which I am loathe to do. Instead, I think supplying a .obj to link in would be preferred in this case. That should already be possible using

DNNE/src/msbuild/DNNE.props

Lines 60 to 62 in 52edb71

	<!-- Set to add linker flags.
	This value will be placed after all other flags passed to the compiler. -->
	<DnneLinkerUserFlags></DnneLinkerUserFlags>