64 Bit Custom Actions

[StephenHodgson]

I have a custom action I wrote using a Wix C# library. The problem is I have a 64bit dll I need to load during the installation process but the custom action server is 32bit.

I'm using dotnet build -c Release and only have x64 build configurations

My c# Custom Action class is pretty simple:

    public static class RegisterDllAction
    {
        [CustomAction]
        public static ActionResult RegisterDll(Session session)
        {
            try
            {
                return InvokeDll(session.CustomActionData["DLL_PATH"], true, session) ? ActionResult.Success : ActionResult.Failure;
            }
            catch (Exception ex)
            {
                session.Log($"Error: {ex.Message}");
                return ActionResult.Failure;
            }
        }

        [CustomAction]
        public static ActionResult UnRegisterDll(Session session)
        {
            try
            {
                return InvokeDll(session.CustomActionData["DLL_PATH"], false, session) ? ActionResult.Success : ActionResult.Failure;
            }
            catch (Exception ex)
            {
                session.Log($"Error: {ex.Message}");
                return ActionResult.Failure;
            }
        }

        private static bool InvokeDll(string dllPath, bool register, Session session)
        {
            try
            {
                if (!Environment.Is64BitProcess)
                {
                    throw new NotSupportedException("This action can only be run in a 64-bit process");
                }

                session.Log($"{nameof(InvokeDll)} {(register ? "register" : "unregister")} on DLL at path: {dllPath}");

                if (!System.IO.File.Exists(dllPath))
                {
                    session.Log($"Error: DLL not found at path: {dllPath}");
                    return false;
                }
                else
                {
                    session.Log($"DLL found at path: {dllPath}");
                }

                var module = LoadLibrary(dllPath);

                if (module == IntPtr.Zero)
                {
                    int errorCode = Marshal.GetLastWin32Error();
                    session.Log($"Error: LoadLibrary failed with error code {errorCode}");
                    return false;
                }

                var procAddress = register
                    ? GetProcAddress(module, "DllRegisterServer")
                    : GetProcAddress(module, "DllUnregisterServer");

                if (procAddress == IntPtr.Zero)
                {
                    session.Log("Error: can't find function in DLL");
                    return false;
                }

                var func = (DllRegisterServerDelegate)Marshal.GetDelegateForFunctionPointer(procAddress, typeof(DllRegisterServerDelegate));
                var hr = func();

                if (hr != 0)
                {
                    session.Log($"Error: operation failed (0x{hr:X8})");
                    return false;
                }

                session.Log($"Hyper Virtual Webcam has been successfully {(register ? "registered" : "unregistered")} to the system");
                return true;
            }
            catch (Exception ex)
            {
                session.Log($"Error: {ex.Message}");
                return false;
            }
        }

        [DllImport("kernel32", CharSet = CharSet.Ansi, ExactSpelling = true, SetLastError = true)]
        private static extern IntPtr GetProcAddress(IntPtr hModule, string procName);

        [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
        private static extern IntPtr LoadLibrary(string lpFileName);

        private delegate int DllRegisterServerDelegate();
    }

here's my installation log:

MSI (s) (FC:08) [14:25:40:622]: Executing op: ActionStart(Name=RegisterDllAction,,)
MSI (s) (FC:08) [14:25:40:633]: Executing op: CustomActionSchedule(Action=RegisterDll,ActionType=3073,Source=BinaryData,Target=RegisterDll,CustomActionData=DLL_PATH=C:\path/to/my/library.dll)
MSI (s) (FC:98) [14:25:40:634]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSIE0BD.tmp, Entrypoint: RegisterDll
MSI (s) (FC:90) [14:22:55:855]: Created Custom Action Server with PID 61332 (0xEF94).
MSI (s) (FC:34) [14:22:55:875]: Running as a service.
MSI (s) (FC:34) [14:22:55:877]: Hello, I'm your 32bit Elevated Non-remapped custom action server.
SFXCA: Extracting custom action to temporary directory: C:\WINDOWS\Installer\MSI5D0D.tmp-\
SFXCA: Binding to CLR version v4.0.30319
Calling custom action...
Error: This action can only be run in a 64-bit process

[StephenHodgson]

Figured out my problem. My solution build configurations were somehow corrupted. I had to remove Any CPU and x86 for both Release and Debug