Wix Custom Action blocked by Smart App Control

[vichuviswan]

I have an installer created with Wix 3.11 having Custom Action logic (using CustomAction attribute) and everything is working fine on Windows 10 and Windows 11, but when the Smart App Control (SAC) is turned ON, I see the installation fails and there is a popup from SAC complaining about my CustomAction dll.

I verified that all my binaries (including the custom action files) are signed. I noticed that the Windows Installer file from Wix has an expired certificate, and assuming that might be the problem, I upgraded to Wix 4.0.1 (latest version), still, the custom action dll gets flagged and the installer fails.

So wanted to check if anyone has come across an issue like this and might know what I might be missing here. Any input/direction to resolve this issue is really appreciated.

Thanks in advance!!!

[BMurri]

Raymond Chen has mentioned that rundll.exe is intended for use by only Windows-created binaries, so I assume that they are looking for specific certificates used when building Windows itself.

There have been instances of malware using rundll32 to elude anti-virus, so that's the likely reason for the SAC scrutiny.

[vichuviswan]

I'm not explicitly using rundll32. I believe it might have been used by some internal process to invoke these custom actions. One more thing I noticed is that my custom action has about 3 individual dlls, and I see all of them are getting packaged into the CA.dll. Even though all the individual files are signed, the files that get archived into the CA.dll are not signed (I assume it's because that file gets generated automatically during the compile time). And I suspect that might be the issue for SAC flagging the custom action. However, I'm not sure how I could make it wait for the signing before it creates the CA.dll. Any thoughts?

[BMurri]

The managed custom action wrapper is using rundll32 (there's a native-stub that has every entry point created, each of which calls a routine that extracts the managed assembly (which is stored in the native DLL as a payload) and calls rundll32.exe to run the custom action contained in the managed assembly. So, it's WiX that is explicitly using rundll32.exe (but only with managed CAs).

[BMurri]

Speaking of which, did you make sure that your managed CA assembly was signed before the build step that wraps it ran?