What is the difference between the MyFile.dll and MyFile.CA.dll generated when building a Custom Action from the WiX template?

[MattWalker414]

I've had some issues pointed out to me that Windows Defender is flagging some binaries related to Custom Actions during our installation, authored using InstallShield, but some MSI dll Custom Actions are written using the WiX template (C#).

I've decided to try to sign all of my Custom Action executables, but when I build the MSI dll CA, I get two .dlls in the output folder...

MyFileCheck.dll
MyFileCheck.CA.dll (smaller in size)

When I compile the dll in Visual Studio, only the MyFile.dll gets signed. Since in InstallShield, the Custom Action is configured by pointing to MyFile.CA.dll, I tried to sign that binary as well, but that presents errors...

EXEC : SignTool error : SignedCode::Sign returned error: 0x800700C1
1>
1> Either the file being signed or one of the DLL specified by /j switch is not a valid Win32 application.
1>
1>EXEC : SignTool error : An error occurred while attempting to sign: C:\Install Source\MyApp\Custom Actions\MyFileCheck\MyFileCheck\bin\Release\MyFileCheck.CA.dll

Again, with using InstallShield, which dll is used from/included in the compiled installation pacakge?

The command im using to sign is....
signtool sign /f my.pfx /p mypswd $(TargetDir)$(TargetFileName)

[pas059]

Hi,
as i know the xxx.dll file contains CLR code, and the xxx.CA.dll contains binary.
regards

[MattWalker414]

Thanks pas059! I think I tried setting the InstallShield custom action to use the xxx.dll and not the xxx.CA.dll and that errored. I believe at runtime. I'm still not sure I understand and if the signing of the xxx.dll will suffice for Defender if the install actually uses the xxx.CA.dll.

[chrpai]

Here's a blog article I wrote 15 years ago when DTF first came out. It's very comprehensive. The only part to ignore is running a postbuild step to call sfxca. This was added to the .wixproj soon after.

http://blog.deploymentengineering.com/2008/05/deployment-tools-foundation-dtf-custom.html

Build an InstallShield Basic MSI with an InstallScript custom action. Export the ISSetup.dll from the Binary table and look at it in Depends. You'll see it has a bunch of exported functions caled F1 F2 F3 and so on. If you look at the CustomAction table you'll see how they map. Now do the same with a WiX DTF CA. Compare that and what my blog post explains and it should all make sense.

[chrpai]

For signing, $(TargetFileName) will get you the .dll. You'll want to hard the DLL.CA.DLL or use another MSBuild property. You probably also want to specify a timestamp server and the digest alogs for signing and timestamp signing.

[chrpai]

While I haven't read every word of this thread, I suspect your over thinking this. On your machine take a look at these files
C:\Program Files (x86)\WiX Toolset v3.11\SDK\x86\sfxca.dll
C:\Program Files (x86)\WiX Toolset v3.11\SDK\x64\sfxca.dll

You will see they are signed. You will see the signature times are a little bit different. Note their file size. When you build your custom action project, you should see the signature still there but invalid because the file has grown with your custom action code.

The rub is these files should have never been signed. If you use delcert on these two files your problem should go away. The output files will not be signed and then signtool will work.

Or just call delcert before calling signtool.

Or just use wix 3.14 or wix 4.0.

Do you need to recompile delcert? I don't know... I've never used it. Why not just compile an x64 and x86 custom action and give it a try. See if it throws an error and look at the file properties to see if it worked.

[MattWalker414]

I guess that is basically my question, do I need to recompile delcert as 64 bit to remove the signature from the 64 bit template dll? I can just give that a try with the x86 delcert.

Initially, I had inadvertently tried to recompile delcert as 64 bit and received a host of errors.

[chrpai]

Maybe you should ask @MadhukarMoogala.

[jsrustad]

I don't think you need to recompile delcert as 64-bit. Try it and see.

Since I gave you the answer, please accept my answer.

[MattWalker414]

Yes, I was definitely over thinking it!

[jsrustad]

Which version of WiX are you using?

WiX 3.11.2 had a problem where the sfxca.dll template was signed. When your MyFile.dll was added to this template to make MyFile.CA.dll, the signature was invalidated. Microsoft's signtool won't sign a file that has an invalid signature. Signtool also won't remove an invalid signature. I only know this because I was bitten by this myself.
To work around this problem, use delcert to remove the signature from the template file.
The bug report for WiX - #6089
Delcert is available at - https://github.com/MadhukarMoogala/delcert

[MattWalker414]

At some point I thought someone said that compiling through Visual Studio and signing using a post build script caused an issue. The encapsulated .dll would not be signed post compile as its already encapsulated in the CA.dll, which does get signed. I guess that might trip defender. I thought all of that information was here, as someone spelled out all of the post build steps to get it right, but it must be in a separate thread. I can't for the life of me find it.

[jsrustad]

You are correct that the embedded DLL (MyFile.dll) doesn't get signed. I don't think that will be a problem because the outer DLL (MyFile.CA.dll) is signed. The only way to tell for sure is to do some testing. I can say that I have not seen a problem with the embedded DLL not being signed.

[MattWalker414]

I believe whoever provided the steps to recompile to have both .dlls signed did so because they say issues with threat detection. I really wish I could find that posting. I was certain that info was here, but I'll keep digging.

[MattWalker414]

Found it! https://community.flexera.com/t5/InstallShield-Forum/Custom-Action-Utilities-Flagged-By-Windows-Defender/m-p/295625

[jsrustad]

Interesting. I will squirrel this away in case I need it.

[chrpai]

Thanks, I wasn't aware of that bug. I had moved onto the dev build 3.14 years ago due to other important fixes to me. I really wish a final 3.x would have shipped. It's nice to have an old stable with all known bug fixes and a v.next to choose from.

https://wixtoolset.org/docs/v3/releases/v3-14-0-6526/