Handling non-interactive sessions in a custom BA for a v4 installer

[csdahlberg]

We use a custom C# BA for our v3-based installers, and are migrating them to v4. Upgrades are often installed by a background Windows service, which is not in an interactive session and does not have any windows. With v3, passingIntPtr.Zero to IEngine.Apply(IntPtr hwndParent) worked just fine, but v4 no longer allows that (possibly to avoid hangs), and instead fails with a Error 0x80070057: BA passed NULL hwndParent to Apply. error.

Some stripped-down example code:

public class SetupViewModel
{
    private readonly CustomBootstrapper _bootstrapper;

    public SetupViewModel(CustomBootstrapper bootstrapper)
    {
        _bootstrapper = bootstrapper;
        _bootstrapper.PlanComplete += OnPlanComplete;
    }

    private void OnPlanComplete(object sender, PlanCompleteEventArgs e)
    {
        if (e.Status >= 0)
        {
            _bootstrapper.Engine.Apply(Process.GetCurrentProcess().MainWindowHandle);
        }
    }
}

Is running installers in this sort of non-UI scenario something that should be expected to work?

[barnson]

Sure. Windows are required for a lot of what Burn is doing under the hood and NULL window handles aren't legal for some of them. But a hidden window is still a window.

[csdahlberg]

Thanks for the quick confirmation -- I got it working.

For anyone else who might encounter this scenario, some details:

• The IEngine.Apply(IntPtr hwndParent) method in v4 requires a valid, non-zero window handle.
• Creating a new dummy window using something like new Window() was throwing an exception due to the thread running our C# BA code not being an STA thread.
• Spinning up a new STA thread to create the dummy window led to successfully creating the window, but ultimately also did not work, seemingly because the thread that created the window was different than the thread that later used the window.
• Using P/Invoke to create the dummy window worked just fine. Example code:

public const int CS_USEDEFAULT = unchecked((int)0x80000000);

public delegate IntPtr WndProc(IntPtr hWnd, uint msg, IntPtr wParam, IntPtr lParam);

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
public struct WNDCLASSEX
{
    [MarshalAs(UnmanagedType.U4)]
    public int cbSize;
    [MarshalAs(UnmanagedType.U4)]
    public int style;
    public IntPtr lpfnWndProc;
    public int cbClsExtra;
    public int cbWndExtra;
    public IntPtr hInstance;
    public IntPtr hIcon;
    public IntPtr hCursor;
    public IntPtr hbrBackground;
    public string? lpszMenuName;
    public string lpszClassName;
    public IntPtr hIconSm;
}

[DllImport("user32.dll", SetLastError = true, EntryPoint = "RegisterClassEx")]
public static extern ushort RegisterClassEx([In] ref WNDCLASSEX lpWndClass);

[DllImport("user32.dll")]
public static extern IntPtr DefWindowProc(IntPtr hWnd, uint uMsg, IntPtr wParam, IntPtr lParam);

[DllImport("user32.dll", SetLastError = true, EntryPoint = "CreateWindowEx")]
public static extern IntPtr CreateWindowEx(int dwExStyle, ushort regResult, string lpWindowName, uint dwStyle,
    int x, int y, int nWidth, int nHeight, IntPtr hWndParent, IntPtr hMenu, IntPtr hInstance, IntPtr lpParam);

public IntPtr CreateDummyWindow()
{
    var wndClass = new WNDCLASSEX
    {
        cbSize = Marshal.SizeOf(typeof(WNDCLASSEX)),
        hInstance = Marshal.GetHINSTANCE(GetType().Module),
        lpszClassName = "DummyWindow",
        lpfnWndProc = Marshal.GetFunctionPointerForDelegate((WndProc)DefWindowProc),
    };

    ushort regResult = RegisterClassEx(ref wndClass);
    return CreateWindowEx(0, regResult, "Setup Window", 0, CS_USEDEFAULT, CS_USEDEFAULT, CS_USEDEFAULT, CS_USEDEFAULT,
        IntPtr.Zero, IntPtr.Zero, wndClass.hInstance, IntPtr.Zero);
}