Crash under winevdm

[X547]

Compiled executables crashes in winevdm v0.7.0 (https://github.com/otya128/winevdm).

winevdm error message:

Limit check at 0x038c5f20 failed. Segment 1207, limit 0000211f, offset 00002120

I am not sure is it this program or winevdm problem.

[alexfru]

hw.exe built with one of these versions of NASM:

NASM version 2.10 compiled on Mar 12 2012
NASM version 2.14.02 compiled on Dec 26 2018
NASM version 2.15.03rc7 compiled on Jul 15 2020

works on 32-bit Windows XP and 32-bit Windows 7.

I cannot guarantee that the program is 100% NE-proper. But it runs on real Windows.

If you find anything specific that's wrong with hw.exe, please share.

[alexfru]

I have one hypothesis...

hw.exe defines the stack size of 8192 bytes and the data segment of size 288 bytes.

8192+288=8480=0x2120.

So, it might be that the data and the stack end up occupying the same segment (I vaguely remember it being a possible option for an NE executable) and SP is set to point to the byte immediately beyond the last byte of this common segment, that is, the initial SP is set to 0x2120.

If that is the case, it should be perfectly fine because the stack grows from larger addresses towards smaller ones. That is, let's say there's the PUSH AX instruction executed on this stack. It would not write AX to offset 0x2120 or beyond. It would first decrement SP by 2 and then write AX to the bytes at offsets 0x211e and 0x211f. The segment limit (0x211f) tells us the last usable offset or one less the segment size. So, it should be OK.

IOW, it's possible that winevdm incorrectly checks the initial SP against the segment size/limit.
Can you check that?

[X547]

I set SP in NE header by

    at __NEHDR.InitSp,                  dw 0x2110

and it started working. I don't know why it happens. Is it a bug in CPU emulator or something else? Clock.exe from Windows 3.1 also use 02:0000 initial stack in NE header, but it runs fine.

Winevdm task startup code: https://github.com/otya128/winevdm/blob/master/krnl386/ne_module.c#L1490.

[alexfru]

NE EXE format specifications:
http://bytepointer.com/resources/win16_ne_exe_format_win3.0.htm
http://bytepointer.com/resources/win16_ne_exe_format_win3.1.htm

This is precisely what I do in hw.exe:

18h     DD  Segment number:offset of SS:SP.
            If SS equals the automatic data segment and SP equals
            zero, the stack pointer is set to the top of the
            automatic data segment just below the additional heap
            area.

                +--------------------------+
                | additional dynamic heap  |
                +--------------------------+ <- SP
                |    additional stack      |
                +--------------------------+
                | loaded auto data segment |
                +--------------------------+ <- DS, SS

I would think it's some problem in winevdm, not sure where yet...

Here's what I find strange, though:

DWORD NE_StartTask(void)
{
    ...
        /* Use DGROUP for 16-bit stack */

        if (!(sp = OFFSETOF(pModule->ne_sssp)))
            sp = pSegTable[SELECTOROF(pModule->ne_sssp)-1].minsize + pModule->ne_stack;
        sp &= ~1;
        sp += 4;
        setWOW32Reserved((void *)MAKESEGPTR(GlobalHandleToSel16(hInstance), sp));
    ...
}

That increment by 4 followed by setWOW32Reserved()... I wonder if that causes SP to be 4 bytes beyond the end of the segment and then there's either:

• a far call made, which tries to use those 4 inaccessible bytes to store a far return address
• a far address is written into those 4 inaccessible bytes by winevdm, SP is lowered by 4 and then a far return is attempted, which fails to read those 4 bytes

Take a look at K32WOWCallback16Ex() in https://github.com/otya128/winevdm/blob/master/krnl386/wowthunk.c, there's some interesting stuff, looking like one of those two possibilities I'm writing above:

BOOL WINAPI K32WOWCallback16Ex( DWORD vpfn16, DWORD dwFlags,
                                DWORD cbArgs, LPVOID pArgs, LPDWORD pdwRetCode )
{
    ...
            DPRINTF(") ss:sp=%04x:%04x",
                    SELECTOROF(getWOW32Reserved()), OFFSETOF(getWOW32Reserved()) );
    ...
            /* push return address */
    ...
                stack -= sizeof(SEGPTR);
                *((SEGPTR *)stack) = call16_ret_addr;
                cbArgs += sizeof(SEGPTR);
    ...
            DPRINTF(") ss:sp=%04x:%04x\n",
                    SELECTOROF(getWOW32Reserved()), OFFSETOF(getWOW32Reserved()) );
    ...
        /* push return address */
        stack -= sizeof(SEGPTR);
        *((SEGPTR *)stack) = call16_ret_addr;
        cbArgs += sizeof(SEGPTR);
    ...
}

I don't know what happens after this in winevdm, I'm not familiar with intimate details of Win16 or wine.