Problems with Total Commander

[FeralChild64]

The driver seems to cause problems with Total Commander 6.58 (legal, registered version - not a cracked one) - from time to time TC complains about damaged binary and quits. Happens randomly, sometimes very quickly, sometimes only after a couple of minutes of playing with the application. Tried with Windows 3.11 for Workgroups, running either using QEMU 6.2.0 or my private DOSBox Staging branch with VMware mouse support (https://github.com/FeralChild64/dosbox-staging/tree/fc/vmware-mouse-2) - happens on both emulators.

It does not seem to be a virus - the Total Commander executable looks intact, fresh installation does not help. Reverting to original PS/2 driver fixes the problem.

y

[NattyNarwhal]

Ugh, I wonder if this is related to #22 - I have some ideas, but not sure (i.e. do we need to restore extended registers if we're mangling them from not 386 protected mode?)

[NattyNarwhal]

Stupid stupid idea: try this patch. I don't think it'll change much, but it'll guard changing extended registers from 16-bit world.

diff --git a/ps2.asm b/ps2.asm
index c8ef944..bf521bf 100644
--- a/ps2.asm
+++ b/ps2.asm
@@ -253,12 +253,12 @@ ps2_int proc    far
 	test    PS2_DATA_FLAG,0FFh
 	jz      ps2_int_exit            ;Not a valid PS/2 mouse interrupt
 
-	push    ax                      ;Save the world
-	push    bx
-	push    cx
-	push    dx
-	push    si
-	push    di
+	push    eax                      ;Save the world
+	push    ebx
+	push    ecx
+	push    edx
+	push    esi
+	push    edi
 	push    bp
 	push    ds
 	push    es
@@ -357,12 +357,12 @@ ps2_no_data:
 	pop     es
 	pop     ds
 	pop     bp
-	pop     di
-	pop     si
-	pop     dx
-	pop     cx
-	pop     bx
-	pop     ax
+	pop     edi
+	pop     esi
+	pop     edx
+	pop     ecx
+	pop     ebx
+	pop     eax
 
 ps2_int_exit:
 	pop ds
@@ -480,6 +480,10 @@ page
 ps2_search      proc    near
 
 	; Check for the VMware backdoor.
+	push	eax
+	push	ebx
+	push	ecx
+	push	edx
 	xor ebx, ebx
 	mov ecx, CMD_GETVERSION
 	call Backdoor
@@ -490,13 +494,18 @@ ps2_search      proc    near
 	; Under hypervisors, always assume the 286/386 PS/2 mouse vector
 	mov vector, 074h
 	stc                             ;Show mouse was found
-	ret
+	jmp	ps2_search_pop
 
 ps2_cant_use_it:
 	mov     vector,-1               ;Restore to "no mouse" value
 
 ps2_machine_not_found:
 	clc                             ;'C' clear shows not found
+ps2_search_pop:
+	pop	edx
+	pop	ecx
+	pop	ebx
+	pop	eax
 	ret
 
 ps2_search      endp
@@ -642,6 +651,10 @@ vmware_load_ini:
 	mov	wheel_enabled, ax
 
 vmware_enable_absolute:
+	push	eax
+	push	ebx
+	push	ecx
+	push	edx
 	; We need to do this *after* successfully setting up our hook.
 	; I don't know if these can fail, but OSDev Wiki doesn't check,
 	; and we do check if we're on something VMware-shaped before...
@@ -663,6 +676,10 @@ vmware_enable_absolute:
 	mov ebx, ABSPOINTER_ABSOLUTE
 	mov ecx, CMD_ABSPOINTER_COMMAND
 	call Backdoor
+	pop	edx
+	pop	ecx
+	pop	ebx
+	pop	eax
 
 ps2_enable_abort:
 	ret
@@ -728,9 +745,17 @@ ps2_disable_exit:
 	; (CB) Put the mouse back to relative on disable?
 
 	; Enable relative
+	push	eax
+	push	ebx
+	push	ecx
+	push	edx
 	mov ebx, ABSPOINTER_RELATIVE
 	mov ecx, CMD_ABSPOINTER_COMMAND
 	call Backdoor
+	pop	edx
+	pop	ecx
+	pop	ebx
+	pop	eax
 
 	ret
 

[FeralChild64]

Seems to work. I have manually applied the patch to current main - and the Total Commander works correctly now, at least for me.

[NattyNarwhal]

Interesting; I wonder what specific hunk is solving the issue. Could you try basically try adding them one by one until you can confirm one solves the issue?

I also wonder if this even solves the issue too, or if it's just luck that it's not triggering. I have to wonder what's happening, and why. I wouldn't think changing the extended registers from 16-bit code would hurt things either, but things surprise me when you drop to this level, unfortunately....

[NattyNarwhal]

FWIW, this also doesn't solve #22, so this is just for this, it seems.

[FeralChild64]

Could you try basically try adding them one by one until you can confirm one solves the issue?

I'll try when I have some more time.

[FeralChild64]

I don't have much time to test (I have full hands with my development), but it seems the first change (in ps2_int / ps2_no_data) is enough to stop the crashes.

[javispedro]

I wouldn't think changing the extended registers from 16-bit code would hurt things either, but things surprise me when you drop to this level, unfortunately....

I am also clobbering the 32-bit upper half of registers during Enable/Disable but I have not seen any issues so far.
As for clobbering inside the actual PS/2 callback, while I don't know the VMware BIOS, the VBox BIOS does push the 32-bit registers before calling the PS/2 callback. If it didn't, clobbering them from the PS/2 callback would be really bad.
Or it could also be that some program is either hooking the PS/2 interrupt or inserting itself in the PS/2 callback chain, and it does not expect the 32-bit registers to be clobbered by the other handlers.