@dschuff @kripken WDYT?

I don't know much about windows, but would we need to build this as part of the emsdk, ship it, etc.? How important are the issues solved by this?

I don't know much about windows, but would we need to build this as part of the emsdk, ship it, etc.? How important are the issues solved by this?

I think since its just a tiny .exe we could just check it into git once.. then the create_entry_points.py script would make N copies of it, one for each script.

I think since its just a tiny .exe we could just check it into git once..

Isn't there windows-on-arm, so a single binary wouldn't work?

I think since its just a tiny .exe we could just check it into git once..

Isn't there windows-on-arm, so a single binary wouldn't work?

True! Although I'm not sure anyone is using that yet. We certainly don't yet provide emsdk binaries for windows/arm64.

I haven't reviewed the code yet but I think this idea is generally good We did something similar for the NaCl compiler (for a different reason, but a similar technique). I think I have a mild preference for building it with emsdk rather than checking it in, but I could be convinced.

3. Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

FWIW PowerShell scripts I added a while back should also already solve this issue, and they're chosen by default on Windows when users type emcc ... in PowerShell.

I don't mind the .exe approach, it might make it easier to solve the long-standing issue with Ctrl+C as well - that one is somewhat difficult to solve with either of scripting systems - just want to make sure it's not a burden to maintain.

We certainly don't yet provide emsdk binaries for windows/arm64.

Windows have long had system similar to macOS Rosetta, it's just that Windows on ARM has only picked up in popularity recently with new laptops, but essentially any x64 .exe is automatically supported on ARM via translation layer.

Native ARM might be nice to have for wasm-binaries, but for something as simple as this invocation script perf shouldn't matter as much.

3. Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

FWIW PowerShell scripts I added a while back should also already solve this issue, and they're chosen by default on Windows when users type emcc ... in PowerShell.

I don't mind the .exe approach, it might make it easier to solve the long-standing issue with Ctrl+C as well - that one is somewhat difficult to solve with either of scripting systems - just want to make sure it's not a burden to maintain.

Having to maintain the .ps1 and .bat files in parallel is another reason to make this switch.

Sadly I don't think the .ps1 files can actually replace the .bat files because IIRC powershell is somehow optional in windows so its not guaranteed to always be there.

Perhaps worth looking at py2exe than rolling something own?

Perhaps worth looking at py2exe than rolling something own?

I have looked at things like that over the years but the complexity always scared me off. It would also be little sad to loose the "edit-in-place" abilities of the plain-old-python files.

I asked Gemini CLI to help me write a better version. Its seems to have done it!

I asked it to find any discrpencies between these two and it came
up with:

launcher.c Analysis:

The C code replaces the batch script by directly launching Python or ccache. It correctly determines the Python executable and the script to run. Key features include:

• Long Path Support: It uses the \?\ prefix, removing the 260-character MAX_PATH limit, a major improvement over the batch script.
• Robust Path Discovery: It uses GetModuleFileNameW to reliably find its own path, avoiding the complex workarounds present in the batch script for PATH lookups.
• ccache Handling: The ccache logic is similar to the batch script.
• Argument and Stdin Handling: It uses the robust CommandLineToArgvW for arguments and a more reliable method for the Python stdin bug workaround.

Potential Issues & Discrepancies:

1. Filename Length Limits: The C launcher is superior. It removes the MAX_PATH limitation of the batch script, so there are no new length restrictions. This is a clear advantage.

2. Features Missing from launcher.c:

   • EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG: The batch script contains a workaround for a Windows 7 bug causing incorrect exit codes. The C launcher, despite being compiled for Windows 7
     compatibility, lacks this workaround. This is a potential regression for Windows 7 users.
   • %~f0 / %~dp0 Workaround: The batch script's workaround for a PATH lookup bug is obsolete, as the C launcher's method of finding its path is more reliable. This is an improvement.

Conclusion:

The primary concern is the missing EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG handling in launcher.c. This could cause issues for users on Windows 7. Otherwise, the C launcher is a significant
improvement, particularly in its handling of long file paths and command-line arguments. It does not introduce any new filename length limitations. I will now prepare my final answer based on
this.

Sadly I don't think the .ps1 files can actually replace the .bat files because IIRC powershell is somehow optional in windows so its not guaranteed to always be there.

Hm just saw this, what do you mean by optional? It's been bundled with Windows since Windows 7, that's around 15 years now.

Command line still exists for legacy usecases and they can never remove it, but it's purely in maintainance mode (critical bug fixing only).

Sadly I don't think the .ps1 files can actually replace the .bat files because IIRC powershell is somehow optional in windows so its not guaranteed to always be there.

Hm just saw this, what do you mean by optional? It's been bundled with Windows since Windows 7, that's around 15 years now.

Command line still exists for legacy usecases and they can never remove it, but it's purely in maintainance mode (critical bug fixing only).

IIRC there is some kind of "enable powershell" setting? Or "enable .ps1 script to lauch" setting? I don't know for sure since I'm not a windows user, but IIRC that is what came up last time we discuessed replacing the .bat files.

Including both .bat and .ps1 was IIRC the resulting compromise.

IIRC there is some kind of "enable powershell" setting? Or "enable .ps1 script to lauch" setting?

I'm not aware of one, no. The only issue is that if you launch PowerShell scripts from command line, you have to explicitly invoke them with pwsh script.ps1 unlike bat which are "self-executable" but as long as you are in PowerShell yourself, it's not something you have to worry about.

I've been using emcc ... which goes to the PowerShell script ever since I added the ps1 wrappers in a PR, and haven't had issues yet.

IIRC there is some kind of "enable powershell" setting? Or "enable .ps1 script to lauch" setting?

I'm not aware of one, no. The only issue is that if you launch PowerShell scripts from command line, you have to explicitly invoke them with pwsh script.ps1 unlike bat which are "self-executable" but as long as you are in PowerShell yourself, it's not something you have to worry about.

I've been using emcc ... which goes to the PowerShell script ever since I added the ps1 wrappers in a PR, and haven't had issues yet.

Right.. so we can't get rid of batch files as long we we want cmd.exe users to be able to type emcc on its own, which I think we absolutely do want.

It is a nice idea to try to use an exe instead of a bat launcher, though there is a complexity involved, namely Windows SmartScreen. It will prevent launching any .exe files downloaded off the internet, unless they are signed with a certificate from developer. So in order for this approach to work, we would have to maintain a code signing process so that Windows won't prevent the executables from working.

Batch files are notoriously painful to work with and mis-understood.
Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

Are there current tickets/bugs related to these .bat files? What is the unsolvable issue?

So in order for this approach to work, we would have to maintain a code signing process so that Windows won't prevent the executables from working.

Clang, wasmopt and other executables would be subject to the same.

Clang, wasmopt and other executables would be subject to the same.

Yes they are, that is a great point. SmartScreen operates in some kind of reputation based fashion: https://learn.microsoft.com/en-us/defender-endpoint/defender-endpoint-demonstration-app-reputation so I presume that clang and other tools are widely used in the database. Not sure of the exact details though.

Clang, wasmopt and other executables would be subject to the same.

Yes they are, that is a great point. SmartScreen operates in some kind of reputation based fashion: https://learn.microsoft.com/en-us/defender-endpoint/defender-endpoint-demonstration-app-reputation so I presume that clang and other tools are widely used in the database. Not sure of the exact details though.

I'm pretty sure we don't so any code signing of the binaryen executables (e.g. wasm-opt.exe). How have we gotten away without that so far?

It is a nice idea to try to use an exe instead of a bat launcher, though there is a complexity involved, namely Windows SmartScreen. It will prevent launching any .exe files downloaded off the internet, unless they are signed with a certificate from developer. So in order for this approach to work, we would have to maintain a code signing process so that Windows won't prevent the executables from working.

Batch files are notoriously painful to work with and mis-understood.
Works around several known issues with .bat files including one that is known unsolvable one (see emcc.bat for more details)

Are there current tickets/bugs related to these .bat files? What is the unsolvable issue?

I was thinking of this one:

:: On Windows 7, the compiler batch scripts are observed to exit with a non-zero errorlevel,    
:: even when the python executable above did succeed and quit with errorlevel 0 above.    
:: On Windows 8 and newer, this issue has not been observed. It is possible that this    
:: issue is related to the above python bug, but this has not been conclusively confirmed,    
:: so using a separate env. var EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG to enable the known    
:: workaround this issue, which is to explicitly quit the calling process with the previous    
:: errorlevel from the above command.   

Maybe we don't need to support windows 7 anymore?

I'm prepared to abandon this change if it turns out there are no perf benefits of bug fixes.

Maybe we don't need to support windows 7 anymore?

AFAIK adding \\?\ to bypass MAX_PATH will fail on Win7 (Win10 only). Since Win7 support ended 10 years ago I'd question if anyone is still running it for development.

I don't think we need to support WIndows 7 anymore. Neither Chrome nor Firefox does; I can't imagine developers using it either.

Also this issue:

:: Work around Windows bug https://github.com/microsoft/terminal/issues/15212 : If this    
:: script is invoked via enclosing the invocation in quotes via PATH lookup, then %~f0 and    
:: %~dp0 expansions will not work.                                                   
:: So first try if %~dp0 might work, and if not, manually look up this script from PATH.    
@if exist "%~f0" (                                                                   
  set "MYDIR=%~dp0"                                                                  
  goto FOUND_MYDIR                                                                   
)                                                                                    
@for %%I in (%~n0.bat) do (                                                          
  @if exist %%~$PATH:I (                                                             
    set MYDIR=%%~dp$PATH:I                                                           
  ) else (                                                                           
    echo Fatal Error! Due to a Windows bug, we are unable to locate the path to %~n0.bat.    
    echo To help this issue, try removing unnecessary quotes in the invocation of emcc,    
    echo or add Emscripten directory to PATH.                                        
    echo See github.com/microsoft/terminal/issues/15212 and                          
    echo github.com/emscripten-core/emscripten/issues/19207 for more details.        
  )                                                                                  
)

Regarding SmartScreeen, I would assume that if the clang and BInaryen binaries that we package are working, then whatever binary we build from this source should work too.
Agreed that we shouldn't do this if it doesn't simplify anything or make anything faster.

I did a test here https://github.com/juj/precompiled_exe_download_test and placed the launcher.c code from this PR in precompiled form there.

When I manually download the same file that I had compiled, I don't get a SmartScreen block. But when I download that file from another PC, it does indeed block the file:

I asked around at Unity's Windows team how SmartScreen works, and the replies I got there was that it works based on positive and negative heuristics on function-level code content hashes and code reputation. Unsigned code is subject to being blocked by SmartScreen depending on heuristics, but signed code can always run.

Also I pointed out there that we haven't been signing Emscripten .exes at Unity from the dawn of time and that hasn't been a problem, and now I got a new priority 0 bug assigned to me from partner legal to make sure that all Emscripten .exes we ship are signed..

So I don't think this will necessarily be a problem for us at Unity after we sign, but there is a chance that this kind of a small launcher file will block on SmartScreen, like it did on the above test.

On the topic of EM_WORKAROUND_WIN7_BAD_ERRORLEVEL_BUG and EM_WORKAROUND_PYTHON_BUG_34780, I think I can plan to remove both of these in to our next Emscripten update. At Unity we will support only Windows 10 + 11 in our next release, so assuming that both of those bugs no longer apply there, then we should be good to remove those from the .bat launchers.

On the performance front, it would be interesting to do some benchmarks to see how much these .bat fronts affect performance. If it is, say, 3-5% of compute time on many small files, then we might be curious to see if those can be avoided.

When I manually download the same file that I had compiled, I don't get a SmartScreen block. But when I download that file from another PC, it does indeed block the file:

Presumably the same thing happen if you run clang.exe or wasm-opt.exe?

It seems like the problem of signing exectuables is pre-existing (and not going away) since we already build and ship a whole bunch of executables that are essential to running emcc.

Presumably the same thing happen if you run clang.exe or wasm-opt.exe?

Curiously it doesn't. Like I mentioned before, Unity has been shipping unsigned clang.exe and wasm-opt.exe since dawn of time, and didn't have users reporting issues about that.

Presumably the same thing happen if you run clang.exe or wasm-opt.exe?

Curiously it doesn't. Like I mentioned before, Unity has been shipping unsigned clang.exe and wasm-opt.exe since dawn of time, and didn't have users reporting issues about that.

Is the difference that python launches wasm-opt.exe as a subprocess?

If you download and run wasm-opt.exe (which is unsigned) I don't see how that could be any different to running this launcher.exe. I can't see what heuristic could possibly result in that.. but I could be missing something.

Unsigned code is subject to being blocked by SmartScreen depending on heuristics, but signed code can always run.

There are cases when even signed code won't run (it's the level of validation on the cert). We had it with Windows tools we give to customers.