[TEST] Build wheels for Windows

[jgmelber]

No description provided.

[thomthehound]

I am interested in this. Can/may I help in some way?

[jgmelber]

I am interested in this. Can/may I help in some way?

That would be great. I will try to provide some more context that would be helpful. This is related to #2288. We currently build the wheels with Vitis, which is closed source and not available for Windows natively with compiler support for AIE AFAIK. Therefore, I have been slowly been working on untangling our CMake build infrastructure to remove the dependences on Vitis and use Peano (llvm-aie). Peano releases wheels nightly compiled for Windows. We could then build and release wheels for mlir-aie compiled natively for Windows.

[thomthehound]

I noticed there is an upstream bug in the llvm Windows nightlies that hard-paths the Visual Code DIA SDK to... I think it is /Professional/ or /Enterprise/ (I can't quite remember). It was reported all the way back in 2024, but they haven't addressed it yet and I seem to recall that they don't take outside PRs. Some time ago I wrote a bit of code to patch it on our end (I think it was just a CMake grep or something like that between build and compile time). I'll see if I can dig that up.

On the topic of Windows I also, some time ago, started a 'universal' Python script to build on Windows/Linux/WSL, consolidating several of our scripts into one cross-platform beast. It relies a little bit on some env var emission black-magic, but I nearly had it working. I'll evaluate the state of that this weekend and show you what I have, and you can tell me if you'd be interested in that.

Edit: Having just pulled it up, it appears I was running into the same issues you mentioned. Namely, getting the two runtime_lib folders to compile without xchesscc. It looks like I was just staging them from the repo as a placeholder.

[thomthehound]

This took me all weekend and I am up past my bedtime, so no PR just yet. Not even for a few more days. It's a load of spaghetti and lots of kludges. Plus several helper scripts, some sloppy syntax, a manual-sized bring-up list, manual env var pathing, plus it needs some suspension work and shocks, and brakes, brake pads, lining, steering box, transmission, rear-end...

But it appears to work.

I realize this cli output block isn't all that illustrative, but the whole thing compiled and ran, end-to-end, on Windows. And I mean the whole thing, including the mlir-aie wheel. This is just me testing that the wheel actually works.

(ironenv) (py312) PS F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul> python .\run_mlir_aie_example.py build
F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\run_mlir_aie_example.py:681: SyntaxWarning: invalid escape sequence '\X'
  Example (PowerShell): $env:XRT_ROOT = 'C:\Xilinx\XRT'
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul
[run] $ F:\dev\mlir-aie-win\mlir-aie\ironenv\Lib\site-packages\llvm-aie\bin\clang++.exe -O2 -std=c++20 --target=aie2p-none-unknown-elf -Wno-parentheses -Wno-attributes -Wno-macro-redefined -Wno-empty-body -DNDEBUG -IF:\dev\mlir-aie-win\mlir-aie\.iron\install\mlir-aie\include -DBIT_WIDTH=16 -c F:\dev\mlir-aie-win\mlir-aie\aie_kernels\aie2\scale.cc -o F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\scale.o
[gen] Writing MLIR: F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\aie_8192.mlir
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build
[run] $ F:\dev\mlir-aie-win\mlir-aie\ironenv\Scripts\python.exe -c "from aie.compiler.aiecc.main import main; main()" --aie-generate-xclbin --no-compile-host --xclbin-name=final_8192.xclbin --aie-generate-npu-insts --npu-insts-name=insts_8192.bin --no-xchesscc --no-xbridge aie_8192.mlir
[done] build
(ironenv) (py312) PS F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul> python .\run_mlir_aie_example.py run
F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\run_mlir_aie_example.py:681: SyntaxWarning: invalid escape sequence '\X'
  Example (PowerShell): $env:XRT_ROOT = 'C:\Xilinx\XRT'
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul
[run] $ F:\dev\mlir-aie-win\mlir-aie\ironenv\Lib\site-packages\llvm-aie\bin\clang++.exe -O2 -std=c++20 --target=aie2p-none-unknown-elf -Wno-parentheses -Wno-attributes -Wno-macro-redefined -Wno-empty-body -DNDEBUG -IF:\dev\mlir-aie-win\mlir-aie\.iron\install\mlir-aie\include -DBIT_WIDTH=16 -c F:\dev\mlir-aie-win\mlir-aie\aie_kernels\aie2\scale.cc -o F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\scale.o
[gen] Writing MLIR: F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\aie_8192.mlir
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build
[run] $ F:\dev\mlir-aie-win\mlir-aie\ironenv\Scripts\python.exe -c "from aie.compiler.aiecc.main import main; main()" --aie-generate-xclbin --no-compile-host --xclbin-name=final_8192.xclbin --aie-generate-npu-insts --npu-insts-name=insts_8192.bin --no-xchesscc --no-xbridge aie_8192.mlir
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul
[run] $ cmake -S F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul -B F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\_build -DTARGET_NAME=vector_scalar_mul_8192 -DIN1_SIZE=8192 -DIN2_SIZE=4 -DOUT_SIZE=8192 -DINT_BIT_WIDTH=16
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.26100.0 to target Windows 10.0.26200.
-- The C compiler identification is MSVC 19.44.35222.0
-- The CXX compiler identification is MSVC 19.44.35222.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (7.6s)
-- Generating done (0.2s)
-- Build files have been written to: F:/dev/mlir-aie-win/mlir-aie/programming_examples/basic/vector_scalar_mul/_build
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul
[run] $ cmake --build F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\_build --config Release
MSBuild version 17.14.23+b0019275e for .NET Framework

  1>Checking Build System
  Building Custom Rule F:/dev/mlir-aie-win/mlir-aie/programming_examples/basic/vector_scalar_mul/CMakeLists.txt
  Scanning sources for module dependencies...
  test_utils.cpp
  Compiling...
  test_utils.cpp
  test_utils.vcxproj -> F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\_build\Release\test_utils.lib
  Building Custom Rule F:/dev/mlir-aie-win/mlir-aie/programming_examples/basic/vector_scalar_mul/CMakeLists.txt
  Scanning sources for module dependencies...
  test.cpp
  Compiling...
  test.cpp
  vector_scalar_mul_8192.vcxproj -> F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\_build\vector_scalar_
  mul_8192.exe
  Building Custom Rule F:/dev/mlir-aie-win/mlir-aie/programming_examples/basic/vector_scalar_mul/CMakeLists.txt
[host] Copied host exe -> F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\vector_scalar_mul_8192.exe
[run] cwd=F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul
[run] $ F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\vector_scalar_mul_8192.exe -x F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\final_8192.xclbin -i F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul\build\insts_8192.bin -k MLIR_AIE

Avg NPU time: 645us.

Min NPU time: 645us.

Max NPU time: 645us.

PASS!

(ironenv) (py312) PS F:\dev\mlir-aie-win\mlir-aie\programming_examples\basic\vector_scalar_mul>

I had no idea how close I really was a few months ago. Almost all of the problems are upstream with llvm-aie.

Right now I'm leaning on a Windows Vitis install for some executables (OpenSSL what giving me trouble on my system, so I'm not compiling bootgen just yet), but I think that can be dealt with as I iterate. A pure Windows user will still need to supply their own xclbinutil.exe, but that can be had easily from the Windows Ryzen AI software (and more difficultly from compiling Windows XRT or downloading Vitis). Additionally, they will need their own pyxrt.pyd because the one from the Windows XRT compile doesn't work, Vitis and Ryzen AI don't supply it,(and obviously the Linux flavor is a problem. But I have a hand-rolled Cmakelist kicking around that takes care of that, so I can wire that in somewhere, too.

I was originally going to use MSYS2 to run the makefiles because vcpkg nmake doesn't work, but that was giving me trouble. So they are now scripts that take a 'best guess' at how to compile the examples. That ended up being a good thing, because it allowed me to alias around some issues with incorrectly named libraries in llvm-aie and a really nasty bug with improperly built Windows elfs pulling in Linux dependencies requires stripping the .deplibs section from, at least, crt1.o. I think the many critical issues with llvm-aie are what made me give up on my last attempt at this months ago. It's enough to make me wonder if they even have any downstream Windows consumers.

The problem really wasn't that much on our end after all. The only thing that gave me trouble there was us using fcntl.flock in the examples. But I was able to kludge around that using msvcrt.locking in a manner that is definitely going to need more testing.

[jgmelber]

Firstly, thanks for all the hard work on this and demonstrated results!! I am very excited to have a path towards getting Windows native compilation.

Right now I'm leaning on a Windows Vitis install for some executables (OpenSSL what giving me trouble on my system, so I'm not compiling bootgen just yet), but I think that can be dealt with as I iterate. A pure Windows user will still need to supply their own xclbinutil.exe, but that can be had easily from the Windows Ryzen AI software (and more difficultly from compiling Windows XRT or downloading Vitis). Additionally, they will need their own pyxrt.pyd because the one from the Windows XRT compile doesn't work, Vitis and Ryzen AI don't supply it,(and obviously the Linux flavor is a problem. But I have a hand-rolled Cmakelist kicking around that takes care of that, so I can wire that in somewhere, too.

It would be great to see if we can completely rely on open-source. Please let me know if there are any particular tools/resources I can help find or ensure are in open-source repos. xclbinutil should be in the XRT repository if compiled from source. I am a bit out of the loop on what is packaged or available for Windows, I can help coordinate XRT packaging from our internal teams once the list of tools is clear.

[thomthehound]

Firstly, thanks for all the hard work on this and demonstrated results!! I am very excited to have a path towards getting Windows native compilation.

Right now I'm leaning on a Windows Vitis install for some executables (OpenSSL what giving me trouble on my system, so I'm not compiling bootgen just yet), but I think that can be dealt with as I iterate. A pure Windows user will still need to supply their own xclbinutil.exe, but that can be had easily from the Windows Ryzen AI software (and more difficultly from compiling Windows XRT or downloading Vitis). Additionally, they will need their own pyxrt.pyd because the one from the Windows XRT compile doesn't work, Vitis and Ryzen AI don't supply it,(and obviously the Linux flavor is a problem. But I have a hand-rolled Cmakelist kicking around that takes care of that, so I can wire that in somewhere, too.

It would be great to see if we can completely rely on open-source. Please let me know if there are any particular tools/resources I can help find or ensure are in open-source repos. xclbinutil should be in the XRT repository if compiled from source. I am a bit out of the loop on what is packaged or available for Windows, I can help coordinate XRT packaging from our internal teams once the list of tools is clear.

I 100% agree that open-source should be the target. The Vitis reliance was just a time-saving convenience on my part so I didn't have to get (further) distracted fussing with my broken OpenSSL install. That is definitely the first thing I am going to fix. I also agree that the user should be using the compiled XRT source, but I didn't mention that I have to submit a PR to them in order to get it to build properly on Windows (the current .bat is broken, so I had to drive it manually). Overall, the build instructions for the user are going to be very convoluted for that, since Windows XRT appears to exist just so AMD can build the drivers. Having the option to grab a pre-compiled xrtbinutil might be convenient.

I haven't even investigated yet what is going on with pyxrt.pyd, but I suspect the official Windows XRT build might be unconditionally packaging an unsigned xrt_coreutil inside of it. Strangely, the Linux version doesn't seem to do that, so I more-or-less copied how that was built. I know we direct people to turn off SecureBoot, but I'm really not comfortable with that being a universal requirement. The way I am doing things right now works just fine with the signed driver binaries. My biggest asks from the XRT team would be for better Windows documentation and the ability to optionally turn off Spectre mitigation from the CLI, because it is such a PITA to compile with that on Windows and I seriously doubt the average user is going to care about that.

Now that I know this works, I'm going to spend the rest of the week cleaning up my mess and consolidate it into ~6-7 files/file-edits. I'm hoping that I'll have something to give you guys by next Monday. By then I should have a better idea of the true requirements and a list of suggestions for ease of packaging. I'll also try to put together a list of things that need to be fixed in llvm-aie, so maybe somebody with more credibility than I on that project can get some movement there. That would save a few hundred lines of code.

[jgmelber]

Having the option to grab a pre-compiled xrtbinutil might be convenient.

Heard.

I know we direct people to turn off SecureBoot, but I'm really not comfortable with that being a universal requirement.

This was from early days when the kernel driver wasn't included in Linux. At this point we can likely move on from that direction. And soon once XRT and xdna-driver are upstream in Linux we can definitely.

I'll also try to put together a list of things that need to be fixed in llvm-aie, so maybe somebody with more credibility than I on that project can get some movement there. That would save a few hundred lines of code.

Please let us know on this side too.

Thanks for the work and looking forward to your PR on ~Monday

[thomthehound]

I'm still working on this. It's just taking me a little bit more time than I had hoped to get it in a form that I consider both presentable and appropriate. The last 10% are always takes 90% of the time.

The extent of the aggravation that the XRT install is going to cause the average -- or even better-than-average -- user is becoming even more clear as I write the instructions. On the bright side, the XRT project accepted my PR to get Windows builds working (for the most part) without the user needing to tinker with the source. I also resolved the pyxrt.pyd issue (that was an error on my end), and bootgen works fine (just one little tweak added as a nice-to-have), so there is no need for Vitis to build Windows wheels. However, the install is so complicated that I am going to have to spin up a fresh Windows just to make sure my instructions are complete, correct, clear, and followable. This isn't a problem with mlir-aie; it's just about getting XRT, Windows, VS, vcpkg, and PowerShell all working together nicely.

I also have not yet fully validated that any of the changes I've made won't negatively impact POSIX users (although I was careful about that, so I don't anticipate problems). Linux is still going to default to using xchesscc everywhere it did before unless the user passes a flag to switch to force Peano.

Overall, this is going to take me several more days, but there is no point is issuing a PR if nobody can make the thing work due to lack of instructions.

[thomthehound]

Also, because I'm sure you are curious, the 'magic bullet' on Windows is to make sure Xilinx_XRT is never set. XRT_ROOT is fine, but Xilinix_XRT seems to be a poison to stop consuming system paths, and that nukes the whole thing. Clearly that is meant for internal driver builds. And, because the Windows install of XRT blithely encourages the user to execute the packaged setup.bat the same way they would source from the Linux install, they are sabotaged before they even start. So annoying, but there is nothing I can do about that but instruct.