Investigate how to run (most?) PyCOMTest in CI

[Avasam]

In #2467 , we (@Mscht and @geppi) first noticed that a few tests we would've expected to run and fail... aren't.

In #2491 (comment) I dig further, and here's what I found:

About the reason the CI doesn't run these tests, I think it's because PyCOMTest is a level 2 test

pywin32/com/win32com/test/testall.py

Lines 179 to 191 in 03e65bc

	custom_test_cases = [
	# Level 1 tests.
	[],
	# Level 2 tests.
	[
	PyCOMTest,
	PippoTest,
	],
	# Level 3 tests
	[],
	# Level 4 tests.
	[],
	]

whilst only level 1 tests are run on the CI

pywin32/win32/scripts/pywin32_testall.py

Lines 87 to 93 in 03e65bc

	# win32com
	maybes = [
	os.path.join(directory, "win32com", "test", "testall.py")
	for directory in [os.path.join(project_root, "com")] + site_packages
	]
	extras = remains + ["1"] # only run "level 1" tests in CI
	find_and_run(maybes, extras)

As @mhammond says:

That seems like a decision which aged poorly :)

Let's see if we can get more tests running on the CI. I'm opening this issue to centralize conversation around this and track blockers, progress, solutions, etc.

[mhammond]

Thanks for opening this! I expect to find most other tests are going to continue to be skipped without error (by handling the "not installed" error trying to create the object). So the most bang for buck here might come from working out how to build and pycomtest and test that.

[Avasam]

As a first order of business, as mentioned in #2491 (comment), I am unable to get the tests running even locally.

I deleted my venv, re-installed pywin32 globally, opened pwsh as admin, run the postinstall script, then tried running the test directly from the test folder. Now I get:

PS C:\Users\Avasam\AppData\Local\Programs\Python\Python310\Lib\site-packages\win32com\test> python testPyComTest.py
Registration command was:
C:\Users\Avasam\AppData\Local\Programs\Python\Python310\python.exe "C:\Users\Avasam\AppData\Local\Programs\Python\Python310\Lib\site-packages\win32com\test\..\servers\test_pycomtest.py" --unattended > nul 2>&1
Traceback (most recent call last):
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python310\Lib\site-packages\win32com\test\testPyComTest.py", line 25, in <module>
    RegisterPythonServer(
  File "C:\Users\Avasam\AppData\Local\Programs\Python\Python310\lib\site-packages\win32com\test\util.py", line 98, in RegisterPythonServer
    raise RuntimeError("Registration of engine '%s' failed" % filename)
RuntimeError: Registration of engine 'C:\Users\Avasam\AppData\Local\Programs\Python\Python310\Lib\site-packages\win32com\test\..\servers\test_pycomtest.py' failed

Edit: Drilling down into where exactly it's failing for me, and it's basically this line: gencache.EnsureModule("{6BCDCB60-5605-11D0-AE5F-CADD4C000000}", 0, 1, 1), (which is both in com/win32com/test/testPyComTest.py and com/win32com/servers/test_pycomtest.py). There's also this text that seems... unhelpful:

**** PyCOMTest is not installed ***
  PyCOMTest is a Python test specific COM client and server.
  It is likely this server is not installed on this machine.
  To install the server, you must get the win32com sources
  and build it using MS Visual C++

is com/win32com/src/ not built when running pip install . ?

There's also

pywin32/com/win32com/test/readme.txt

Lines 16 to 18 in 03e65bc

	The win32com source tree has source code to a C++ and VB component used purely
	for testing. You may like to build and register these, particularly if you
	are doing anything related to argument/result handling.

that seem to imply I need to build something in com/win32com, but I'm not sure what or how.

#2491 (comment)

To install the server, you must get the win32com sources
and build it using MS Visual C++

Yeah, that's frustrating :( We should probably work out how to build it in CI.

[geppi]

When I started to work on the enhancements for the COM Record support, I discovered that the related tests are not performed in the CI pipeline, see here and here.

A major obstacle to run pycomtest locally was the fact that the sources for the PyComTest server-dll only contain completely outdated Visual Studio 6.0 project files.
As mentioned in the second comment, I managed to compile the PyComTest server-dll with Visual Studio 2022. However, back at the time it looked like there was not much interest in this, so I kept running pycomtest just locally. If it would help, I can provide the Visual Studio 2022 solution and project files. Currently those files are excluded from revision control by .gitignore.

The next step for running pycomtest in the CI pipeline would be to integrate the build and registration process for the PyComTest server-dll into the github Actions workflow. We did something similar in the comtypes repository. In the comtypes case an external COM server executable is built with nmake. I'm neither very familiar with Visual Studio nor with github Actions. Therefore I don't know if it's possible to integrate the build process with Visual Studio into the workflow for the CI pipeline. nmake is a command line utility and was therefore easy to integrate.

[mhammond]

Thanks! And yeah, I agree the path forward here is probably a makefile (but an even hackier approach might be a script that just builds the entire project from scratch every time it is run - when used in CI it's always going to be doing a complete build anyway)

[Avasam]

FWIW MSVC and nmake should just be available in the CI from GitHub's Windows runner.
If it's not available from PATH, this script does a bit of special handling to find the location of the nmake executable that you can inspire yourself from if needed: https://github.com/mhammond/pywin32/blob/main/AutoDuck/make.py

I of course wouldn't care about anything related to ARM64 and cross-compilation (those are already not tested atm, since we'd need a VM or an ARM64 runner)

I have more experience dealing with GitHub actions, but yeah, if the PyComTest server can be built, it shouldn't be too hard (I hope) getting it running on the CI as well.

Then we can cleanup the implementation.

[geppi]

I've added PR #2493 to make the Visual Studio solution and project file available and modified the CI workflow configuration file to include the build process for the PyCOMTest server dll.

However, the build is failing due to missing v143 build tools.
@Avasam do you know which github Action would provide them?

In a first shot I gave uses: microsoft/setup-msbuild@v2 a try but it's obviously not providing the necessary tools.
On the comtypes side we were successful with uses: ilammy/msvc-dev-cmd@v1 but the CI workflow there is using nmake instead of msbuild.
Before I further experiment, maybe you have a better idea?

[geppi]

No need to bother with the github Actions anymore. I've retargeted the solution to MSVC v142 and now it builds in the CI workflow.

Next step would be to modify the Python unit tests and include the PyCOMTest stuff in the CI workflow.

[geppi]

In PR #2493 I now moved PyCOMTest into Level 1 of the unit tests to run it in the CI workflow.
However, there are some issues in the unit tests.

[Avasam]

Looks like we're not completely done here. testMultiQueryInterface and testVTableLocalServer are still failing in Python Debug Mode: https://github.com/mhammond/pywin32/actions/runs/14096220446/job/39483990776?pr=2502#step:10:549

You can further comment on #2502 directly if you have any interest in that.