Creating hooks for packages

[maxbachmann]

I am the maintainer of RapidFuzz (https://github.com/rapidfuzz/RapidFuzz). Tools like cx-freeze or pyinstaller have a hard time with the package since all functions are dynamically dispatched. This is done for the following reasons:

• use pure python fallback if the C++ version failed to compile
• switch between SSE2 and AVX2 implementations based on CPU support

For pyinstaller I can provide a simple hook that informs the tool about all packages (see https://github.com/rapidfuzz/RapidFuzz/blob/main/src/rapidfuzz/__pyinstaller/hook-rapidfuzz.py). I did see that cx-freeze also has hooks for these things:

1. is it possible to ship these hooks with the package or do they have to be included here?
2. is there a simple equivalent of what I do with pyinstaller (iterate all modules and just exclude the __pyinstaller module)

[marcelotduarte]

Hooks need to be distributed with cx_Freeze. The previous maintainer had an idea of ​​creating something more dynamic, but he left the project and I haven't had time to study something like that yet.
But there is another alternative. I don't know if it will be possible to do this with Rapidfuzz because of the switching between implementations. But if it is possible, it is interesting because it only copies used modules.
It works like this: As I noticed, packages like pyqt*/pyside* and numpy distribute *.pyi files with imports (not just with type annotations), which are used by cx_Freeze to find the functions.
I keep some pyi files of pyqt/pyside with only the imports because they are not distributed in the conda-forge environment.

Here is an example: https://github.com/marcelotduarte/cx_Freeze/blob/main/cx_Freeze/importshed/PyQt5/QtCore.pyi
In the original, there are these imports and also the type annotations.

[maxbachmann]

The way it's implemented in rapidfuzz is something like the following:

_mod = "rapidfuzz.fuzz"
ratio = _fallback_import(_mod, "ratio")

Where _fallback_import is using importlib to import the modules. Thinking about this again I could absolutely inline this code by generating these files. This would lead to something like:

_impl = os.environ.get("RAPIDFUZZ_IMPLEMENTATION")
if _impl == "cpp":
    imported = False
    if supports(AVX2):
        with contextlib.suppress(ImportError):
            from rapidfuzz.fuzz_cpp_avx2 import ratio
            imported = True

    if not imported and supports(SSE2):
        with contextlib.suppress(ImportError):
            from rapidfuzz.fuzz_cpp_sse2 import ratio
            imported = True

    if not imported:
        from rapidfuzz.fuzz_cpp  import ratio
...

Would this lead to all these imports getting added to the binary (if they are available)?

[marcelotduarte]

The short answer is yes.
But if this variable is not set at runtime, it will not be used even if cx_Freeze copies the .pyd/.so files. Wouldn't it be better to just evaluate support for the feature and use it if it is available?

[maxbachmann]

That's actually what's done. There are just some users that would like to override the automatic behaviour. E.g. because their application absolutely requires the performance of the C++ version and so they want to get an error if something in their build process is broken leading to the fallback version being used. So the full version is:

_impl = os.environ.get("RAPIDFUZZ_IMPLEMENTATION")
if _impl == "cpp":
    # use c++
elif _impl == "python":
    # use python
else:
    # use c++ but fall back to python if unavailable

[maxbachmann]

I am now generating the files instead of using importlib. This appears to make them work as expected with cx-freeze and pyinstaller without needing any lists for hidden imports.