[MLIR][Python] fix stubgen for downstream users

[makslevental]

This PR fixes 4 things related to stubgen:

1. Support for both find_package(nanobind) and FetchContent_Declare(nanobind);
2. Disambiguate stubgen target names by package prefix correctly (prior using a global MLIR_PYTHON_PACKAGE_PREFIX caused collisions for some users);
3. Fix generation for declare_mlir_python_extension(StandalonePythonSources.NanobindExtension);
4. Emits DEPFILEs for the add_custom_command to prevent regenerating each build.

[llvmbot]

@llvm/pr-subscribers-mlir

Author: Maksim Levental (makslevental)

Changes

FetchContent reports something like /build/CMakeFiles/pkgRedirects for nanobind_DIR, and the correct thing for nanobind_SOURCE_DIR.

---

Full diff: https://github.com/llvm/llvm-project/pull/157589.diff

1 Files Affected:

• (modified) mlir/cmake/modules/AddMLIRPython.cmake (+6)

diff --git a/mlir/cmake/modules/AddMLIRPython.cmake b/mlir/cmake/modules/AddMLIRPython.cmake
index 85c80276c1bcf..cd9c00df4c82c 100644
--- a/mlir/cmake/modules/AddMLIRPython.cmake
+++ b/mlir/cmake/modules/AddMLIRPython.cmake
@@ -116,10 +116,16 @@ function(generate_type_stubs MODULE_NAME DEPENDS_TARGET MLIR_DEPENDS_TARGET OUTP
     ""
     "OUTPUTS"
     ${ARGN})
+  # for people doing pip install nanobind
   if(EXISTS ${nanobind_DIR}/../src/stubgen.py)
     set(NB_STUBGEN "${nanobind_DIR}/../src/stubgen.py")
   elseif(EXISTS ${nanobind_DIR}/../stubgen.py)
     set(NB_STUBGEN "${nanobind_DIR}/../stubgen.py")
+  # for people using FetchContent or ExternalProject
+  elseif(EXISTS ${nanobind_SOURCE_DIR}/src/stubgen.py)
+    set(NB_STUBGEN "${nanobind_SOURCE_DIR}/src/stubgen.py")
+  elseif(EXISTS ${nanobind_SOURCE_DIR}/stubgen.py)
+    set(NB_STUBGEN "${nanobind_SOURCE_DIR}/stubgen.py")
   else()
     message(FATAL_ERROR "generate_type_stubs(): could not locate 'stubgen.py'!")
   endif()

[makslevental]

@joker-eph can you try this patch and see if it fixes this error that you mentioned:

# RUN: at line 7
"/home/mamini/bin/cmake-3.26.4-linux-x86_64/bin/cmake" --build . --target check-standalone | tee /home/mamini/projects/llvm-project/build/tools/mlir/test/Examples/standalone/Output/test.toy.tmp
# executed command: /home/mamini/bin/cmake-3.26.4-linux-x86_64/bin/cmake --build . --target check-standalone
# .---command stderr------------
# | ninja: error: '/home/mamini/projects/llvm-project/mlir/python/mlir/_mlir_libs/_mlirRegisterEverything.pyi', needed by 'python_packages/standalone/mlir_standalone/_mlirRegisterEverything.pyi', missing and no known rule to make it
# `-----------------------------
# error: command failed with exit status: 1
# executed command: tee /home/mamini/projects/llvm-project/build/tools/mlir/test/Examples/standalone/Output/test.toy.tmp

[rkayaith]

I don't fully follow the cmake-fu, but I've tried this out and it does indeed fix the issues I hit downstream.

[joker-eph]

Right now it still fails:

# | ninja: error: 'build/tools/mlir/python/_mlir_libs/_mlir/__init__.pyi', needed by 'python_packages/standalone/mlir_standalone/_mlir/__init__.pyi', missing and no known rule to make it

[makslevental]

Right now it still fails:

# | ninja: error: 'build/tools/mlir/python/_mlir_libs/_mlir/__init__.pyi', needed by 'python_packages/standalone/mlir_standalone/_mlir/__init__.pyi', missing and no known rule to make it

can you paste your CMake configure

[joker-eph]

I'm using the "getting started" basic command somehow:

cmake -G Ninja ../llvm    -DLLVM_ENABLE_PROJECTS=mlir    -DLLVM_BUILD_EXAMPLES=ON    -DLLVM_TARGETS_TO_BUILD="Native;NVPTX;AMDGPU"    -DCMAKE_BUILD_TYPE=Release    -DLLVM_ENABLE_ASSERTIONS=ON  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_ENABLE_LLD=ON   -DLLVM_CCACHE_BUILD=ON -DMLIR_ENABLE_BINDINGS_PYTHON=ON 

[makslevental]

Closed in favor of #157853