Fails to run find_package(Torch) on Windows with libtorch package

[jeongseok-meta]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Description:

The libtorch package for Windows in CMake is currently not working as expected. When using find_package(Torch) in a CMakeLists.txt file, it returns an error due to missing .dll files:

-- Caffe2: Found protobuf with new-style protobuf targets.
-- Caffe2: Protobuf version 28.3.0
CMake Error at .pixi/envs/default/Library/share/cmake/Caffe2/Caffe2Targets.cmake:122 (message):
  The imported target "c10" references the file
     "<root>/test_libtorch/.pixi/envs/default/Library/lib/c10.dll"
  but this file does not exist.  Possible reasons include:
  * The file was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and contained
     "<root>/test_libtorch/.pixi/envs/default/Library/share/cmake/Caffe2/Caffe2Targets.cmake"
  but not all the files it references.
Call Stack (most recent call first):
  .pixi/envs/default/Library/share/cmake/Caffe2/Caffe2Config.cmake:114 (include)
  .pixi/envs/default/Library/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:5 (find_package)
  
-- Configuring incomplete, errors occurred!

Reproduction Steps:

1. Create a new CMake project with the following CMakeLists.txt:

   cmake_minimum_required(VERSION 3.16.3)
   project(LibTorchExample)
   find_package(Torch CONFIG REQUIRED)

2. Run cmake on the project and observe the error message.

Note: This issue only occurs on Windows, while macOS and Linux work fine.

The .dll files are located in <prefix>/Library/bin/*, but the configuration is set to find them in <prefix>/Library/lib/*. I'm unsure where to look into to resolve this issue. Any help would be appreciated!

Installed packages

Package             Version                Build                     Size       Kind   Source
cuda-cudart         12.6.77                he0c23c2_0                161.1 KiB  conda  cuda-cudart
cuda-cudart_win-64  12.6.77                he0c23c2_0                22.1 KiB   conda  cuda-cudart_win-64
cuda-cupti          12.6.80                he0c23c2_0                2.8 MiB    conda  cuda-cupti
cuda-nvrtc          12.6.85                he0c23c2_0                29.2 MiB   conda  cuda-nvrtc
cuda-version        12.6                   h7480c83_3                20.4 KiB   conda  cuda-version
cudnn               9.3.0.75               h1361d0a_2                374.8 MiB  conda  cudnn
cusparselt          0.6.3.2                hffc9a7f_1                55.1 MiB   conda  cusparselt
intel-openmp        2024.2.1               h57928b3_1083             1.8 MiB    conda  intel-openmp
libabseil           20240722.0             cxx17_h4eb7d71_4          1.7 MiB    conda  libabseil
libblas             3.9.0                  26_win64_mkl              3.6 MiB    conda  libblas
libcblas            3.9.0                  26_win64_mkl              3.6 MiB    conda  libcblas
libcublas           12.6.4.1               he0c23c2_0                287.2 MiB  conda  libcublas
libcudss0           0.4.0.2                hdb9b9d5_2                21.7 MiB   conda  libcudss0
libcufft            11.3.0.4               he0c23c2_0                155.9 MiB  conda  libcufft
libcurand           10.3.7.77              he0c23c2_0                41.7 MiB   conda  libcurand
libcusolver         11.7.1.2               he0c23c2_0                91.3 MiB   conda  libcusolver
libcusparse         12.5.4.2               he0c23c2_0                116.9 MiB  conda  libcusparse
libhwloc            2.11.2                 default_ha69328c_1001     2.3 MiB    conda  libhwloc
libiconv            1.17                   hcfcfb64_2                621.2 KiB  conda  libiconv
liblapack           3.9.0                  26_win64_mkl              3.6 MiB    conda  liblapack
libmagma            2.8.0                  h630bcb8_2                69.8 MiB   conda  libmagma
libnvjitlink        12.6.85                he0c23c2_0                12.5 MiB   conda  libnvjitlink
libprotobuf         5.28.3                 h8309712_1                5.9 MiB    conda  libprotobuf
libtorch            2.5.1                  cuda126_mkl_h5239056_310  382.3 MiB  conda  libtorch
libuv               1.50.0                 h2466b09_0                285.1 KiB  conda  libuv
libwinpthread       12.0.0.r4.gg4f2fc60ca  h57928b3_9                35 KiB     conda  libwinpthread
libxml2             2.13.5                 he286e8c_1                1.5 MiB    conda  libxml2
libzlib             1.3.1                  h2466b09_2                54.2 KiB   conda  libzlib
mkl                 2024.2.2               h66d3029_15               98.3 MiB   conda  mkl
sleef               3.7                    h7e360cc_2                2 MiB      conda  sleef
tbb                 2021.13.0              h62715c5_1                147.9 KiB  conda  tbb
ucrt                10.0.22621.0           h57928b3_1                546.6 KiB  conda  ucrt
vc                  14.3                   h5fd82a7_24               17.3 KiB   conda  vc
vc14_runtime        14.42.34433            h6356254_24               735.9 KiB  conda  vc14_runtime

Environment info

System
------------
       Pixi version: 0.40.3
           Platform: win-64
   Virtual packages: __win=0=0
                   : __cuda=12.5=0
                   : __archspec=1=x86_64_v3
          Cache dir: C:\Users\jeongseok\AppData\Local\rattler\cache
       Auth storage: C:\Users\jeongseok\.rattler\credentials.json
   Config locations: No config files found

Global
------------
            Bin dir: <root>\.pixi\bin
    Environment dir: <root>\.pixi\envs
       Manifest dir: <root>\.pixi\manifests\pixi-global.toml

Project
------------
               Name: test_libtorch
            Version: 0.1.0
      Manifest file: <root>\test_libtorch\pixi.toml
       Last updated: 27-01-2025 12:58:50

Environments
------------
        Environment: default
           Features: default
           Channels: conda-forge
   Dependency count: 1
       Dependencies: libtorch
   Target platforms: win-64
              Tasks: config

[h-vetinari]

Yeah, I just noticed this in conda-forge/torchaudio-feedstock#14. The issue is that we move c10.dll from %SP_DIR%\torch\lib to %PREFIX%\Library\bin (note the switch from lib to bin), and so it looks in the wrong place (relative to a Torch_Root directory, which I think changes successfully to LIBRARY_PREFIX):

 │ │   CMake Error at %PREFIX%/Library/share/cmake/Caffe2/Caffe2Targets.cmake:122 (message):
 │ │     The imported target "c10" references the file
 │ │        "%PREFIX%/Library/lib/c10.dll"
 │ │     but this file does not exist.  Possible reasons include:

In this way, this is a follow-up to of #327 & #328.

I think the easiest might be to patch the CMake metadata here? Otherwise we'd have to patch the build system itself to ensure DLLs get installed into ...\bin.

Thoughts @mgorny @hmaarrfk @isuruf @danpetry?

[h-vetinari]

Actually, https://github.com/pytorch/pytorch/blob/v2.5.1/cmake/TorchConfig.cmake.in looks patchable

[traversaro]

I think (but did not test) that to install the c10 DLLs in bin it should be sufficient to remove the DESTINATION lib from https://github.com/pytorch/pytorch/blob/7d01f6e6f25835f67f564414b2f66953fd9af67a/c10/CMakeLists.txt#L166 and similar invocation of the install( command, as the default installation prefix for RUNTIME libraries is bin. If that does not work, I guess we need to explicitly add RUNTIME DESTINATION bin to the install( invocation.

[jeongseok-meta]

As a note, c10 is not the only issue; other targets' DLLs should also be set to bin, such as torch_cpu, torch, and so on. It seems we have two options: either update this line to copy files to lib (if I'm correct), which is the simplest solution, or update upstream (or adding a patch) for all targets, a more long-term approach suggested by @traversaro.

[h-vetinari]

I'm working on a patch + test.

[h-vetinari]

If that does not work, I guess we need to explicitly add RUNTIME DESTINATION bin to the install( invocation.

Indeed, simply removing the destination does not work:

  CMake Error at caffe2/CMakeLists.txt:1501 (install):
    install FILES given no DESTINATION!

[traversaro]

If that does not work, I guess we need to explicitly add RUNTIME DESTINATION bin to the install( invocation.

Indeed, simply removing the destination does not work:

  CMake Error at caffe2/CMakeLists.txt:1501 (install):
    install FILES given no DESTINATION!

Default DESTINATION argument is available for install(TARGETS, not for install(FILES like the command use there, so for those indeed we manually need to change the install dir to bin (related cmake issue: https://gitlab.kitware.com/cmake/cmake/-/issues/18393).

[h-vetinari]

@traversaro, I've switched things to (only for TARGETS)

-install(TARGETS torch_python DESTINATION "${TORCH_INSTALL_LIB_DIR}")
+install(TARGETS torch_python
+        LIBRARY DESTINATION lib
+        RUNTIME DESTINATION bin)

(and so on) but looking at the logs, it seems that the import libraries do not get installed anymore.

You're right though, before I made the mistake of also changing some install(FILES directives, so perhaps I should just go back to deleting DESTINATION from install(TARGETS...?

[traversaro]

Either we delete DESTINATION from install(TARGETS, or add ARCHIVE DESTINATION for installing import libraries.

[h-vetinari]

or add ARCHIVE DESTINATION for installing import libraries.

gah, why would import libraries be handled by ARCHIVE DESTINATION, it would make much more sense under LIBRARY DESTINATION IMO. In any case, I've deleted DESTINATION from targets in 46d06c1

[traversaro]

or add ARCHIVE DESTINATION for installing import libraries.

gah, why would import libraries be handled by ARCHIVE DESTINATION, it would make much more sense under LIBRARY DESTINATION IMO. In any case, I've deleted DESTINATION from targets in 46d06c1

I get your pain, but just for reference the related docs is in https://cmake.org/cmake/help/latest/command/install.html#signatures .