Build from source doesn't work

[mergennachin]

🐛 Describe the bug

Repro:

git clone -b release/0.7 https://github.com/pytorch/executorch.git && cd executorch
conda create -yn executorch-test python=3.10.0 && conda activate executorch-test
./install_executorch.sh

Error:

  INFO:root:writing top-level names to torchao.egg-info/top_level.txt
      INFO:root:writing manifest file 'torchao.egg-info/SOURCES.txt'
      INFO:root:reading manifest file 'torchao.egg-info/SOURCES.txt'
      INFO:root:adding license file 'LICENSE'
      INFO:root:writing manifest file 'torchao.egg-info/SOURCES.txt'
      /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'torchao.csrc.rocm.swizzle' is absent from the `packages` configuration.
      !!

              ********************************************************************************
              ############################
              # Package would be ignored #
              ############################
              Python recognizes 'torchao.csrc.rocm.swizzle' as an importable package[^1],
              but it is absent from setuptools' `packages` configuration.

              This leads to an ambiguous overall configuration. If you want to distribute this
              package, please make sure that 'torchao.csrc.rocm.swizzle' is explicitly added
              to the `packages` configuration field.

              Alternatively, you can also rely on setuptools' discovery methods
              (for example by using `find_namespace_packages(...)`/`find_namespace:`
              instead of `find_packages(...)`/`find:`).

              You can read more about "package discovery" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

              If you don't want 'torchao.csrc.rocm.swizzle' to be distributed and are
              already explicitly excluding 'torchao.csrc.rocm.swizzle' via
              `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
              you can try to use `exclude_package_data`, or `include-package-data=False` in
              combination with a more fine grained `package-data` configuration.

              You can read more about "package data files" on setuptools documentation page:

              - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


              [^1]: For Python, any directory (with suitable naming) can be imported,
                    even if it does not contain any `.py` files.
                    On the other hand, currently there is no concept of package data
                    directory, all directories are treated like packages.
              ********************************************************************************

      !!
        check.warn(importable)
      INFO:root:creating build/lib.macosx-11.1-arm64-cpython-310/torchao/csrc/rocm/swizzle
      INFO:root:copying torchao/csrc/rocm/swizzle/swizzle.cpp -> build/lib.macosx-11.1-arm64-cpython-310/torchao/csrc/rocm/swizzle
      INFO:root:running build_ext
      -- The C compiler identification is AppleClang 17.0.0.17000013
      -- The CXX compiler identification is AppleClang 17.0.0.17000013
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      TORCHAO_INCLUDE_DIRS: /Users/mnachin/test/executorch/third-party/ao/torchao/experimental/../..
      -- Building with cpu/aarch64
      -- Building with ARM NEON dot product support
      CMake Deprecation Warning at /Users/mnachin/test/executorch/third-party/ao/build/temp.macosx-11.1-arm64-cpython-310/_deps/cpuinfo-src/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
        Compatibility with CMake < 3.10 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.


      -- Downloading Google Test to /Users/mnachin/test/executorch/third-party/ao/torchao/experimental/deps/googletest (define GOOGLETEST_SOURCE_DIR to avoid it)
      CMake Error at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
        Compatibility with CMake < 3.5 has been removed from CMake.

        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.

        Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.


      -- Configuring incomplete, errors occurred!
      Error: could not find CMAKE_PROJECT_NAME in Cache
      -- Downloading Google Benchmark to /Users/mnachin/test/executorch/third-party/ao/torchao/experimental/deps/googlebenchmark (define GOOGLEBENCHMARK_SOURCE_DIR to avoid it)
      CMake Error at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
        Compatibility with CMake < 3.5 has been removed from CMake.

        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.

        Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.


      -- Configuring incomplete, errors occurred!
      Error: could not find CMAKE_PROJECT_NAME in Cache
      CMake Error at /Users/mnachin/test/executorch/third-party/ao/build/temp.macosx-11.1-arm64-cpython-310/_deps/cpuinfo-src/CMakeLists.txt:346 (ADD_SUBDIRECTORY):
        ADD_SUBDIRECTORY given source
        "/Users/mnachin/test/executorch/third-party/ao/torchao/experimental/deps/googlebenchmark"
        which is not an existing directory.


      CMake Error at /Users/mnachin/test/executorch/third-party/ao/build/temp.macosx-11.1-arm64-cpython-310/_deps/cpuinfo-src/CMakeLists.txt:367 (ADD_SUBDIRECTORY):
        ADD_SUBDIRECTORY given source
        "/Users/mnachin/test/executorch/third-party/ao/torchao/experimental/deps/googletest"
        which is not an existing directory.


      CMake Warning at /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:125 (append_torchlib_if_found)
        CMakeLists.txt:109 (find_package)


      -- Found Torch: /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/lib/libtorch.dylib
      -- Building with TORCHAO_PARALLEL_BACKEND=ATEN_OPENMP
      CMake Warning at /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
        static library kineto_LIBRARY-NOTFOUND not found.
      Call Stack (most recent call first):
        /Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:125 (append_torchlib_if_found)
        Utils.cmake:19 (find_package)
        CMakeLists.txt:118 (target_link_torchao_parallel_backend)


      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/Users/mnachin/test/executorch/third-party/ao/setup.py", line 657, in <module>
          setup(
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
        File "/Users/mnachin/test/executorch/third-party/ao/setup.py", line 288, in build_extensions
          self.build_cmake(ext)
        File "/Users/mnachin/test/executorch/third-party/ao/setup.py", line 302, in build_cmake
          subprocess.check_call(
        File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/Users/mnachin/test/executorch/third-party/ao/torchao/experimental', '-DCMAKE_BUILD_TYPE=Release', '-DTORCHAO_BUILD_CPU_AARCH64=ON', '-DTORCHAO_BUILD_KLEIDIAI=OFF', '-DTORCHAO_BUILD_MPS_OPS=OFF', '-DTORCHAO_ENABLE_ARM_NEON_DOT=ON', '-DTORCHAO_ENABLE_ARM_I8MM=OFF', '-DTORCHAO_PARALLEL_BACKEND=aten_openmp', '-DTorch_DIR=/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/site-packages/torch/share/cmake/Torch', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/mnachin/test/executorch/third-party/ao/build/lib.macosx-11.1-arm64-cpython-310/torchao']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torchao
  Running setup.py clean for torchao
  Building wheel for pytorch_tokenizers (pyproject.toml) ... done
  Created wheel for pytorch_tokenizers: filename=pytorch_tokenizers-0.1.0-py3-none-any.whl size=10999 sha256=b315142727e2b41ce8d14a36a7a66535a62cbf92d6ffc28eda9ce41bda261f6f
  Stored in directory: /private/var/folders/_1/z_wzgpv50gn73c02j5xmcnf40000gn/T/pip-ephem-wheel-cache-5k41ati1/wheels/5b/8f/0b/c9659d45656d7f956e079caf2442c1f28a58d83aaa1156a1cf
Successfully built pytorch_tokenizers
Failed to build torchao
ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchao)
Traceback (most recent call last):
  File "/Users/mnachin/test/executorch/./install_executorch.py", line 235, in <module>
    main(sys.argv[1:])
  File "/Users/mnachin/test/executorch/./install_executorch.py", line 208, in main
    install_requirements(use_pytorch_nightly)
  File "/Users/mnachin/test/executorch/install_requirements.py", line 110, in install_requirements
    subprocess.run(
  File "/Users/mnachin/miniconda/envs/executorch-test/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/mnachin/miniconda/envs/executorch-test/bin/python', '-m', 'pip', 'install', '--no-build-isolation', 'third-party/ao', 'extension/llm/tokenizers']' returned non-zero exit status 1.
(executorch-test) mnachin@mnachin-mbp executorch %

Versions

0.7 release branch

[mergennachin]

cc @lucylq, @metascroy, @larryliu0820 , @psiddh

[larryliu0820]

We probably want to add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 before installing third-party/ao

[larryliu0820]

@mergennachin what is your cmake version?

[mergennachin]

@larryliu0820

(executorch-test) mnachin@mnachin-mbp executorch % cmake --version
cmake version 4.0.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
(executorch-test) mnachin@mnachin-mbp executorch %

[mergennachin]

But the cmake in conda environment is 3.31.6. During ./install_executorch.sh, I'm not sure which one it is using.

(executorch-test) mnachin@mnachin-mbp executorch % conda list
# packages in environment at /Users/mnachin/miniconda/envs/executorch-test:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h80987f9_6
ca-certificates           2025.2.25            hca03da5_0
certifi                   2025.7.14                pypi_0    pypi
click                     8.2.1                    pypi_0    pypi
cmake                     3.31.6                   pypi_0    pypi
filelock                  3.18.0                   pypi_0    pypi
fsspec                    2025.7.0                 pypi_0    pypi
jinja2                    3.1.6                    pypi_0    pypi
libffi                    3.4.4                hca03da5_1
lintrunner                0.12.7                   pypi_0    pypi
lintrunner-adapters       0.12.4                   pypi_0    pypi
markupsafe                3.0.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h313beb8_0
networkx                  3.4.2                    pypi_0    pypi
openssl                   1.1.1w               h1a28f6b_0
packaging                 25.0                     pypi_0    pypi
pip                       25.1               pyhc872135_2
python                    3.10.0               hbdb9e5c_5
pyyaml                    6.0.2                    pypi_0    pypi
readline                  8.2                  h1a28f6b_0
setuptools                78.1.1          py310hca03da5_0
sqlite                    3.50.2               h79febb2_1
sympy                     1.14.0                   pypi_0    pypi
tk                        8.6.14               h6ba3021_1
tomli                     2.2.1                    pypi_0    pypi
torch                     2.8.0                    pypi_0    pypi
typing-extensions         4.14.1                   pypi_0    pypi
tzdata                    2025b                h04d1e81_0
wheel                     0.45.1          py310hca03da5_0
xz                        5.6.4                h80987f9_1
zlib                      1.2.13               h18a0788_1
zstd                      1.5.7.2                  pypi_0    pypi

[mergennachin]

#12779