RPATH Fix for portable_lib Python Extension #13254

mergennachin · 2025-08-09T08:38:35Z

RPATH Fix for portable_lib Python Extension

Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from
the CI build environment.

Error:
ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002):
Library not loaded: @rpath/libtorch_python.dylib
Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so
Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file)

Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for
runtime library resolution.

Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669):

macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location
Linux: Uses $ORIGIN/../../../torch/lib for the same purpose
Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency

Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue
when using ExecutorTorch wheels built on CI.

Note: The same fix may be needed for _training_lib if it experiences similar issues.

Test Plan:

  # Build the wheel locally
  python setup.py bdist_wheel

  # create fresh conda env
  conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11
  
  # install 
  pip install ./dist/executorch-*.whl
  
  # Verify
  python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')"

pytorch-bot · 2025-08-09T08:38:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13254

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 22 Pending

As of commit af215ba with merge base de24489 ():

NEW FAILURE - The following job has failed:

pull / test-mediatek-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t 99962e0cd8a999a0708de642f7631fe38c1c26b8b14cab8265e543c7377005bf /exec failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

mergennachin · 2025-08-09T08:45:06Z

Note: if it's successful, make sure to merge to main as well

Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues.

CMakeLists.txt

mergennachin · 2025-08-11T19:07:57Z

@pytorchbot cherry-pick --onto main -c critical

RPATH Fix for portable_lib Python Extension Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues. Test Plan: ``` # Build the wheel locally python setup.py bdist_wheel # create fresh conda env conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11 # install pip install ./dist/executorch-*.whl # Verify python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')" ``` (cherry picked from commit 38062f8)

pytorchbot · 2025-08-11T19:10:16Z

Cherry picking #13254

The cherry pick PR is at #13290 and it is recommended to link a critical cherry pick PR with an issue.

Details for Dev Infra team

Raised by workflow job

RPATH Fix for portable_lib Python Extension Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues. Test Plan: ``` # Build the wheel locally python setup.py bdist_wheel # create fresh conda env conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11 # install pip install ./dist/executorch-*.whl # Verify python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')" ```

**Note: This is an attempt to cherry-pick Mergen's RPATH fix from #13254 onto main. Fixes #14421. Original description below.** Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues. Test Plan: ``` # Build the wheel locally python setup.py bdist_wheel # create fresh conda env conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11 # install pip install ./dist/executorch-*.whl # Verify python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')" ``` Co-authored-by: Mergen Nachin <[email protected]>

**Note: This is an attempt to cherry-pick Mergen's RPATH fix from #13254 onto main. Fixes #14421. Original description below.** Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues. Test Plan: ``` # Build the wheel locally python setup.py bdist_wheel # create fresh conda env conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11 # install pip install ./dist/executorch-*.whl # Verify python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')" ``` Co-authored-by: Mergen Nachin <[email protected]> (cherry picked from commit 641e737)

Co-authored-by: Gregory Comer <[email protected]> Co-authored-by: Mergen Nachin <[email protected]> fix from #13254 Fixes #14421.

**Note: This is an attempt to cherry-pick Mergen's RPATH fix from pytorch#13254 onto main. Fixes pytorch#14421. Original description below.** Problem: The _portable_lib.so Python extension built on CI couldn't find PyTorch libraries when installed locally because it had hardcoded absolute paths from the CI build environment. Error: ImportError: dlopen(.../_portable_lib.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libtorch_python.dylib Referenced from: .../executorch/extension/pybindings/_portable_lib.cpython-311-darwin.so Reason: tried: '/Users/runner/work/_temp/.../torch/lib/libtorch_python.dylib' (no such file) Root Cause: The CMake build was linking to PyTorch libraries using absolute paths from the build environment, without setting proper relative RPATHs for runtime library resolution. Solution: Added platform-specific relative RPATH settings to the portable_lib target in /Users/mnachin/executorch/CMakeLists.txt (lines 657-669): - macOS: Uses @loader_path/../../../torch/lib to find PyTorch libraries relative to the .so file location - Linux: Uses $ORIGIN/../../../torch/lib for the same purpose - Sets both BUILD_RPATH and INSTALL_RPATH to ensure consistency Impact: This allows the wheel-packaged _portable_lib.so to find PyTorch libraries regardless of the installation location, fixing the runtime linking issue when using ExecutorTorch wheels built on CI. Note: The same fix may be needed for _training_lib if it experiences similar issues. Test Plan: ``` # Build the wheel locally python setup.py bdist_wheel # create fresh conda env conda create -yn executorch_test_11 python=3.11.0 && conda activate executorch_test_11 # install pip install ./dist/executorch-*.whl # Verify python -c "from executorch.extension.pybindings._portable_lib import _load_for_executorch; print('Success!')" ``` Co-authored-by: Mergen Nachin <[email protected]>

mergennachin requested review from kirklandsign and larryliu0820 as code owners August 9, 2025 08:38

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 9, 2025

mergennachin force-pushed the 0.7_release_ready_pip_wheel_fix branch from 45f821c to 4230024 Compare August 9, 2025 08:42

mergennachin changed the title ~~0.7 release ready pip wheel fix~~ RPATH Fix for portable_lib Python Extension Aug 9, 2025

mergennachin requested a review from psiddh August 9, 2025 08:44

mergennachin mentioned this pull request Aug 9, 2025

verify regression test #13255

Closed

mergennachin force-pushed the 0.7_release_ready_pip_wheel_fix branch from 4230024 to af215ba Compare August 9, 2025 09:14

larryliu0820 reviewed Aug 9, 2025

View reviewed changes

CMakeLists.txt Show resolved Hide resolved

larryliu0820 force-pushed the 0.7_release_ready_pip_wheel_fix branch from 0793348 to af215ba Compare August 9, 2025 22:26

larryliu0820 approved these changes Aug 9, 2025

View reviewed changes

larryliu0820 merged commit 38062f8 into release/0.7 Aug 9, 2025
366 of 379 checks passed

larryliu0820 deleted the 0.7_release_ready_pip_wheel_fix branch August 9, 2025 22:53

GregoryComer mentioned this pull request Sep 18, 2025

Fix RPATH issues on MacOS #14421

Closed

GregoryComer mentioned this pull request Sep 18, 2025

RPATH Fix for portable_lib Python Extension #14422

Merged

pytorchbot mentioned this pull request Sep 19, 2025

RPATH Fix for portable_lib Python Extension #14442

Merged

abhinaykukkadapu pushed a commit that referenced this pull request Sep 23, 2025

RPATH Fix for portable_lib Python Extension (#14442)

9c172fe

Co-authored-by: Gregory Comer <[email protected]> Co-authored-by: Mergen Nachin <[email protected]> fix from #13254 Fixes #14421.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

RPATH Fix for portable_lib Python Extension #13254

RPATH Fix for portable_lib Python Extension #13254

Uh oh!

mergennachin commented Aug 9, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 9, 2025 •

edited

Loading

Uh oh!

mergennachin commented Aug 9, 2025

Uh oh!

Uh oh!

Uh oh!

mergennachin commented Aug 11, 2025

Uh oh!

pytorchbot commented Aug 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RPATH Fix for portable_lib Python Extension #13254

RPATH Fix for portable_lib Python Extension #13254

Uh oh!

Conversation

mergennachin commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13254

❌ 1 New Failure, 22 Pending

Uh oh!

mergennachin commented Aug 9, 2025

Uh oh!

Uh oh!

Uh oh!

mergennachin commented Aug 11, 2025

Uh oh!

pytorchbot commented Aug 11, 2025

Cherry picking #13254

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mergennachin commented Aug 9, 2025 •

edited

Loading

pytorch-bot bot commented Aug 9, 2025 •

edited

Loading