Skip to content

Use sccache to accelerate android build #9587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kirklandsign merged 9 commits into from
Apr 16, 2025
Merged

Use sccache to accelerate android build #9587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0c2c7bc
[TEST] Try to see if sccache is in use
kirklandsign Mar 25, 2025
3ff84b0
Update _android.yml
kirklandsign Mar 25, 2025
9350bd3
Update pull.yml
kirklandsign Mar 26, 2025
35c7080
Trigger sccache
kirklandsign Mar 26, 2025
e97df94
Update
kirklandsign Mar 26, 2025
2041aa1
Use stub in docker
kirklandsign Mar 27, 2025
35592ef
Update install_cache.sh
kirklandsign Mar 27, 2025
b14d160
Update install_cache.sh
kirklandsign Mar 27, 2025
06727c3
Set in yml
kirklandsign Apr 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/_android.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ jobs:
script: |
set -eux

# Use sccache for NDK compiler as well
export CMAKE_CXX_COMPILER_LAUNCHER=sccache
export CMAKE_C_COMPILER_LAUNCHER=sccache
Comment on lines +25 to +27
Copy link
Contributor

@jathu jathu Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are your thoughts on exporting this in .ci/scripts/setup-linux.sh so we get it for all CI builds?

Copy link
Contributor Author

@kirklandsign kirklandsign Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rely on https://github.com/pytorch/executorch/blob/main/.ci/docker/common/install_cache.sh#L47 to trigger sccache for general linux.

Seems that if I put these in https://github.com/pytorch/executorch/blob/main/.ci/scripts/setup-linux.sh, it will be used not only on android, but generic, and I would avoid doing that to mess up with the set up above.

Copy link
Contributor

@jathu jathu Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, I'm hoping we can find one place to change and export these variables instead of multiple places. If someone adds another android job, they do not get caching if they forget these lines

Copy link
Contributor

@huydhn huydhn Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, you could try the simpler fix to install NDK before sccache in https://github.com/pytorch/executorch/blob/main/.ci/docker/ubuntu/Dockerfile#L57-L83. We install sccache as part of building Docker image and /opt/cache/bin is added to PATH in https://github.com/pytorch/executorch/blob/main/.ci/docker/common/install_cache.sh#L43. Atm, installing NDK happens after installing sccache, so I guess NDK path takes the priority?

Copy link
Contributor Author

@kirklandsign kirklandsign Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry not sure if that's the situation here. Currently for android build, clang path is /opt/ndk/toolchains/llvm/prebuilt/...bin and we should not use /opt/cache/bin/clang (stub which points to the system clang)?

Basically for android, we need /opt/ndk/.../clang to point to a stub as well?

Copy link
Contributor

@huydhn huydhn Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a stub works transparently as part of the Docker image. Setting CMAKE_CXX_COMPILER_LAUNCHER needs to be done later, i.e. people need to remember to set that explicitly.

Copy link
Contributor Author

@kirklandsign kirklandsign Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I feel like it's hard to do stub... In NDK we have so many variants of clang

armv7a-linux-androideabi21-clang    i686-linux-android29-clang++  x86_64-linux-android21-clang
armv7a-linux-androideabi21-clang++  i686-linux-android30-clang    x86_64-linux-android21-clang++
armv7a-linux-androideabi22-clang    i686-linux-android30-clang++  x86_64-linux-android22-clang
armv7a-linux-androideabi22-clang++  i686-linux-android31-clang    x86_64-linux-android22-clang++
armv7a-linux-androideabi23-clang    i686-linux-android31-clang++  x86_64-linux-android23-clang
armv7a-linux-androideabi23-clang++  i686-linux-android32-clang    x86_64-linux-android23-clang++

Copy link
Contributor

@jathu jathu Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kirklandsign I think you can use $ANDROID_NDK/ndk-which clang to get the specific location

Copy link
Contributor Author

@kirklandsign kirklandsign Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jathu I can have a try - just use clang instead of these variants (which are basically for clang --target=x86_64-linux-android21 stuff)

I was just worried if somehow android NDK triggered clang via x86_64-linux-android21-clang not clang

Copy link
Contributor Author

@kirklandsign kirklandsign Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huydhn @jathu I feel like for android it's still hard to get the stub working. See error like https://github.com/pytorch/executorch/actions/runs/14115338695/job/39544105035?pr=9587 for my version f2e12cb#diff-0a862d15a4e6dff44196719a594b983aece596c7dc38eba416d83e12a95206e6R92.


# The generic Linux job chooses to use base env, not the one setup by the image
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
conda activate "${CONDA_ENV}"
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/android-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,10 @@ jobs:
script: |
set -eux

# Use sccache for NDK compiler as well
export CMAKE_CXX_COMPILER_LAUNCHER=sccache
export CMAKE_C_COMPILER_LAUNCHER=sccache

# The generic Linux job chooses to use base env, not the one setup by the image
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
conda activate "${CONDA_ENV}"
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/android-release-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ jobs:
script: |
set -eux

# Use sccache for NDK compiler as well
export CMAKE_CXX_COMPILER_LAUNCHER=sccache
export CMAKE_C_COMPILER_LAUNCHER=sccache

# The generic Linux job chooses to use base env, not the one setup by the image
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
conda activate "${CONDA_ENV}"
Expand Down
Loading