[polly][CUDA][clang] How to only enable polly flags for host code compilation when using clang++ for CUDA compilation?

[BwL1289]

We are compiling CUDA libraries (i.e. cutlass, cudnn_frontend, nccl, etc) from sources using clang++ for both host and device code.

At linkage time with polly enabled (-mllvm=-polly -mllvm=-polly-vectorizer=stripmine), for both host and device code, compilation can take hours.

Without polly, compilation drops to a few minutes.

Since polly is only relevant for host code anyway, how can I enable polly flags only for host code compilation and disable it for device code compilation?

Some relevant context:

• Using CMake, here are our default CUDAFLAGS. Using these elapsed compilation time for cutlass was ~4 hours.

export CUDAFLAGS="
-O3 \
-flto=thin -ffat-lto-objects -Wl,--lto-whole-program-visibility \
-mllvm=-polly -mllvm=-polly-vectorizer=stripmine \
-pipe -Qunused-arguments -fident -fcolor-diagnostics \
-Wno-cuda-compat
"

• Removing polly, like the following, drops cutlass compilation time to ~9 minutes:

export CUDAFLAGS="
-O3 \
-flto=thin -ffat-lto-objects -Wl,--lto-whole-program-visibility \
-pipe -Qunused-arguments -fident -fcolor-diagnostics \
-Wno-cuda-compat
"

• We've also tried various experiments like removing just --lto-whole-program-visibility. All other experiments yielded the same results — cutlass compilation time was ~4 hours, with 99% of time spent on linking steps.

Let me know if there's other context I can provide.

[BwL1289]

@EugeneZelenko bumping this. Haven't been able to locate anything in the polly documentation.

[EugeneZelenko]

@BwL1289: Please ping developers who were active in polly recently. I help with issues labeling which should notify relevant people.

[BwL1289]

@BwL1289: Please ping developers who were active in polly recently. I help with issues labeling which should notify relevant people.

Thanks. @el-ev CC @kazutakahirata any guidance is appreciated.

[el-ev]

Please ping someone else. I'm not familiar with polly, and my changes to polly were non-functional.

[BwL1289]

Please ping someone else. I'm not familiar with polly, and my changes to polly were non-functional.

Sorry... and apologies in advance... @Meinersbur @kostasalv @jtstogel

[Meinersbur]

This is more like a driver question than a Polly question. The intended way to pass arguments to only the host and not the device compilation is clang -Xarch_host -polly.

During testing it I noticed it does not work when compiling CUDA. The test

llvm-project/clang/test/Driver/offload-Xarch.c

Line 38 in 65d5301

// CUDA: "-cc1" "-triple" "x86_64-unknown-linux-gnu" {{.*}}"-O3"

is failing when replacing "-Xarch_host -O3" with "-Xarch -polly" @jhuber6 ?

[jhuber6]

This is more like a driver question than a Polly question. The intended way to pass arguments to only the host and not the device compilation is clang -Xarch_host -polly.

During testing it I noticed it does not work when compiling CUDA. The test

llvm-project/clang/test/Driver/offload-Xarch.c

Line 38 in 65d5301
// CUDA: "-cc1" "-triple" "x86_64-unknown-linux-gnu" {{.*}}"-O3"
is failing when replacing "-Xarch_host -O3" with "-Xarch -polly" @jhuber6 ?

That's not a clang option as far as I'm aware. Maybe -Xarch_host -mllvm=-polly?

[BwL1289]

This is more like a driver question than a Polly question. The intended way to pass arguments to only the host and not the device compilation is clang -Xarch_host -polly.
During testing it I noticed it does not work when compiling CUDA. The test
llvm-project/clang/test/Driver/offload-Xarch.c
Line 38 in 65d5301
// CUDA: "-cc1" "-triple" "x86_64-unknown-linux-gnu" {{.*}}"-O3"
is failing when replacing "-Xarch_host -O3" with "-Xarch -polly" @jhuber6 ?

That's not a clang option as far as I'm aware. Maybe -Xarch_host -mllvm=-polly?

Thanks. We got this working with the following. We're testing a few variations this morning.

...
-Xarch_host=-mllvm=-polly \
-Xarch_host=-mllvm=-polly-vectorizer=stripmine

[jhuber6]

Thanks. We got this working with the following. We're testing a few variations this morning.

...
-Xarch_host=-mllvm=-polly \
-Xarch_host=-mllvm=-polly-vectorizer=stripmine

I don't think -Xarch_host works as a joined argument, you need -Xarch_host -mllvm=-polly.