How to compile a subtree with intermediate opt passes

[uduse]

What I want to achieve:
The project is already buildable with scons. In the project, I have a file math.cpp that depends on add.h and some other files. There are other files in the project, such as fruit.cpp that I don't want to touch. The final executable depends on all these files. My goal is to apply additional LLVM passes such as -sroa to math.cpp and its dependencies.

What I have tried:
I added compilation_db to SConsstruct, and built the project normally. This exported a compilation database as a JSON file. I used python scons.py -Q --tree=prune math.o to get a tree of dependencies for my math module. This gave me a list of *.h files. I replaced *.h with *.o and locate these symbols in the compilation database. For each command I located this way in the compilation database, I transformed it into three commands (1) original command but with -emit-llvm (2) opt with bunch of passes (3) output *.o. I manually tested these steps with one dependency and worked okay.

My questions are:
(1) When I run the commands one by one through Python using subprocess.run, the compilation is extremely slow. Also it seems it does not skip the files that do not need to be touched. I am not sure using clang's Python interface could help. Does this sound like a good idea to you?
(2) Dependency resolution is already a part of scons, and I image it is possible to do this in scons. However, it is not obvious to me how to split one command action into three based on the dependency of a specific target. Can someone give me some hints how to do this?

[bdbaddog]

You'll have to explain more how you use -sroa .
I've not used it.

If you're running subprocess.run() chances are you're not using SCons properly.

Sounds like (and I could be entirely wrong so please correct me), you want to compile (at least) one source file with a different set of compiler flags?

Maybe -mllvm sroa added to CFLAGS ?

[bdbaddog]

Yes remove math.cpp from the sources for that library and use the output from the SharedObject() builder as the source for that.
So adding to the previous example

math_object = env.SharedObject(
    "math.cpp",
    SHCXXCOM=[
        "clang -emit-llvm -S -o ${TARGET.base}.ll $SOURCE",
        "opt -S -o ${TARGET.base}-1.ll -reg2mem ${TARGET.base}.ll",
        "llc -o $TARGET ${TARGET.base}-1.ll",
    ],
)

sources = [s for s in sources if "math.cpp" not in s]
mylib = env.StaticLibrary('mylib', sources + math_object)

[uduse]

This almost works 😄 So first I tried it on some file minus.cpp, and it had missing header. If I understand it correctly, SHCXXCOM supposed to inherit those flags, but maybe it doesn't do so when there are three commands. I found _CPPINCFLAGS seems useful here, and it works for minus.cpp. I tried the same trick for add.cpp but this time though it found things to include, it reports undefined reference from a file that uses add.cpp. My guess is that my linking process is also missing some flags too. I tried adding linker related flags here and there without success.

What I have now:

add_obj = env.SharedObject(
    "add.cpp",
    SHCXXCOM=[
        "$CXX -emit-llvm -S -o ${TARGET.base}.ll $SOURCE $_CPPINCFLAGS",
        "opt -S -o ${TARGET.base}-2.ll -O0 ${TARGET.base}.ll",
        "llc -o $TARGET ${TARGET.base}-2.ll",
    ],
)

[bdbaddog]

Is it possible whatever optimizers you're running are dropping symbols from the generated .o?

[uduse]

It's unlikely, -O0 should barely do anything. I think it has something to do with the fact that I'm doing cross-platform compilation. I need to get the flags in the first command right to give them access to everything... Are you aware of anything that might be different here if we cross-compile?

[bdbaddog]

There's lots different about cross compiling.
Do a regular compile of the object, run nm on it see if the reference is there, then nm the object you produce with your modified llvm command, see if the reference isn't there.

That's the best way to debug this issue.
Once you find what's causing the symbol to no longer be there, modify your more complicated compile flow to resolve.

This bit is out of scope of SCons.

[bdbaddog]

Really sounds like you just want to modify the Action()s which are used by the object builder for some source files?

[mwichmann]

If you're going to replace SHCXXCOM or some other core variable, it's useful to see what it was doing before and try to replicate. These aren't "absolute", they might differ by compiler (that is, msvc usually does something different), but for starters, on my Linux system, in addition to $SHCXXFLAGS $SHCCFLAGS, it contains a "common" variable $_CCCOMCOM (also used by SHCCCOM, CCCOM, CXXCOM), which is defined as: '$CPPFLAGS $_CPPDEFFLAGS $_CPPINCFLAGS'.
The generated defines would seem to be fairly important...

I'd say pursuing changing the action will probably involve less hit-and-miss than trying to change the command variable.

[uduse]

This is actually the first thing we tried. We tried to add some code in Action.py but it really sounds like a bad idea (it's part of scons's source code). It maybe a better we do it in SConstruct passing some executor, but it's very unclear how I should proceed...

[bdbaddog]

Don't modify Action.py. That would be the wrong way to go about this.