CLANGPDB vs CLANGDWARF Binary Size Discussion

[os-d]

@mdkinney pointed out that CLANGDWARF produces binaries that are smaller than CLANGPDB. There have been various discussions on this offline and online so I am centralizing the conversation here.

An example for OVMF X64 CLANGPDB DXE Core:

vs OVMF X64 CLANGDWARF DXE Core

@nate-desimone had suggested there may be an LLVM bug here. For the actual file size, I realized that neither CLANGPDB/CLANGDWARF for X64 is setting 0x1000 common-page-size/ALIGN (which also means memory protections can't be applied), setting that to match VS/GCC gives:

CLANGPDB:

CLANGDWARF:

So that makes sense where CLANGDWARF now has a larger file size, because the ELF conversion logic uses common-page-size for file alignment and section alignment whereas the PE image can have a lower file alignment. The virtual (in memory) size delta is also reduced from ~10% greater for CLANGPDB to ~0.6% greater for CLANGPDB, or in other words noise.

So, I think we are actually okay here when we go to 4K section alignment, which is what we already want. There may be other tuning opportunities, to be sure. There is a PR that is attempting to update CLANGPDB IA32/X64 to 4K section alignment here: #11454.

I'll note that CLANGPDB AARCH64 already has /ALIGN:4096.

[nate-desimone]

@os-d, Thanks for centralizing this and making an excellent write-up!

Interesting findings!

One thing to note is that the LLVM PE/COFF emitter does not support section alignment values smaller than 4KB. That is the reason the size of the non-4K aligned CLANGPDB image is byte-for-byte identical to the 4K aligned image. They both have 4K alignment even though /ALIGN:4096 wasn't explicitly supplied to the linker.

The thing that I find surprising is the .rdata section size. I suspect that in the case of ELF->PE/COFF conversion, the .rdata is getting combined with .text. When you consider that, the combined .rdata+.text section is still 2.34KB (~1.7%) larger. My hypothesis is that the LTO culling of read-only globals isn't as effective for the LLVM PE/COFF linker compared to the ELF linker. The nature of UEFI code does involve a fair number of read-only globals (protocol GUIDs for example) so I suspect that if we disassembled both images, we would find that the CLANGPDB image has more left-over unused symbols for constant globals (GUIDs, DevicePaths, etc.) than the other.

[os-d]

One thing to note is that the LLVM PE/COFF emitter does not support section alignment values smaller than 4KB. That is the reason the size of the non-4K aligned CLANGPDB image is byte-for-byte identical to the 4K aligned image. They both have 4K alignment even though /ALIGN:4096 wasn't explicitly supplied to the linker.

Ah, interesting, that does make sense, thanks for the context. Do you know if this is true for ARM64 as well? If so, @ardbiesheuvel that explains why CLANGPDB was okay emitting ADRP/LDR without caring if the section alignment was < 4KB.

A quick test says yes (I changed /ALIGN:4096 to /ALIGN:32 and adjusted the FILEALIGN for XIPFLAGS to 32):

Sections look identical between this and the /ALIGN:4096 binary.

The thing that I find surprising is the .rdata section size. I suspect that in the case of ELF->PE/COFF conversion, the .rdata is getting combined with .text. When you consider that, the combined .rdata+.text section is still 2.34KB (~1.7%) larger. My hypothesis is that the LTO culling of read-only globals isn't as effective for the LLVM PE/COFF linker compared to the ELF linker. The nature of UEFI code does involve a fair number of read-only globals (protocol GUIDs for example) so I suspect that if we disassembled both images, we would find that the CLANGPDB image has more left-over unused symbols for constant globals (GUIDs, DevicePaths, etc.) than the other.

Agreed. I was musing with @kuqin12 on this as well. Not sure if there is anything actionable here for us. Luckily the delta is relatively small. I assume changing that on the LLVM side would be complicated :)

[nate-desimone]

One thing to note is that the LLVM PE/COFF emitter does not support section alignment values smaller than 4KB. That is the reason the size of the non-4K aligned CLANGPDB image is byte-for-byte identical to the 4K aligned image. They both have 4K alignment even though /ALIGN:4096 wasn't explicitly supplied to the linker.

Ah, interesting, that does make sense, thanks for the context. Do you know if this is true for ARM64 as well? If so, @ardbiesheuvel that explains why CLANGPDB was okay emitting ADRP/LDR without caring if the section alignment was < 4KB.
A quick test says yes (I changed /ALIGN:4096 to /ALIGN:32 and adjusted the FILEALIGN for XIPFLAGS to 32):
Sections look identical between this and the /ALIGN:4096 binary.

Make's sense to me. I can't think of many reasons why ISA would change what the linker does here.

The thing that I find surprising is the .rdata section size. I suspect that in the case of ELF->PE/COFF conversion, the .rdata is getting combined with .text. When you consider that, the combined .rdata+.text section is still 2.34KB (~1.7%) larger. My hypothesis is that the LTO culling of read-only globals isn't as effective for the LLVM PE/COFF linker compared to the ELF linker. The nature of UEFI code does involve a fair number of read-only globals (protocol GUIDs for example) so I suspect that if we disassembled both images, we would find that the CLANGPDB image has more left-over unused symbols for constant globals (GUIDs, DevicePaths, etc.) than the other.

Agreed. I was musing with @kuqin12 on this as well. Not sure if there is anything actionable here for us. Luckily the delta is relatively small. I assume changing that on the LLVM side would be complicated :)

I think the one action that we should take is compare the symbol table generated by both images and see if any symbols are kept in one vs. the other. If yes, then take a look at where those extra symbols are. If they are in .rdata, then our shared hypothesis will be proven correct. At that point, we can take these findings and file an LLVM bug. I suspect that none of us are compiler experts, but we can at least let the compiler experts know about the issue.

[os-d]

@nate-desimone , actually, looking at the symbols, they seem essentially the same modulo some DWARF specific things. I think this difference is actually because the unwind info is being generated for CLANGPDB DEBUG and not for CLANGDWARF DEBUG. Then the .xdata section is getting merged with .rdata.

This seems to be the case, because when I set -fasynchronous-unwind-tables on CLANGDWARF, we get:

Which matches the .text + .rdata from CLANGPDB (aligned to 4K).

We also see the same thing in the opposite case, in the RELEASE build where both have no unwind tables:

CLANGPDB:

CLANGDWARF:

I'm not sure why the D60 doesn't have a section name, but it is marked as initialized data, which when combined with the .text section equals the CLANGDWARF .text section (aligned to 4K).

So I think mystery solved? Just the difference between CLANGDWARF not using 4K common-page-size and not having unwind tables? @nate-desimone @mdkinney do you have any open questions? From my point of view I think we can close this and say the CLANGPDB vs CLANGDWARF size is roughly the same.

[nate-desimone]

@os-d, I agree that it does seem like we solved the mystery. Does the size end up the same if you set -fno-asynchronous-unwind-tables in the CLANGPDB CC_FLAGS and /ALIGN:4096 in DLINK_FLAGS?

[os-d]

@nate-desimone yep, this is with CLANGPDB having /ALIGN:4096 and -fno-unwind-tables and CLANGDWARF having -z common-page-size=0x1000

[os-d]

@nate-desimone moved this to a separate comment thread because it isn't really relevant to the main point of this: I'm seeing that lld-link does support < 4K section alignment. My ARM64 test I had missed that ArmVirt.dsc.inc was overriding what I set in tools_def.txt to be 4KB.

Looks like they added support for it awhile back: https://reviews.llvm.org/D65736.

[os-d]

Per the findings in the thread, marking this discussion resolved.

[mdkinney]

Should this discussion spawn a GitHub issue to align CLANGPDB and CLANGDWARF sizes?

[os-d]

It certainly could. The 4KB section alignment for both toolchains is being handled already in this PR: #11454.

So then the only open question is whether CLANGDWARF wants to generate unwind tables on debug/no-opt builds. Probably, but if no one has wanted it so far, then it doesn't have to. My use case for CLANGDWARF is host applications on Linux, having unwind tables for debugging unit tests is nice.

If we are in agreement on that part, I can just put up a PR to include it for debug/no-opt CLANGDWARF, like other toolchains have.

[mdkinney]

For FW builds, I think they should be aligned in tools_def.txt.

In the host app EmulatorPkg build and host-based Unit Test builds, we can set additional flags to improve debug. That is already being done for some debug settings.

[os-d]

Addressed here: #12323