-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Use an ObjectWriter-based PE writer for crossgen2 #120454
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…ctory code a little
… need real unwind/debug info for R2R in the same way that NAOT does
…we needed this are CodeView debug sections, which aren't supported in PE files
… top of PEObjectWriter/ObjectWriter
…e runtime functions data into .pdata
…to ObjectWriter and move R2R usage over to be based on that.
…pple will use it (only Mach-O uses it)
…of it) can be used from within the ObjectWriter when provided.
…te a file on disk in the new path. (Does it work? That's the next step)
/azp list |
/azp run runtime-coreclr r2r-extra, runtime-coreclr r2r, runtime-coreclr crossgen2, runtime-coreclr crossgen2 outerloop, runtime-coreclr crossgen2-composite, runtime-nativeaot-outerloop |
Azure Pipelines successfully started running 6 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR converts crossgen2 to use an ObjectWriter-based PE writer for ReadyToRun image generation, replacing the previous PE builder implementation. This change aims to unify the code generation infrastructure and prepare for supporting Mach-O format ReadyToRun images.
Key Changes
- Replaces the existing PE builder with a new PEObjectWriter that reuses COFF object writer infrastructure
- Adds support for ReadyToRun container format selection via command line option
- Removes several large files (SectionBuilder.cs, R2RPEBuilder.cs, RelocationHelper.cs) containing custom PE building logic
- Updates various ReadyToRun nodes to use proper section assignments and new ObjectWriter APIs
Reviewed Changes
Copilot reviewed 76 out of 84 changed files in this pull request and generated 4 comments.
Show a summary per file
File | Description |
---|---|
src/coreclr/tools/aot/crossgen2/Program.cs | Adds UseContainerFormat call for ReadyToRun compilation |
src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs | Adds OutputFormat command line option with PE format support |
src/coreclr/tools/aot/ILCompiler.ReadyToRun/ObjectWriter/SectionBuilder.cs | Removed - large PE building implementation |
src/coreclr/tools/aot/ILCompiler.ReadyToRun/ObjectWriter/R2RPEBuilder.cs | Removed - PE builder class |
src/coreclr/tools/aot/ILCompiler.ReadyToRun/ObjectWriter/RelocationHelper.cs | Removed - relocation helper class |
src/coreclr/tools/aot/ILCompiler.ReadyToRun/CodeGen/ReadyToRunObjectWriter.cs | Major refactoring to use PEObjectWriter instead of custom PE builder |
Various ReadyToRun node files | Updates to use proper section assignments and new ObjectWriter APIs |
src/coreclr/tools/Common/Compiler/ObjectWriter/PEObjectWriter.cs | New - PE image writer built on COFF infrastructure |
...oreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/ImportThunk.cs
Outdated
Show resolved
Hide resolved
src/coreclr/tools/aot/ILCompiler.ReadyToRun/ObjectWriter/ProfileFileBuilder.cs
Outdated
Show resolved
Hide resolved
.../aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/DebugDirectoryEntryNode.cs
Show resolved
Hide resolved
src/coreclr/tools/Common/Compiler/ObjectWriter/CoffObjectWriter.cs
Outdated
Show resolved
Hide resolved
...r/tools/aot/ILCompiler.ReadyToRun/Compiler/DependencyAnalysis/ReadyToRun/MethodWithGCInfo.cs
Show resolved
Hide resolved
/azp run runtime-coreclr r2r-extra, runtime-coreclr r2r, runtime-coreclr crossgen2, runtime-coreclr crossgen2 outerloop, runtime-coreclr crossgen2-composite |
Azure Pipelines successfully started running 5 pipeline(s). |
/azp run runtime-coreclr r2r-extra, runtime-coreclr r2r, runtime-coreclr crossgen2 |
Azure Pipelines successfully started running 3 pipeline(s). |
/azp run runtime-coreclr r2r-extra, runtime-coreclr r2r, runtime-coreclr crossgen2, runtime-coreclr crossgen2 outerloop, runtime-coreclr crossgen2-composite |
Azure Pipelines successfully started running 5 pipeline(s). |
My analysis of the failures: crossgen2-composite is in main: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1173552&view=ms.vss-test-web.build-test-results-tab r2r-extra: There's inconsistent occurrences of flaky tests like the ones I'm seeing here (stackoverflowtester in particular shows up quite a bit). The TestReadyToRun_Libraries runs were broken due to an infra issue. Running them again now. |
/azp run runtime-coreclr crossgen2 |
Azure Pipelines successfully started running 1 pipeline(s). |
Today, crossgen2 uses its own PE builder built on top of
System.Reflection.PortableExecutable
. However, this implementation ends up manually redoing the vast majority of the featuresSystem.Reflection.PortableExecutable
provides manually to support a number of edge cases that the library does not provide. Additionally, we plan to introduce a ReadyToRun format based on Mach-O (see #120065). There is no library likeSystem.Reflection.PortableExecutable
with a similar API that we could use for the Mach-O format, so we will either need to add some parallel path or move the PE path to something that can be shared with the Mach-O support.For NativeAOT's ILCompiler, we use an
ObjectWriter
library based loosely around LibObjectFile to enable us to write COFF, ELF, and Mach-O objects. This library provides a consistent API for all three formats.This PR changes crossgen2 to use a PE writer based on the COFF object writer from our
ObjectWriter
library. Additionally, this PR adds a number of features to theObjectWriter
library to support various constructs that are necessary for emitting a ReadyToRun image (symbol ranges, node size "relocs", deterministic file checksums, etc.).This should make completing #120065 much easier as the vast majority of the "How do we emit code/data correctly so we can meet Apple's requirements?" question is already met by the
MachObjectWriter
in theObjectWriter
library, leaving a much smaller tail of work when adding support for it.