Use compiler-lowered DAM when trimming (#117624)

This uses compiler-lowered DAM for trimming compiler-generated code,
and adds test infrastructure and coverage for
DynamicallyAccessedMembers that is lowered by us for the compiler.

We also want to use the compiler-lowered attributes for generated
state machines, when trimming a .NET10+ assembly (leaving the old
heuristics for .NET9 and below, to avoid breaking trim analysis for
existing assemblies). However, the tests uncovered an issue with the
Roslyn implementation:
dotnet/roslyn#79333. Until that is fixed, we
don't rely on the lowered attribute for mapping type parameters by
default (the new behavior is opt-in under a new flag). Once we get a
fix we can remove this.

The old heuristics are still run for all TFMs simply for coverage, but
we don't use the result.

This adds test infrastructure to let us test against a polyfilled DAM
with CompilerLoweringPreserve (since we don't run ILLink/ILC tests
against a live corelib).

The existing ILLink tests (including those shared with ILC trimming
tests) are now built with TargetFrameworkAttribute marking the
assembly as targeting .NET 10. The compiler-generated code tests have
been duplicated for .NET 9 and .NET 10 to test both the old and new
behaviors.

Also fixes a test infrastructure issue where in a "copy" assembly, the
DAM and CompilerLoweringPreserve types that are compiled in are kept,
but the test infrastructure was filtering these out of the
"linkedMembers", leading to validation errors.

Author: sbomer

docs/design/tools/illink/compiler-generated-code-handling.md

@@ -95,6 +95,24 @@ static IEnumerable<int> TestLocalVariable ()
 }
 ```
 
+## Attribute propagation via CompilerLoweringPreserveAttribute
+
+To address the challenges of propagating user-authored attributes to compiler-generated code, .NET 10 introduced a general mechanism: `[CompilerLoweringPreserveAttribute]`. This attribute can be applied to other attribute types to instruct compilers to propagate those attributes to compiler-generated code.
+
+`DynamicallyAccessedMembersAttribute` is now marked with `[CompilerLoweringPreserve]`, so when the compiler generates new fields or type parameters (such as for local functions, iterator/async state machines, or primary constructor parameters), the relevant `DynamicallyAccessedMembers` annotations are automatically applied to the generated members. This allows trimming tools to directly use the annotations present in the generated code, without needing to reverse-engineer the mapping to user code.
+
+### .NET 10 and later
+
+For .NET 10 and later, trimming tools should rely on the compiler to propagate attributes such as `DynamicallyAccessedMembersAttribute` to all relevant compiler-generated code, as indicated by `[CompilerLoweringPreserve]`. No heuristics are needed for these assemblies. This isn't perfect because it's possible for such assemblies to be compiled with new Roslyn versions that could use different lowering strategies, so it's possible that the existing heuristics will break for new releases of a pre-`net10.0` assembly.
+
+To mitigate this there are a few options:
+
+1. Multitarget the library to `net10.0` (so that it is built with the new `CompilerLoweringPreserve` behavior and will avoid the heuristics)
+2. Fix the heuristics to work for code produced by new Roslyn versions
+3. The trimming tools could detect the presence of a polyfilled `DynamicallyAccessedMembersAttribute` type with `CompilerLoweringPreserve`. When present this would turn off the heuristics for the containing assembly.
+
+Another issue is that .NET 10 libraries might be built with `<GenerateTargetFrameworkAttribute>false</GenerateTargetFrameworkAttribute>`, and the tooling would not be able to detect the TargetFramework. Aside from setting `<GenerateTargetFrameworkAttribute>true</GenerateTargetFrameworkAttribute>`, mitigations 1. and 2. above would also apply to this scenario.
+
 ### Compiler dependent behavior
 
 Since the problems are all caused by compiler generated code, the behaviors depend on the specific compiler in use. The main focus of this document is the Roslyn C# compiler right now. Mainly since it's by far the most used compiler for .NET code. That said, we would like to design the solution in such a way that other compilers using similar patterns could also benefit from it.
@@ -560,9 +578,9 @@ and fields on the closure types.
 ### Long term solution
 
 Detecting which compiler generated items are used by any given user method is currently relatively tricky.
-There's no definitive marker in the IL which would let the trimmer confidently determine this information.
-Good long term solution will need the compilers to produce some kind of marker in the IL so that
-static analysis tools can reliably detect all of the compiler generated items.
+There's no definitive marker in the IL which would let the trimmer confidently determine this information
+for all of the above cases. Good long term solution will need the compilers to produce some kind of marker
+in the IL so that static analysis tools can reliably detect all of the compiler generated items.
 
 This ask can be described as:
 For a given user method, ability to determine all of the items (methods, fields, types, IL code) which were
@@ -573,6 +591,10 @@ helpers and other infrastructure which may be needed but is not directly attribu
 
 This should be enough to implement solutions for both suppression propagation and data flow analysis.
 
+For `DynamicallyAccessedMembersAttribute`, we have a long-term solution that relies on the
+`[CompilerLoweringPreserve]` attribute, which tells Roslyn to propagate `DynamicallyAccessedMembers`
+annotations to compiler-generated code.
+
 ### Possible short term solution
 
 #### Heuristic based solution

src/coreclr/tools/aot/ILCompiler.Compiler.Tests/DependencyGraphTests.cs

@@ -67,7 +67,7 @@ public void TestDependencyGraphInvariants(EcmaMethod method)
             CompilationModuleGroup compilationGroup = new SingleFileCompilationModuleGroup();
 
             NativeAotILProvider ilProvider = new NativeAotILProvider();
-            CompilerGeneratedState compilerGeneratedState = new CompilerGeneratedState(ilProvider, Logger.Null);
+            CompilerGeneratedState compilerGeneratedState = new CompilerGeneratedState(ilProvider, Logger.Null, disableGeneratedCodeHeuristics: true);
 
             UsageBasedMetadataManager metadataManager = new UsageBasedMetadataManager(compilationGroup, context,
                 new FullyBlockedMetadataBlockingPolicy(), new FullyBlockedManifestResourceBlockingPolicy(),

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/AssemblyExtensions.cs

@@ -0,0 +1,41 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System;
+using System.Diagnostics;
+using System.Reflection.Metadata;
+using Internal.TypeSystem;
+using Internal.TypeSystem.Ecma;
+
+#nullable enable
+
+namespace ILCompiler
+{
+    public static class AssemblyExtensions
+    {
+        public static Version? GetTargetFrameworkVersion(this EcmaAssembly assembly)
+        {
+            // Get the custom attributes from the assembly's metadata
+            MetadataReader reader = assembly.MetadataReader;
+            CustomAttributeHandle attrHandle = reader.GetCustomAttributeHandle(assembly.AssemblyDefinition.GetCustomAttributes(),
+                "System.Runtime.Versioning", "TargetFrameworkAttribute");
+            if (!attrHandle.IsNil)
+            {
+                CustomAttribute attr = reader.GetCustomAttribute(attrHandle);
+                CustomAttributeValue<TypeDesc> decoded = attr.DecodeValue(new CustomAttributeTypeProvider(assembly));
+                if (decoded.FixedArguments.Length == 1 && decoded.FixedArguments[0].Value is string tfm && !string.IsNullOrEmpty(tfm))
+                {
+                    var versionPrefix = "Version=v";
+                    var idx = tfm.IndexOf(versionPrefix);
+                    if (idx >= 0)
+                    {
+                        var versionStr = tfm.Substring(idx + versionPrefix.Length);
+                        if (Version.TryParse(versionStr, out var version))
+                            return version;
+                    }
+                }
+            }
+            return null;
+        }
+    }
+}

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Dataflow/CompilerGeneratedState.cs

@@ -24,17 +24,20 @@ public class CompilerGeneratedState
         private readonly record struct TypeArgumentInfo(
             /// <summary>The method which calls the ctor for the given type</summary>
             MethodDesc CreatingMethod,
-            /// <summary>Attributes for the type, pulled from the creators type arguments</summary>
+            /// <summary>Generic parameters of the creator used as type arguments for the type</summary>
             IReadOnlyList<GenericParameterDesc?>? OriginalAttributes);
 
         private readonly TypeCacheHashtable _typeCacheHashtable;
 
         private readonly Logger _logger;
 
-        public CompilerGeneratedState(ILProvider ilProvider, Logger logger)
+        private readonly bool _disableGeneratedCodeHeuristics;
+
+        public CompilerGeneratedState(ILProvider ilProvider, Logger logger, bool disableGeneratedCodeHeuristics)
         {
             _typeCacheHashtable = new TypeCacheHashtable(ilProvider);
             _logger = logger;
+            _disableGeneratedCodeHeuristics = disableGeneratedCodeHeuristics;
         }
 
         private sealed class TypeCacheHashtable : LockFreeReaderHashtable<MetadataType, TypeCache>
@@ -659,6 +662,16 @@ public bool TryGetCompilerGeneratedCalleesForUserMethod(MethodDesc method, [NotN
             MetadataType generatedType = (MetadataType)type.GetTypeDefinition();
             Debug.Assert(CompilerGeneratedNames.IsStateMachineOrDisplayClass(generatedType.Name));
 
+            // Avoid the heuristics for .NET10+, where DynamicallyAccessedMembers flows to generated code
+            // because it is annotated with CompilerLoweringPreserveAttribute.
+            if (_disableGeneratedCodeHeuristics &&
+                generatedType.Module.Assembly is EcmaAssembly asm && asm.GetTargetFrameworkVersion() >= new Version(10, 0))
+            {
+                // Still run the logic for coverage to help us find bugs, but don't use the result.
+                GetCompilerGeneratedStateForType(generatedType);
+                return null;
+            }
+
             var typeCache = GetCompilerGeneratedStateForType(generatedType);
             if (typeCache is null)
                 return null;

src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/Logger.cs

@@ -37,7 +37,7 @@ public class Logger
         private readonly bool _treatWarningsAsErrors;
         private readonly Dictionary<int, bool> _warningsAsErrors;
 
-        public static Logger Null = new Logger(new TextLogWriter(TextWriter.Null), null, false);
+        public static Logger Null = new Logger(new TextLogWriter(TextWriter.Null), null, false, true);
 
         public bool IsVerbose { get; }
 
@@ -51,7 +51,8 @@ public Logger(
             IEnumerable<string> singleWarnDisabledModules,
             IEnumerable<string> suppressedCategories,
             bool treatWarningsAsErrors,
-            IDictionary<int, bool> warningsAsErrors)
+            IDictionary<int, bool> warningsAsErrors,
+            bool disableGeneratedCodeHeuristics)
         {
             _logWriter = writer;
             IsVerbose = isVerbose;
@@ -62,22 +63,22 @@ public Logger(
             _suppressedCategories = new HashSet<string>(suppressedCategories, StringComparer.Ordinal);
             _treatWarningsAsErrors = treatWarningsAsErrors;
             _warningsAsErrors = new Dictionary<int, bool>(warningsAsErrors);
-            _compilerGeneratedState = ilProvider == null ? null : new CompilerGeneratedState(ilProvider, this);
+            _compilerGeneratedState = ilProvider == null ? null : new CompilerGeneratedState(ilProvider, this, disableGeneratedCodeHeuristics);
             _unconditionalSuppressMessageAttributeState = new UnconditionalSuppressMessageAttributeState(_compilerGeneratedState, this);
         }
 
-        public Logger(TextWriter writer, ILProvider ilProvider, bool isVerbose, IEnumerable<int> suppressedWarnings, bool singleWarn, IEnumerable<string> singleWarnEnabledModules, IEnumerable<string> singleWarnDisabledModules, IEnumerable<string> suppressedCategories, bool treatWarningsAsErrors, IDictionary<int, bool> warningsAsErrors)
-            : this(new TextLogWriter(writer), ilProvider, isVerbose, suppressedWarnings, singleWarn, singleWarnEnabledModules, singleWarnDisabledModules, suppressedCategories, treatWarningsAsErrors, warningsAsErrors)
+        public Logger(TextWriter writer, ILProvider ilProvider, bool isVerbose, IEnumerable<int> suppressedWarnings, bool singleWarn, IEnumerable<string> singleWarnEnabledModules, IEnumerable<string> singleWarnDisabledModules, IEnumerable<string> suppressedCategories, bool treatWarningsAsErrors, IDictionary<int, bool> warningsAsErrors, bool disableGeneratedCodeHeuristics)
+            : this(new TextLogWriter(writer), ilProvider, isVerbose, suppressedWarnings, singleWarn, singleWarnEnabledModules, singleWarnDisabledModules, suppressedCategories, treatWarningsAsErrors, warningsAsErrors, disableGeneratedCodeHeuristics)
         {
         }
 
-        public Logger(ILogWriter writer, ILProvider ilProvider, bool isVerbose)
-            : this(writer, ilProvider, isVerbose, Array.Empty<int>(), singleWarn: false, Array.Empty<string>(), Array.Empty<string>(), Array.Empty<string>(), false, new Dictionary<int, bool>())
+        public Logger(ILogWriter writer, ILProvider ilProvider, bool isVerbose, bool disableGeneratedCodeHeuristics)
+            : this(writer, ilProvider, isVerbose, Array.Empty<int>(), singleWarn: false, Array.Empty<string>(), Array.Empty<string>(), Array.Empty<string>(), false, new Dictionary<int, bool>(), disableGeneratedCodeHeuristics)
         {
         }
 
-        public Logger(TextWriter writer, ILProvider ilProvider, bool isVerbose)
-            : this(new TextLogWriter(writer), ilProvider, isVerbose)
+        public Logger(TextWriter writer, ILProvider ilProvider, bool isVerbose, bool disableGeneratedCodeHeuristics)
+            : this(new TextLogWriter(writer), ilProvider, isVerbose, disableGeneratedCodeHeuristics)
         {
         }
 

src/coreclr/tools/aot/ILCompiler.Compiler/ILCompiler.Compiler.csproj

@@ -343,6 +343,7 @@
 
     <Compile Include="Compiler\AnalysisBasedInteropStubManager.cs" />
     <Compile Include="Compiler\AnalysisBasedMetadataManager.cs" />
+    <Compile Include="Compiler\AssemblyExtensions.cs" />
     <Compile Include="Compiler\BodySubstitution.cs" />
     <Compile Include="Compiler\BodySubstitutionParser.cs" />
     <Compile Include="Compiler\DependencyAnalysis\AddressTakenMethodNode.cs" />

src/coreclr/tools/aot/ILCompiler.Trimming.Tests/ILCompiler.Trimming.Tests.csproj

@@ -50,6 +50,12 @@
     <RuntimeHostConfigurationOption Include="Mono.Linker.Tests.TargetFramework">
       <Value>$(TargetFramework)</Value>
     </RuntimeHostConfigurationOption>
+    <RuntimeHostConfigurationOption Include="Mono.Linker.Tests.TargetFrameworkMoniker">
+      <Value>$(TargetFrameworkMoniker)</Value>
+    </RuntimeHostConfigurationOption>
+    <RuntimeHostConfigurationOption Include="Mono.Linker.Tests.TargetFrameworkMonikerDisplayName">
+      <Value>$(TargetFrameworkMonikerDisplayName)</Value>
+    </RuntimeHostConfigurationOption>
     <RuntimeHostConfigurationOption Include="Mono.Linker.Tests.LinkerTestDir">
       <Value>$(ToolsProjectRoot)illink/test/</Value>
     </RuntimeHostConfigurationOption>

src/coreclr/tools/aot/ILCompiler.Trimming.Tests/TestCasesRunner/AssemblyChecker.cs

@@ -59,6 +59,10 @@ class LinkedMethodEntity : LinkedEntity
             "<Module>.StartupCodeMain(Int32,IntPtr)",
             "<Module>.MainMethodWrapper()",
             "<Module>.MainMethodWrapper(String[])",
+            "System.Diagnostics.CodeAnalysis.DynamicallyAccessedMembersAttribute.__GetFieldHelper(Int32,MethodTable*&)",
+            "System.Runtime.InteropServices.TypeMapping",
+            "System.Runtime.InteropServices.TypeMapping.GetOrCreateExternalTypeMapping<TTypeMapGroup>()",
+            "System.Runtime.InteropServices.TypeMapping.GetOrCreateProxyTypeMapping<TTypeMapGroup>()",
 
             // Ignore compiler generated code which can't be reasonably matched to the source method
             "<PrivateImplementationDetails>",
@@ -102,7 +106,7 @@ IEnumerable<string> VerifyImpl()
 
             // TODO - this is mostly attribute verification
             // foreach (var originalModule in originalAssembly.Modules)
-            //   VerifyModule(originalModule, linkedAssembly.Modules.FirstOrDefault(m => m.Name == originalModule.Name));
+            //   VerifyModule(originalModule, linkedAssembly.Modules.FirstOrDefault (m => m.Name == originalModule.Name));
 
             // TODO
             // VerifyResources(originalAssembly, linkedAssembly);
@@ -291,12 +295,9 @@ static bool ShouldIncludeType(TypeDesc type)
                     if (metadataType.Namespace.StartsWith("Internal"))
                         return false;
 
-                    // Simple way to filter out system assemblies - the best way would be to get a list
-                    // of input/reference assemblies and filter on that, but it's tricky and this should work for basically everything
-                    if (metadataType.Namespace.StartsWith("System"))
+                    if (metadataType.Module.Assembly is EcmaAssembly asm && asm.Assembly.GetName().Name == "System.Private.CoreLib")
                         return false;
 
-
                     return ShouldIncludeEntityByDisplayName(type);
                 }
 
@@ -2059,7 +2060,10 @@ private IEnumerable<string> VerifyKeptAllTypesAndMembersInAssembly(string assemb
             var missingInLinked = originalTypes.Keys.Except(linkedTypes.Keys);
 
             if (missingInLinked.Any())
+            {
                 yield return $"Expected all types to exist in the linked assembly {assemblyName}, but one or more were missing";
+                yield break;
+            }
 
             foreach (var originalKvp in originalTypes)
             {

src/coreclr/tools/aot/ILCompiler.Trimming.Tests/TestCasesRunner/ILCompilerOptions.cs

@@ -20,5 +20,6 @@ public class ILCompilerOptions
         public bool TreatWarningsAsErrors;
         public Dictionary<int, bool> WarningsAsErrors = new Dictionary<int, bool>();
         public List<string> SuppressedWarningCategories = new List<string>();
+        public bool DisableGeneratedCodeHeuristics;
     }
 }

src/coreclr/tools/aot/ILCompiler.Trimming.Tests/TestCasesRunner/ResultChecker.cs

@@ -78,17 +78,17 @@ public virtual void Check(TrimmedTestCaseResult testResult)
             {
                 _originalsResolver.Dispose();
             }
+        }
 
-            bool HasActiveSkipKeptItemsValidationAttribute(ICustomAttributeProvider provider)
+        internal static bool HasActiveSkipKeptItemsValidationAttribute(ICustomAttributeProvider provider)
+        {
+            if (TryGetCustomAttribute(provider, nameof(SkipKeptItemsValidationAttribute), out var attribute))
             {
-                if (TryGetCustomAttribute(provider, nameof(SkipKeptItemsValidationAttribute), out var attribute))
-                {
-                    object? by = attribute.GetPropertyValue(nameof(SkipKeptItemsValidationAttribute.By));
-                    return by is null ? true : ((Tool)by).HasFlag(Tool.NativeAot);
-                }
-
-                return false;
+                object? by = attribute.GetPropertyValue(nameof(SkipKeptItemsValidationAttribute.By));
+                return by is null ? true : ((Tool)by).HasFlag(Tool.NativeAot);
             }
+
+            return false;
         }
 
         protected virtual AssemblyChecker CreateAssemblyChecker(AssemblyDefinition original, TrimmedTestCaseResult testResult)