Merge branch 'main' into ber.a/defaultSeverity

Author: DedSec256

.github/agents/compiler-perf-investigator.md

@@ -0,0 +1,144 @@
+---
+name: Compile perf investigator
+description: Specialized agent for investigating F# build performance issues using the local compiler, trace/dump/benchmark tools, and rigorous, resumable insight-centric documentation.
+---
+
+# Compile perf investigator
+
+These are **general investigation instructions** for this agent, a template for perf analysis of slow/problematic F# compilation and build, suitable for a variety of scenarios (repos, snippets, gists).
+
+---
+
+## PRINCIPLES OF OPERATION
+
+- **Build insight, not just logs:** The ultimate goal is meaningful insights and verified hypotheses, not just raw data or trace files.
+- **Resumable workflow:** The agent must support investigation suspension/resumption: `TODO.md` is canonical for next steps.
+- **Iterative, hypothesis-driven:** Maintain and update a knowledge base of hypotheses (HYPOTHESIS.md) marking them as tested, confirmed, or denied to prevent repetition.
+- **Insight publication:** For every analysis, the primary artifact is INSIGHTS.md. This is the durable, published output which drives the investigation forward.
+- **Intermediate findings tracking:** Collect `HOT_PATHS.md` for all key observed code/activity paths and patterns before promoting to formal hypotheses or insights.
+- **Self-tooling:** When appropriate, the agent should generate and use .fsx (F# script) tools to parse, reduce, or extract patterns/insights from traces or build logs. These scripts are valuable artifacts.
+
+---
+
+## Absolute Requirements
+
+- **Always use** a fresh, local F# compiler and FSharp.Core where possible. Log full paths and proof the correct compiler is used.
+- **Matrix over** Debug/Release and ParallelCompilation on/off.
+- **For each build and analysis step:**
+  - Record all commands, timings, and results.
+  - Log all intermediate results (e.g., hot call stacks, wallclock timings, error messages) to `HOT_PATHS.md` as they are found.
+- **For every hypothesis or question:** 
+  - State it in `HYPOTHESIS.md` and update with status (untested/confirmed/denied) and pointers to relevant runs/artifacts.
+- **Primary artifact must be** clear, actionable, human-written text in `INSIGHTS.md`—this should summarize what the agent has learned, not just what it ran.
+
+---
+
+## Procedure (Resumable by TODO.md)
+
+### 1. Preparation
+- **Setup:** Clone/generate repo/snippet/etc.
+- **Clear old config:** Remove `global.json` unless needed.
+- **Prepare local compiler:** Use `PrepareRepoForRegressionTesting.fsx` and absolute env paths.
+
+### 2. Experiment Matrix
+
+For each:
+- ParallelCompilation: true, false
+- Configuration: Debug, Release
+
+Record:
+- build command and tool invocation
+- timings, exit codes
+- trace and log extraction
+
+### 3. Immediate Documentation (Always Do After Step)
+
+- **TODO.md:** Append completed step and state next.
+- **HOT_PATHS.md:** Add all notable methods, stacks, resource spikes, or patterns seen.
+- **ANALYSIS.md:** For each run, summarize key metrics, patterns, and questions that arise from data.
+
+### 4. Hypothesis Management
+
+- **HYPOTHESIS.md:** Maintain numbered/dated hypotheses about causes, behaviors, or fixes.  
+  - Mark each as untested/confirmed/denied (and why/where).
+  - Reference experimental runs or insights which test them.
+  - Never re-run a denied or already confirmed hypothesis: record, cross-reference, and always consult HYPOTHESIS.md on starting or resuming work.
+
+### 5. Insights as Product
+
+- **INSIGHTS.md:** After every meaningful update or analysis, synthesize a summary sentence/paragraph here:
+    - What was learned?
+    - What does the evidence suggest or refute?
+    - Are there new lines of investigation?
+    - (Do not just paste traces—extract the “so what”.)
+- At the end of a session, **publish** key conclusions and unresolved questions in INSIGHTS.md.
+
+### 6. Generate & Use .fsx Tools
+
+- When repetitive or large analysis is required (e.g., extracting hot methods from .nettrace, filtering logs), generate F# scripts (`.fsx`) that automate this, and:
+    - Save these as versioned artifacts.
+    - Log usage, output, and location for regeneration and proof.
+    - Use script outputs as input to HOT_PATHS.md or directly to INSIGHTS.md if discovery is significant.
+
+### 7. Best Practices
+
+- **If agent is interrupted:** On restart, consult `TODO.md`, `HYPOTHESIS.md`, and `INSIGHTS.md` to resume exactly where left off—never duplicate work and always cross-check hypotheses.
+- **If insight contradicts previous belief:** Update both `INSIGHTS.md` and point back to affected hypotheses.
+- **If a hypothesis is tested and denied,** record why and what observation/finding proves it false (with links/log references).
+- **If unable to explain a result with current hypotheses,** propose a new one and add it as untested.
+
+---
+
+## Documentation File Roles
+
+| File            | Purpose                                                              |
+|-----------------|-----------------------------------------------------------------------|
+| TODO.md         | Backlog of next actions; always current, supports resuming work       |
+| HOT_PATHS.md    | Key stacks, methods, or code paths surfaced during analysis           |
+| HYPOTHESIS.md   | All generated hypotheses, status-tracked, cross-referenced           |
+| INSIGHTS.md     | Published, reasoned summaries—main knowledge product of the agent     |
+| ANALYSIS.md     | Per-experiment metrics, summary comments, supplemental data           |
+| [tool].fsx      | Any custom F# scripts generated/executed as part of the investigation |
+
+---
+
+## Tool/Artifact Use Policy
+
+- **Never** treat a .nettrace, dump, or log as an end in itself—always extract, summarize, and relate to hypotheses and insights.
+- **Artifacts serve as reproducibility aids**, not terminal outputs. End users should rely on INSIGHTS.md and HYPOTHESIS.md for final conclusions.
+- **All generated tools (.fsx)** are themselves artifacts and must be logged and published for full transparency and reusability.
+
+---
+
+## Example Workflow
+
+1. Start by reading/setting TODO.md (`what next?`)
+2. Run experiment cell; immediately:
+   - Document run, logs, and .fsx tools (if any)
+   - Summarize in ANALYSIS.md
+   - Extract hot paths to HOT_PATHS.md
+3. Update hypothesis status in HYPOTHESIS.md
+4. Formulate/publish a new insight if progress is made
+5. Set new TODO in TODO.md, and finish session
+6. On any restart, agent must consult all MD files to avoid redundancy and repeat work only if hypothesis or scenario materially changes.
+
+---
+
+## Output Expectation
+
+At the end of an investigation, the primary deliverables are:
+- A well-maintained TODO.md to allow handoff/resume
+- A fully updated HYPOTHESIS.md (including rejected ideas, with reasons)
+- HOT_PATHS.md with all relevant interim findings, promoted as needed
+- INSIGHTS.md containing all meaningful conclusions and next questions, written for humans
+- Any custom `.fsx` tools, with location, invocation, and sample output
+
+---
+
+## Recap
+
+**This is an insight and hypothesis-driven agent for F# build perf analysis.  
+Its mission is to extract and publish meaningful, durable understanding—not just build logs or raw data.  
+The workflow is always resumable, transparent, and maximally reusable for community and future agents.**
+
+---

UseLocalCompiler.Directory.Build.props

@@ -18,6 +18,12 @@
     <FSharpPrefer64BitTools>True</FSharpPrefer64BitTools>
   </PropertyGroup>
 
+  <!-- 
+    Use FSharpTargetsShim to redirect the SDK to use the locally built F# targets.
+    This replaces all the individual UsingTask overrides since Microsoft.FSharp.NetSdk.targets
+    imports Microsoft.FSharp.Targets which registers all the F# build tasks.
+    See: https://github.com/dotnet/sdk/blob/main/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.Sdk.FSharpTargetsShim.targets
+  -->
   <PropertyGroup Condition="'$(LoadLocalFSharpBuild)' == 'True'">
     <LocalFSharpBuildBinPath>$(LocalFSharpCompilerPath)/artifacts/bin/fsc/$(LocalFSharpCompilerConfiguration)/net10.0</LocalFSharpBuildBinPath>
     <FSharpBuildAssemblyFile>$(LocalFSharpBuildBinPath)/FSharp.Build.dll</FSharpBuildAssemblyFile>
@@ -27,9 +33,6 @@
     <FSharpOverridesTargetsShim>$(LocalFSharpBuildBinPath)/Microsoft.FSharp.Overrides.NetSdk.targets</FSharpOverridesTargetsShim>
   </PropertyGroup>
 
-  <UsingTask Condition="'$(LoadLocalFSharpBuild)' == 'True'" TaskName="FSharpEmbedResourceText" AssemblyFile="$(FSharpBuildAssemblyFile)" Override="true" />
-  <UsingTask Condition="'$(LoadLocalFSharpBuild)' == 'True'" TaskName="FSharpEmbedResXSource" AssemblyFile="$(FSharpBuildAssemblyFile)" Override="true" />
-
   <ItemGroup>
     <Reference Include="$(LocalFSharpCompilerPath)/artifacts/bin/FSharp.Core/$(LocalFSharpCompilerConfiguration)/netstandard2.0/FSharp.Core.dll" />
   </ItemGroup>

eng/Version.Details.props

@@ -27,7 +27,7 @@ This file should be imported by eng/Versions.props
     <MicrosoftCodeAnalysisFeaturesPackageVersion>5.0.0-2.25480.7</MicrosoftCodeAnalysisFeaturesPackageVersion>
     <MicrosoftVisualStudioLanguageServicesPackageVersion>5.0.0-2.25480.7</MicrosoftVisualStudioLanguageServicesPackageVersion>
     <!-- dotnet/arcade dependencies -->
-    <MicrosoftDotNetArcadeSdkPackageVersion>11.0.0-beta.25601.2</MicrosoftDotNetArcadeSdkPackageVersion>
+    <MicrosoftDotNetArcadeSdkPackageVersion>11.0.0-beta.25603.2</MicrosoftDotNetArcadeSdkPackageVersion>
     <!-- _git/dotnet-optimization dependencies -->
     <optimizationlinuxarm64MIBCRuntimePackageVersion>1.0.0-prerelease.25502.1</optimizationlinuxarm64MIBCRuntimePackageVersion>
     <optimizationlinuxx64MIBCRuntimePackageVersion>1.0.0-prerelease.25502.1</optimizationlinuxx64MIBCRuntimePackageVersion>

eng/Version.Details.xml

@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <Dependencies>
-  <Source Uri="https://github.com/dotnet/dotnet" Mapping="fsharp" Sha="cc7f6e84d8dae36ad9ea51a5112627b8235982cc" BarId="291900" />
+  <Source Uri="https://github.com/dotnet/dotnet" Mapping="fsharp" Sha="29eefe27a350eb8b0bcbababa7863a0d1086295d" BarId="293166" />
   <ProductDependencies>
     <Dependency Name="Microsoft.Build" Version="18.1.0-preview-25515-01">
       <Uri>https://github.com/dotnet/msbuild</Uri>
@@ -76,9 +76,9 @@
     </Dependency>
   </ProductDependencies>
   <ToolsetDependencies>
-    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="11.0.0-beta.25601.2">
+    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="11.0.0-beta.25603.2">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>846929727a388f2b1fd3ccf21f67694f44b91b4d</Sha>
+      <Sha>9851192f7f7a7ee352358cce2627160fd1f2a54e</Sha>
     </Dependency>
     <Dependency Name="optimization.windows_nt-x64.MIBC.Runtime" Version="1.0.0-prerelease.25502.1">
       <Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-optimization</Uri>

eng/common/core-templates/job/job.yml

@@ -73,6 +73,8 @@ jobs:
     templateContext: ${{ parameters.templateContext }}
 
   variables:
+  - name: AllowPtrToDetectTestRunRetryFiles
+    value: true
   - ${{ if ne(parameters.enableTelemetry, 'false') }}:
     - name: DOTNET_CLI_TELEMETRY_PROFILE
       value: '$(Build.Repository.Uri)'

global.json

@@ -22,7 +22,7 @@
     "perl": "5.38.2.2"
   },
   "msbuild-sdks": {
-    "Microsoft.DotNet.Arcade.Sdk": "11.0.0-beta.25601.2",
+    "Microsoft.DotNet.Arcade.Sdk": "11.0.0-beta.25603.2",
     "Microsoft.DotNet.Helix.Sdk": "8.0.0-beta.23255.2"
   }
 }

src/Compiler/Symbols/Exprs.fs

@@ -508,14 +508,47 @@ module FSharpExprConvert =
 
     and GetWitnessArgs cenv (env: ExprTranslationEnv) (vref: ValRef) m tps tyargs : FSharpExpr list =
         let g = cenv.g
-        if g.langVersion.SupportsFeature(Features.LanguageFeature.WitnessPassing) && not env.suppressWitnesses then 
-            let witnessExprs = 
+        if g.langVersion.SupportsFeature(Features.LanguageFeature.WitnessPassing) && not env.suppressWitnesses then
+            /// There are two *conditional* properties a typar can have: equality and comparison.
+            /// A generic type having that constraint may be conditional on whether a specific type parameter to that generic has that
+            /// constraint.
+            /// This function returns `true` iff after unification, the type definition contains any conditional typars.
+            ///
+            /// Note that these conditions are only marked on typars that actually appear in the code, *not* on phantom types.
+            /// So `hasConditionalTypar` should tell us exactly when the type parameter is actually being used in the type's equality or
+            /// comparison.
+            let rec hasConditionalTypar ty =
+                match stripTyEqns g ty with
+                | TType_var (tp, _) -> tp.ComparisonConditionalOn || tp.EqualityConditionalOn
+                | TType_app (_, tinst, _) -> tinst |> List.exists hasConditionalTypar
+                | _ -> false
+
+            let witnessExprs =
                 match ConstraintSolver.CodegenWitnessesForTyparInst cenv.tcValF g cenv.amap m tps tyargs with
                 // There is a case where optimized code makes expressions that do a shift-left on the 'char'
                 // type.  There is no witness for this case.  This is due to the code
                 //    let inline HashChar (x:char) = (# "or" (# "shl" x 16 : int #) x : int #)
-                // in FSharp.Core. 
+                // in FSharp.Core.
                 | ErrorResult _  when vref.LogicalName = "op_LeftShift" && List.isSingleton tyargs -> []
+                // We don't need a witness either at compile time or runtime when there are conditional typars.
+                // Attempting to call a comparison operation with the type causes a compile-time check that all the generic type args
+                // support comparison (thanks to the ComparisonConditionalOn mechanism); the compile-time check doesn't need witnesses,
+                // it's just pure constraint solving.
+                // Nor do we need a witness for runtime logic: the compiler generates a `CompareTo` method (see
+                // `MakeValsForCompareAugmentation`) which handles the comparison by dynamically type-testing, not going through a witness.
+                //
+                // So we don't need to generate a witness.
+                //
+                // In fact, we *can't* generate a witness, because the constraint on the type parameter is only conditional: a rigid type
+                // parameter, defined without the `comparison` constraint, cannot have the constraint added to it later (that's what "rigid"
+                // means). It would change the type signature of the type to add this constraint to the type parameter!
+                //
+                // This code path is only reached through the auto-generated comparison/equality code, which only calls single-constraint
+                // intrinsics: there's exactly one constraint per type parameter in each of those two cases.
+                // In theory, if a function had an autogenerated `'a : comparison and 'b : SomethingElse`, where the `SomethingElse` was
+                // not comparison and failed for a different reason, we'd spuriously hide that failure here; but in fact the only code
+                // paths which get here have no other constraints.
+                | ErrorResult _ when List.exists hasConditionalTypar tyargs -> []
                 | res -> CommitOperationResult res
             let env = { env with suppressWitnesses = true }
             witnessExprs |> List.map (fun arg -> 

src/Compiler/TypedTree/TypedTreeBasics.fsi

@@ -141,6 +141,8 @@ val tryShortcutSolvedUnitPar: canShortcut: bool -> r: Typar -> Measure
 
 val stripUnitEqnsAux: canShortcut: bool -> unt: Measure -> Measure
 
+/// Follows type variable solutions: when a type variable has been solved by unifying it with another type,
+/// replaces that type variable with its solution.
 val stripTyparEqnsAux: nullness0: Nullness -> canShortcut: bool -> ty: TType -> TType
 
 val replaceNullnessOfTy: nullness: Nullness -> ty: TType -> TType

src/Compiler/TypedTree/TypedTreeOps.fsi

@@ -604,9 +604,41 @@ val reduceTyconRefMeasureableOrProvided: TcGlobals -> TyconRef -> TypeInst -> TT
 
 val reduceTyconRefAbbrevMeasureable: TyconRef -> Measure
 
-/// set bool to 'true' to allow shortcutting of type parameter equation chains during stripping
-val stripTyEqnsA: TcGlobals -> bool -> TType -> TType
-
+/// <summary>
+/// Normalizes types.
+/// </summary>
+/// <remarks>
+/// Normalizes a type by:
+/// <list>
+///   <item>replacing type variables with their solutions found by unification</item>
+///   <item>expanding type abbreviations</item>
+/// </list>
+/// as well as a couple of special-case normalizations:
+/// <list>
+///   <item>identifying <c>int&lt;1&gt;</c> with <c>int</c> (for any measurable type)</item>
+///   <item>identifying <c>byref&lt;'T&gt;</c> with <c>byref&lt;'T, ByRefKinds.InOut&gt;</c></item>
+/// </list>
+/// </remarks>
+/// <param name="canShortcut">
+/// <c>true</c> to allow shortcutting of type parameter equation chains during stripping
+/// </param>
+val stripTyEqnsA: TcGlobals -> canShortcut: bool -> TType -> TType
+
+/// <summary>
+/// Normalizes types.
+/// </summary>
+/// <remarks>
+/// Normalizes a type by:
+/// <list>
+///   <item>replacing type variables with their solutions found by unification</item>
+///   <item>expanding type abbreviations</item>
+/// </list>
+/// as well as a couple of special-case normalizations:
+/// <list>
+///   <item>identifying <c>int&lt;1&gt;</c> with <c>int</c> (for any measurable type)</item>
+///   <item>identifying <c>byref&lt;'T&gt;</c> with <c>byref&lt;'T, ByRefKinds.InOut&gt;</c></item>
+/// </list>
+/// </remarks>
 val stripTyEqns: TcGlobals -> TType -> TType
 
 val stripTyEqnsAndMeasureEqns: TcGlobals -> TType -> TType
@@ -707,6 +739,8 @@ val tcrefOfAppTy: TcGlobals -> TType -> TyconRef
 
 val tryTcrefOfAppTy: TcGlobals -> TType -> TyconRef voption
 
+/// Returns ValueSome if this type is a type variable, even after abbreviations are expanded and
+/// variables have been solved through unification.
 val tryDestTyparTy: TcGlobals -> TType -> Typar voption
 
 val tryDestFunTy: TcGlobals -> TType -> (TType * TType) voption

src/FSharp.Build/Microsoft.FSharp.NetSdk.targets

@@ -176,7 +176,7 @@ WARNING:  DO NOT MODIFY this file unless you are knowledgeable about MSBuild and
     <GenerateILLinkSubstitutions
       AssemblyName="$(AssemblyName)"
       IntermediateOutputPath="$(IntermediateOutputPath)">
-      <Output TaskParameter="GeneratedItems" ItemName="Embed" />
+      <Output TaskParameter="GeneratedItems" ItemName="EmbeddedResource" />
     </GenerateILLinkSubstitutions>
   </Target>