Merge pull request #23800 from dotnet-maestro-bot/merge/master-to-release/3.0

[automated] Merge branch 'master' => 'release/3.0'

Author: wtgodbe

BuildToolsVersion.txt

@@ -1 +1 @@
-3.0.0-preview4-03828-01
+3.0.0-preview4-03906-01

Documentation/Profiling/davbr-blog-archive/Creating an IL-rewriting profiler.md

@@ -19,12 +19,12 @@ Keep in mind there are many ways to do this. I'll outline one of the more straig
 
 - In your **ICorProfilerCallback2::ModuleLoadFinished** callback, you call **ICorProfilerInfo2::GetModuleMetadata** to get a pointer to a metadata interface on that module. 
 - QI for the metadata interface you want. Search MSDN for "IMetaDataImport", and grope through the table of contents to find topics on the metadata interfaces. 
-- Once you're in metadata-land, you have access to all the types in the module, including their fields and function prototypes. You may need to parse metadata signatures and [this signature parser](samples/sigparser.cpp) may be of use to you. 
+- Once you're in metadata-land, you have access to all the types in the module, including their fields and function prototypes. You may need to parse metadata signatures and [this signature parser](samples/sigparse.cpp) may be of use to you. 
 - In your **ICorProfilerCallback2::JITCompilationStarted** callback, you may use **ICorProfilerInfo2::GetILFunctionBody** to inspect the original IL, and **ICorProfilerInfo2::GetILFunctionBodyAllocator** and then **ICorProfilerInfo2::SetILFunctionBody** to replace that IL with your own. 
 
 **Q: What about NGEN?**
 
-If you want to rewrite IL of NGENd modules, well, it's kind of too late because the original IL has already been compiled into native code. However, you do have some options.  If your profiler sets the **COR\_PRF\_USE\_PROFILE\_IMAGES** monitor event flag, that will force the "NGEN /Profile" version of the modules to load if they're available.  (I've already blogged a little about "NGEN /Profile", including how to generate those modules, [here](enter-leave-tailcall-hooks-part-1-basics.md).)  So, at run-time, one of two things will happen for any given module.
+If you want to rewrite IL of NGENd modules, well, it's kind of too late because the original IL has already been compiled into native code. However, you do have some options.  If your profiler sets the **COR\_PRF\_USE\_PROFILE\_IMAGES** monitor event flag, that will force the "NGEN /Profile" version of the modules to load if they're available.  (I've already blogged a little about "NGEN /Profile", including how to generate those modules, [here](ELT Hooks - The Basics.md).)  So, at run-time, one of two things will happen for any given module.
 
 1) If you set **COR\_PRF\_USE\_PROFILE\_IMAGES** and the NGEN /Profile version is available, it will load.  You will then have the opportunity to respond to the **JITCachedFunctionSearchStarted** callback.  When a function from an NGEN /Profile module is about to be executed for the first time, your profiler receives the **JITCachedFunctionSearchStarted** callback.  You may then set the \*pbUseCachedFunction [out] parameter to FALSE, and that will force the CLR to JIT the function instead of using the version that was already compiled into the NGEN /Profile module.  Then, when the CLR goes to JIT the function, your profiler receives the **JITCompilationStarted** callback and can perform IL rewriting just as it does above for functions that exist in non-NGENd mdoules.  What's nice about this approach is that, if you only need to instrument a few functions here and there, it can be faster not to have to JIT everything, just so you get the **JITCompilationStarted** callback for the few functions you're interested in.  This approach can therefore improve startup performance of the application while it's being profiled.  The disadvantage, though, is that your profiler must ensure the NGEN /Profile versions of all the modules get generated beforehand and get installed onto the user's machine.  Depending on your scenarios and customers, this may be too cumbersome to ensure.
 

ILAsmVersion.txt

@@ -1 +1 @@
-3.0.0-preview4-27603-71
+3.0.0-preview5-27606-71

build-test.sh

@@ -160,6 +160,75 @@ generate_layout()
         echo "Resolve runtime dependences via $nextCommand"
         eval $nextCommand
     fi
+
+    # Precompile framework assemblies with crossgen if required
+    if [ $__DoCrossgen -ne 0 ]; then
+        precompile_coreroot_fx
+    fi
+}
+
+precompile_coreroot_fx()
+{
+    echo "${__MsgPrefix}Running crossgen on framework assemblies in CORE_ROOT: '${CORE_ROOT}'"
+
+    # Read the exclusion file for this platform
+    skipCrossGenFiles=($(read_array "$(dirname "$0")/tests/skipCrossGenFiles.${__BuildArch}.txt"))
+
+    local overlayDir=$CORE_ROOT
+
+    filesToPrecompile=$(find -L $overlayDir -iname \*.dll -not -iname \*.ni.dll -not -iname \*-ms-win-\* -type f )
+    for fileToPrecompile in ${filesToPrecompile}
+    do
+        local filename=${fileToPrecompile}
+        if is_skip_crossgen_test "$(basename $filename)"; then
+                continue
+        fi
+        echo Precompiling $filename
+        $overlayDir/crossgen /Platform_Assemblies_Paths $overlayDir $filename 1> $filename.stdout 2>$filename.stderr
+        local exitCode=$?
+        if [[ $exitCode != 0 ]]; then
+            if grep -q -e '0x80131018' $filename.stderr; then
+                printf "\n\t$filename is not a managed assembly.\n\n"
+            else
+                echo Unable to precompile $filename.
+                cat $filename.stdout
+                cat $filename.stderr
+                exit $exitCode
+            fi
+        else
+            rm $filename.{stdout,stderr}
+        fi
+    done
+}
+
+declare -a skipCrossGenFiles
+
+function is_skip_crossgen_test {
+    for skip in "${skipCrossGenFiles[@]}"; do
+        if [ "$1" == "$skip" ]; then
+            return 0
+        fi
+    done
+    return 1
+}
+
+# Get an array of items by reading the specified file line by line.
+function read_array {
+    local theArray=()
+
+    if [ ! -f "$1" ]; then
+        return
+    fi
+
+    # bash in Mac OS X doesn't support 'readarray', so using alternate way instead.
+    # readarray -t theArray < "$1"
+    # Any line that starts with '#' is ignored.
+    while IFS='' read -r line || [ -n "$line" ]; do
+        if [[ $line != "#"* ]]; then
+            theArray[${#theArray[@]}]=$line
+        fi
+    done < "$1"
+    echo ${theArray[@]}
 }
 
 generate_testhost()
@@ -499,6 +568,7 @@ usage()
     echo "generatelayoutonly - only pull down dependencies and build coreroot"
     echo "generatetesthostonly - only pull down dependencies and build coreroot and the CoreFX testhost"
     echo "skiprestorepackages - skip package restore"
+    echo "crossgen - Precompiles the framework managed assemblies in coreroot"
     echo "runtests - run tests after building them"
     echo "bindir - output directory (defaults to $__ProjectRoot/bin)"
     echo "msbuildonunsupportedplatform - build managed binaries even if distro is not officially supported."
@@ -630,6 +700,7 @@ __GenerateLayoutOnly=
 __GenerateTestHostOnly=
 __priority1=
 __BuildTestWrappersOnly=
+__DoCrossgen=0
 CORE_ROOT=
 
 while :; do
@@ -808,6 +879,10 @@ while :; do
             __SkipRestorePackages=1
             ;;
 
+        crossgen)
+            __DoCrossgen=1
+            ;;
+
         bindir)
             if [ -n "$2" ]; then
                 __RootBinDir="$2"

dependencies.props

@@ -26,13 +26,13 @@
 
   <!-- Source of truth for dependency tooling: the commit hash of the dotnet/versions master branch as of the last auto-upgrade. -->
   <PropertyGroup>
-    <CoreClrCurrentRef>effbce0711085d54a498120d91e0bddbb9f8d8ad</CoreClrCurrentRef>
-    <BuildToolsCurrentRef>79eb261c83005ea1c93c7861d374d1dd16a07353</BuildToolsCurrentRef>
+    <CoreClrCurrentRef>66f3bb05de8ac6063fbedff5816ef25cc6ecfa11</CoreClrCurrentRef>
+    <BuildToolsCurrentRef>66f3bb05de8ac6063fbedff5816ef25cc6ecfa11</BuildToolsCurrentRef>
   </PropertyGroup>
 
   <!-- Tests/infrastructure dependency versions. -->
   <PropertyGroup>
-    <MicrosoftNETCoreRuntimeCoreCLRPackageVersion>3.0.0-preview4-27603-71</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
+    <MicrosoftNETCoreRuntimeCoreCLRPackageVersion>3.0.0-preview5-27606-71</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
     <XunitPackageVersion>2.4.1</XunitPackageVersion>
     <XunitPerformanceApiPackageVersion>1.0.0-beta-build0015</XunitPerformanceApiPackageVersion>
     <MicrosoftDiagnosticsTracingTraceEventPackageVersion>2.0.36</MicrosoftDiagnosticsTracingTraceEventPackageVersion>

eng/Version.Details.xml

@@ -3,25 +3,25 @@
   <ProductDependencies>
   </ProductDependencies>
   <ToolsetDependencies>
-    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="1.0.0-beta.19202.13">
+    <Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="1.0.0-beta.19205.6">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>764f362c8e92e41905fe5f6c782ab9980c86c909</Sha>
+      <Sha>4217db4a23ffd15abb3771d635b66162994fb9e4</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="2.0.0-beta.19202.13">
+    <Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="2.0.0-beta.19205.6">
       <Uri>https://github.com/dotnet/arcade</Uri>
-      <Sha>764f362c8e92e41905fe5f6c782ab9980c86c909</Sha>
+      <Sha>4217db4a23ffd15abb3771d635b66162994fb9e4</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview4.19201.9">
+    <Dependency Name="Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview5.19205.9">
       <Uri>https://github.com/dotnet/corefx</Uri>
-      <Sha>85cec01822bc70fbc45a25001997b0c1e71b1d22</Sha>
+      <Sha>35249a0072b41a89ce1542deacb2611c2393dec0</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.NETCore.Platforms" Version="3.0.0-preview4.19201.9">
+    <Dependency Name="Microsoft.NETCore.Platforms" Version="3.0.0-preview5.19205.9">
       <Uri>https://github.com/dotnet/corefx</Uri>
-      <Sha>85cec01822bc70fbc45a25001997b0c1e71b1d22</Sha>
+      <Sha>35249a0072b41a89ce1542deacb2611c2393dec0</Sha>
     </Dependency>
-    <Dependency Name="Microsoft.NETCore.App" Version="3.0.0-preview4-27602-14">
+    <Dependency Name="Microsoft.NETCore.App" Version="3.0.0-preview4-27604-13">
       <Uri>https://github.com/dotnet/core-setup</Uri>
-      <Sha>8fa16d3d543ffa2babd5c8a45b5f29ec2cbdf55f</Sha>
+      <Sha>209213958004859a79ca586e7b278b5c7fe1f4b0</Sha>
     </Dependency>
     <Dependency Name="optimization.IBC.CoreCLR" Version="99.99.99-master-20190313.3">
       <Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-optimization</Uri>

eng/Versions.props

@@ -8,9 +8,9 @@
     <UsingToolXliff>false</UsingToolXliff>
     <UsingToolNetFrameworkReferenceAssemblies>true</UsingToolNetFrameworkReferenceAssemblies>
     <MicrosoftNetFrameworkReferenceAssembliesVersion>1.0.0-alpha-004</MicrosoftNetFrameworkReferenceAssembliesVersion>
-    <MicrosoftPrivateCoreFxNETCoreAppVersion>4.6.0-preview4.19201.9</MicrosoftPrivateCoreFxNETCoreAppVersion>
-    <MicrosoftNETCorePlatformsVersion>3.0.0-preview4.19201.9</MicrosoftNETCorePlatformsVersion>
-    <MicrosoftNETCoreAppVersion>3.0.0-preview4-27602-14</MicrosoftNETCoreAppVersion>
+    <MicrosoftPrivateCoreFxNETCoreAppVersion>4.6.0-preview5.19205.9</MicrosoftPrivateCoreFxNETCoreAppVersion>
+    <MicrosoftNETCorePlatformsVersion>3.0.0-preview5.19205.9</MicrosoftNETCorePlatformsVersion>
+    <MicrosoftNETCoreAppVersion>3.0.0-preview4-27604-13</MicrosoftNETCoreAppVersion>
     <optimizationIBCCoreCLRVersion>99.99.99-master-20190313.3</optimizationIBCCoreCLRVersion>
     <optimizationPGOCoreCLRVersion>99.99.99-master-20190313.3</optimizationPGOCoreCLRVersion>
   </PropertyGroup>

eng/common/PublishToPackageFeed.proj

@@ -1,11 +1,13 @@
-<!--
-  This MSBuild file is intended to be used as the body of the default 
-  publishing release pipeline. The release pipeline will use this file
-  to invoke the PushToStaticFeed task that will read the build asset
-  manifest and publish the assets described in the manifest to
-  informed target feeds.
--->
+<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
 <Project Sdk="Microsoft.NET.Sdk">
+  <!--
+    This MSBuild file is intended to be used as the body of the default 
+    publishing release pipeline. The release pipeline will use this file
+    to invoke the PushToStaticFeed task that will read the build asset
+    manifest and publish the assets described in the manifest to
+    informed target feeds.
+  -->
+
   <PropertyGroup>
     <TargetFramework>netcoreapp2.1</TargetFramework>
   </PropertyGroup>
@@ -41,6 +43,16 @@
     <PropertyGroup>
       <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == '.NETCORE'">https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json</TargetStaticFeed>
       <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == '.NETCOREVALIDATION'">https://dotnetfeed.blob.core.windows.net/arcade-validation/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETCORE'">https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETCORETOOLING'">https://dotnetfeed.blob.core.windows.net/aspnet-aspnetcore-tooling/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ENTITYFRAMEWORKCORE'">https://dotnetfeed.blob.core.windows.net/aspnet-entityframeworkcore/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'ASPNETEXTENSIONS'">https://dotnetfeed.blob.core.windows.net/aspnet-extensions/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'CORECLR'">https://dotnetfeed.blob.core.windows.net/dotnet-coreclr/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'CORESDK'">https://dotnetfeed.blob.core.windows.net/dotnet-sdk/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'TOOLSINTERNAL'">https://dotnetfeed.blob.core.windows.net/dotnet-tools-internal/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'TOOLSET'">https://dotnetfeed.blob.core.windows.net/dotnet-toolset/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'WINDOWSDESKTOP'">https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/index.json</TargetStaticFeed>
+      <TargetStaticFeed Condition="'$(ArtifactsCategory.ToUpper())' == 'NUGETCLIENT'">https://dotnetfeed.blob.core.windows.net/nuget-nugetclient/index.json</TargetStaticFeed>
     </PropertyGroup>
 
     <Error 

eng/common/PublishToSymbolServers.proj

@@ -0,0 +1,74 @@
+<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
+<Project Sdk="Microsoft.NET.Sdk">
+  <!--
+    This MSBuild file is intended to be used as the body of the default 
+    publishing release pipeline. The release pipeline will use this file
+    to invoke the PublishSymbols tasks to publish symbols to MSDL and SymWeb.
+  
+    Parameters:
+  
+      - PDBArtifactsDirectory   : Full path to directory containing PDB files to be published.
+      - BlobBasePath            : Full path containing *.symbols.nupkg packages to be published.
+      - DotNetSymbolServerTokenMsdl   : PAT to access MSDL.
+      - DotNetSymbolServerTokenSymWeb : PAT to access SymWeb.
+      - DotNetSymbolExpirationInDays  : Expiration days for published packages. Default is 3650.
+  -->
+
+  <PropertyGroup>
+    <TargetFramework>netcoreapp2.1</TargetFramework>
+  </PropertyGroup>
+
+  <Import Project="$(NuGetPackageRoot)microsoft.symboluploader.build.task\$(SymbolUploaderVersion)\build\PublishSymbols.targets" />
+    
+  <Target Name="PublishSymbols">
+    <ItemGroup>
+      <FilesToPublishToSymbolServer Include="$(PDBArtifactsDirectory)\*.pdb"/>
+      <PackagesToPublishToSymbolServer Include="$(BlobBasePath)\*.symbols.nupkg"/>
+    </ItemGroup>
+
+    <PropertyGroup>
+      <DotNetSymbolExpirationInDays Condition="'$(DotNetSymbolExpirationInDays)' == ''">3650</DotNetSymbolExpirationInDays>
+      <PublishToSymbolServer>true</PublishToSymbolServer>
+      <PublishToSymbolServer Condition="'@(FilesToPublishToSymbolServer)' == '' and '@(PackagesToPublishToSymbolServer)' == ''">false</PublishToSymbolServer>
+    </PropertyGroup>
+
+    <Message
+      Importance="High"
+      Text="No symbol package(s) were found to publish." 
+      Condition="$(PublishToSymbolServer) == false" />
+    
+    <!-- Symbol Uploader: MSDL -->
+    <Message Importance="High" Text="Publishing symbol packages to MSDL ..." Condition="$(PublishToSymbolServer)" />
+    <PublishSymbols PackagesToPublish="@(PackagesToPublishToSymbolServer)"
+                    FilesToPublish="@(FilesToPublishToSymbolServer)"
+                    PersonalAccessToken="$(DotNetSymbolServerTokenMsdl)"
+                    SymbolServerPath="https://microsoftpublicsymbols.artifacts.visualstudio.com/DefaultCollection"
+                    ExpirationInDays="$(DotNetSymbolExpirationInDays)"
+                    VerboseLogging="true"
+                    DryRun="false"
+                    ConvertPortablePdbsToWindowsPdbs="false"
+                    PdbConversionTreatAsWarning=""
+                    Condition="$(PublishToSymbolServer)"/>
+
+    <!-- 
+      Symbol Uploader: SymWeb 
+      Watson, VS insertion testings and the typical internal dev usage require SymWeb.
+      Currently we need to call the task twice (https://github.com/dotnet/core-eng/issues/3489).
+    -->
+    <Message Importance="High" Text="Publishing symbol packages to SymWeb ..." Condition="$(PublishToSymbolServer)" />
+    <PublishSymbols PackagesToPublish="@(PackagesToPublishToSymbolServer)"
+                    FilesToPublish="@(FilesToPublishToSymbolServer)"
+                    PersonalAccessToken="$(DotNetSymbolServerTokenSymWeb)"
+                    SymbolServerPath="https://microsoft.artifacts.visualstudio.com/DefaultCollection"
+                    ExpirationInDays="$(DotNetSymbolExpirationInDays)"
+                    VerboseLogging="true"
+                    DryRun="false"
+                    ConvertPortablePdbsToWindowsPdbs="false"
+                    PdbConversionTreatAsWarning=""
+                    Condition="$(PublishToSymbolServer)"/>
+  </Target>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.SymbolUploader.Build.Task" Version="$(SymbolUploaderVersion)" />
+  </ItemGroup>
+</Project>

eng/common/SigningValidation.proj

@@ -1,18 +1,20 @@
-<!--
-  This MSBuild file is intended to be used as the body of the default 
-  publishing release pipeline. The release pipeline will use this file
-  to invoke the the SignCheck tool to validate that packages about to
-  be published are correctly signed.
+<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
+<Project Sdk="Microsoft.NET.Sdk">
+  <!--
+    This MSBuild file is intended to be used as the body of the default 
+    publishing release pipeline. The release pipeline will use this file
+    to invoke the the SignCheck tool to validate that packages about to
+    be published are correctly signed.
   
-  Parameters:
+    Parameters:
   
-    - PackageBasePath   : Directory containing all files that need to be validated.
-    - SignCheckVersion  : Version of SignCheck package to be used.
-    - SignValidationExclusionList   : ItemGroup containing exclusion list to be forwarded to SignCheck.
-    - EnableJarSigningCheck    : Whether .jar files should be validated.
-    - EnableStrongNameCheck    : Whether strong name check should be performed.
--->
-<Project Sdk="Microsoft.NET.Sdk">
+      - PackageBasePath   : Directory containing all files that need to be validated.
+      - SignCheckVersion  : Version of SignCheck package to be used.
+      - SignValidationExclusionList   : ItemGroup containing exclusion list to be forwarded to SignCheck.
+      - EnableJarSigningCheck    : Whether .jar files should be validated.
+      - EnableStrongNameCheck    : Whether strong name check should be performed.
+  -->
+
   <PropertyGroup>
     <TargetFramework>netcoreapp2.1</TargetFramework>
   </PropertyGroup>