Merge pull request #17347 from weshaggard/MergeMaster21

[release/2.1] Merge master to release/2.1

Author: weshaggard

.gitignore

@@ -19,8 +19,6 @@ syntax: glob
 [Dd]ebugPublic/
 [Rr]elease/
 [Rr]eleases/
-x64/
-x86/
 build/
 bld/
 [Bb]in/
@@ -54,7 +52,6 @@ dlldata.c
 
 *_i.c
 *_p.c
-*_i.h
 *.ilk
 *.meta
 *.obj
@@ -239,6 +236,7 @@ $RECYCLE.BIN/
 
 # Common binary extensions on Windows
 *.exe
+*.exe.stackdump
 *.dll
 *.lib
 

BuildToolsVersion.txt

@@ -1 +1 @@
-2.1.0-preview2-02621-01
+2.1.0-preview3-02629-02

CODE_OWNERS.TXT

@@ -15,8 +15,8 @@ N: Mike McLaughlin
 G: mikem8361
 D: Debugging APIs and implementation (anything under the debug directory), sos
 
-N: Kshama Pawar
-G: kspawa
+N: Noah Falk, David Mason
+G: noahfalk, davmason
 D: Profiling APIs and implementation
 
 N: Gaurav Khanna
@@ -28,4 +28,4 @@ G: ericstj
 D: Nuget packaging implementation
 
 This list is incomplete, if you need to find other owners work with Jan Kotas
-to get them filled in.
+to get them filled in.

Documentation/building/debugging-instructions.md

@@ -52,6 +52,8 @@ You can combine steps 4-8 and pass everything on the lldb command line:
 
 `lldb-3.9 -o "plugin load libsosplugin.so" -o "process launch -s" -o "process handle -s false SIGUSR1 SIGUSR2" -o "breakpoint set -n LoadLibraryExW" corerun HelloWorld.exe linux`
 
+For .NET Core version 1.x and 2.0.x, libsosplugin.so is built for and will only work with version 3.6 of lldb. For .NET Core 2.1, the plugin is built for 3.9 lldb and will work with 3.8 and 3.9 lldb.
+
 ### SOS commands ###
 
 This is the full list of commands currently supported by SOS. lldb is case-sensitive unlike windbg.
@@ -164,6 +166,13 @@ lldb should start debugging successfully at this point. You should see stacktrac
 ```
 lldb-3.9 -O "settings set target.exec-search-paths /home/parallels/Downloads/System.Drawing.Common.Tests/home/helixbot/dotnetbuild/work/2a74cf82-3018-4e08-9e9a-744bb492869e/Payload/shared/Microsoft.NETCore.App/9.9.9/" -o "plugin load /home/parallels/Downloads/System.Drawing.Common.Tests/home/helixbot/dotnetbuild/work/2a74cf82-3018-4e08-9e9a-744bb492869e/Payload/shared/Microsoft.NETCore.App/9.9.9/libsosplugin.so" --core /home/parallels/Downloads/System.Drawing.Common.Tests/home/helixbot/dotnetbuild/work/2a74cf82-3018-4e08-9e9a-744bb492869e/Work/f6414a62-9b41-4144-baed-756321e3e075/Unzip/core /home/parallels/Downloads/System.Drawing.Common.Tests/home/helixbot/dotnetbuild/work/2a74cf82-3018-4e08-9e9a-744bb492869e/Payload/shared/Microsoft.NETCore.App/9.9.9/dotnet
 ```
+Disabling Managed Attach/Debugging
+==================================
+
+The "COMPlus_EnableDiagnostics" environment variable can be used to disable managed debugging. This prevents the various OS artifacts used for debugging like the named pipes and semaphores on Linux/MacOS and shared memory on Windows from being created.
+
+    export COMPlus_EnableDiagnostics=0
+
 
 Using Visual Studio Code
 ========================

Documentation/building/viewing-jit-dumps.md

@@ -8,8 +8,9 @@ To make sense of the results, it is recommended you also read the [Reading a Jit
 
 The first thing we want to do is setup the .NET Core app we want to dump. Here are the steps to do this, if you don't have one ready:
 
-* Perform a debug build of the CoreCLR repo. You don't need to build tests, so you can pass `skiptests` to the build command to make it faster.
-* Install the [.NET CLI 2.0 preview](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md), which we'll use to compile/publish our app.
+* Perform a release build of the CoreCLR repo by passing `release` to the build command. You don't need to build tests, so you can pass `skiptests` to the build command to make it faster. Note: the release build can be skipped, but in order to see optimized in the core library it is needed.
+* Perform a debug build of the CoreCLR repo. Tests aren't needed as in the release build, so you can pass `skiptests` to the build command. Note: the debug build is necessary, so that the JIT recognizes the configuration knobs.
+* Install the [.NET CLI 2.1 preview](https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/dogfooding.md), which we'll use to compile/publish our app.
 * `cd` to where you want your app to be placed, and run `dotnet new console`.
 * Modify your `csproj` file so that it contains a RID (runtime ID) corresponding to the OS you're using in the `<RuntimeIdentifier>` tag. For example, for Windows 10 x64 machine, the project file is:
 
@@ -61,10 +62,12 @@ The first thing we want to do is setup the .NET Core app we want to dump. Here a
 
     ```shell
     # Windows
-    robocopy /e <coreclr path>\bin\Product\Windows_NT.<arch>.Debug <app root>\bin\Release\netcoreapp2.0\<rid>\publish > NUL
+    robocopy /e <coreclr path>\bin\Product\Windows_NT.<arch>.Release <app root>\bin\Release\netcoreapp2.0\<rid>\publish > NUL
+    copy <coreclr path>\bin\Product\Windows_NT.<arch>.Debug\clrjit.dll <app root>\bin\Release\netcoreapp2.0\<rid>\publish > NUL
 
     # Unix
-    cp -rT <coreclr path>/bin/Product/<OS>.<arch>.Debug <app root>/bin/Release/netcoreapp2.0/<rid>/publish
+    cp -rT <coreclr path>/bin/Product/<OS>.<arch>.Release <app root>/bin/Release/netcoreapp2.0/<rid>/publish
+    cp <coreclr path>/bin/Product/<OS>.<arch>.Debug/libclrjit.so <app root>/bin/Release/netcoreapp2.0/<rid>/publish
     ```
 
 * Set the configuration knobs you need (see below) and run your published app. The info you want should be dumped to stdout.

Documentation/workflow/OfficalAndDailyBuilds.md

@@ -16,6 +16,13 @@ repositories.  You can see what is available from
  * CoreCLR at <https://dotnet.myget.org/feed/dotnet-core/package/nuget/Microsoft.NETCore.Runtime.CoreCLR>
  * NETCore.App at <https://dotnet.myget.org/feed/dotnet-core/package/nuget/Microsoft.NETCore.App>
 
+These builds have a version number that follows the the versioning scheme described below (month number/day of month), 
+but they also will have a component that indicate which Git Branch the are working from (note these names were
+correct as of 1/2018 and may change but the concept of a suffix that designates the branch is likely to persist)
+
+ * preview1 - are daily builds from the 'release/\*' branch where \* is the next official version to be released
+ * preview2 - are daily builds from the 'master' branch (where active work happens first (typically))
+
 Thus if your goal is just to get the latest bug fixes and features, you don't need to build CoreCLR yourself you 
 can simply add <https://dotnet.myget.org/F/dotnet-core/api/v3/index.json> to your Nuget Feed list and set the
 `RuntimeFrameworkVersion` in your project file to a `Microsoft.NETCore.App` version. You need to restore

ILAsmVersion.txt

@@ -1 +1 @@
-2.1.0-preview2-26324-04
+2.1.0-preview3-26329-01

build-test.cmd

@@ -51,6 +51,12 @@ set __ZipTests=
 set __TargetsWindows=1
 set __DoCrossgen=
 
+@REM CMD has a nasty habit of eating "=" on the argument list, so passing:
+@REM    -priority=1
+@REM appears to CMD parsing as "-priority 1". Handle -priority specially to avoid problems,
+@REM and allow the "-priority=1" syntax.
+set __Priority=0
+
 :Arg_Loop
 if "%1" == "" goto ArgsDone
 
@@ -75,6 +81,7 @@ if /i "%1" == "crossgen"              (set __DoCrossgen=1&set processedArgs=!pro
 if /i "%1" == "runtimeid"             (set __RuntimeId=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
 if /i "%1" == "targetsNonWindows"     (set __TargetsWindows=0&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
 if /i "%1" == "Exclude"               (set __Exclude=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
+if /i "%1" == "-priority"             (set __Priority=%2&shift&set processedArgs=!processedArgs! %1=%2&shift&goto Arg_Loop)
 
 if [!processedArgs!]==[] (
   set __UnprocessedBuildArgs=%__args%
@@ -87,6 +94,9 @@ if [!processedArgs!]==[] (
 
 :ArgsDone
 
+@REM Special handling for -priority=N argument.
+if %__Priority% GTR 0 (set "__UnprocessedBuildArgs=!__UnprocessedBuildArgs! -priority=%__Priority%")
+
 if defined __BuildAgainstPackagesArg (
     if not defined __RuntimeID (
         echo %__MsgPrefix%Error: When building against packages, you must supply a target Runtime ID.
@@ -267,22 +277,43 @@ if not defined VSINSTALLDIR (
     echo %__MsgPrefix%Error: build-test.cmd should be run from a Visual Studio Command Prompt.  Please see https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/developer-guide.md for build instructions.
     exit /b 1
 )
-
+set __AppendToLog=false
 set __BuildLogRootName=Tests_Managed
 set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
 set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn
 set __BuildErr=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.err
-set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%"
-set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
-set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
 
-call "%__ProjectDir%\run.cmd" build -Project=%__ProjectDir%\tests\build.proj -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %__unprocessedBuildArgs%
-if errorlevel 1 (
-    echo %__MsgPrefix%Error: build failed. Refer to the build log files for details:
-    echo     %__BuildLog%
-    echo     %__BuildWrn%
-    echo     %__BuildErr%
-    exit /b 1
+REM Execute msbuild test build in stages - workaround for excessive data retention in MSBuild ConfigCache
+REM See https://github.com/Microsoft/msbuild/issues/2993
+
+set __SkipPackageRestore=false
+set __SkipTargetingPackBuild=false
+set __BuildLoopCount=2
+set __TestGroupToBuild=1
+
+if %__Priority% GTR 0 (set __BuildLoopCount=16&set __TestGroupToBuild=2)
+echo %__MsgPrefix%Building tests group %__TestGroupToBuild% with %__BuildLoopCount% subgroups
+
+for /l %%G in (1, 1, %__BuildLoopCount%) do (
+
+    set __msbuildLog=/flp:Verbosity=normal;LogFile="%__BuildLog%";Append=!__AppendToLog!
+    set __msbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%";Append=!__AppendToLog!
+    set __msbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%";Append=!__AppendToLog!
+
+    set TestBuildSlice=%%G
+    call "%__ProjectDir%\run.cmd" build -Project=%__ProjectDir%\tests\build.proj -MsBuildLog=!__msbuildLog! -MsBuildWrn=!__msbuildWrn! -MsBuildErr=!__msbuildErr! %__RunArgs% %__BuildAgainstPackagesArg% %__unprocessedBuildArgs%
+
+    if errorlevel 1 (
+        echo %__MsgPrefix%Error: build failed. Refer to the build log files for details:
+        echo     %__BuildLog%
+        echo     %__BuildWrn%
+        echo     %__BuildErr%
+        exit /b 1
+    )
+
+    set __SkipPackageRestore=true
+    set __SkipTargetingPackBuild=true
+    set __AppendToLog=true
 )
 
 REM Prepare the Test Drop

build-test.sh

@@ -338,7 +338,7 @@ usage()
     echo "BuildType can be: debug, checked, release"
     echo "coverage - optional argument to enable code coverage build (currently supported only for Linux and OSX)."
     echo "ninja - target ninja instead of GNU make"
-    echo "clangx.y - optional argument to build using clang version x.y."
+    echo "clangx.y - optional argument to build using clang version x.y - supported version 3.5 - 6.0"
     echo "cross - optional argument to signify cross compilation,"
     echo "      - will use ROOTFS_DIR environment variable if set."
     echo "crosscomponent - optional argument to build cross-architecture component,"
@@ -543,31 +543,46 @@ while :; do
         __VerboseBuild=1
         ;;
 
-        clang3.5)
+        clang3.5|-clang3.5)
             __ClangMajorVersion=3
             __ClangMinorVersion=5
             ;;
 
-        clang3.6)
+        clang3.6|-clang3.6)
             __ClangMajorVersion=3
             __ClangMinorVersion=6
             ;;
 
-        clang3.7)
+        clang3.7|-clang3.7)
             __ClangMajorVersion=3
             __ClangMinorVersion=7
             ;;
 
-        clang3.8)
+        clang3.8|-clang3.8)
             __ClangMajorVersion=3
             __ClangMinorVersion=8
             ;;
 
-        clang3.9)
+        clang3.9|-clang3.9)
             __ClangMajorVersion=3
             __ClangMinorVersion=9
             ;;
 
+        clang4.0|-clang4.0)
+            __ClangMajorVersion=4
+            __ClangMinorVersion=0
+            ;;
+
+        clang5.0|-clang5.0)
+            __ClangMajorVersion=5
+            __ClangMinorVersion=0
+            ;;
+
+        clang6.0|-clang6.0)
+            __ClangMajorVersion=6
+            __ClangMinorVersion=0
+            ;;
+
         ninja)
             __UseNinja=1
             ;;
@@ -686,9 +701,9 @@ export __CMakeBinDir="$__BinDir"
 
 if [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ]; then
 
-    echo "Cannot find build directory for the CoreCLR Product."
-    echo "Please make sure CoreCLR is built before building tests."
-    echo "Example use: './build.sh $__BuildArch $__BuildType'"
+    echo "Cannot find build directory for the CoreCLR Product or native tests."
+    echo "Please make sure CoreCLR and native tests are built before building managed tests."
+    echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
     exit 1
 fi
 

build.cmd

@@ -12,23 +12,6 @@ set __ThisScriptDir="%~dp0"
 call "%__ThisScriptDir%"\setup_vs_tools.cmd
 if NOT '%ERRORLEVEL%' == '0' exit /b 1
 
-REM Make the work-around to a bug in the microsoft.dotnet.buildtools.coreclr package until it is fixed.  
-reg query HKEY_CLASSES_ROOT\WOW6432Node\CLSID\{3BFCEA48-620F-4B6B-81F7-B9AF75454C7D}\InprocServer32 > NUL: 2>&1
-if NOT '%ERRORLEVEL%' == '0' (
-    echo.
-    echo.**********************************************************************************
-    echo.Error: We have detected that the msdia120.dll is not registered.   
-    echo.This is necessary for the build to complete without a Class_Not_Registered error.
-    echo.
-    echo.You can fix this by 
-    echo.  1. Launching the "Developer Command Prompt for VS2017" with Administrative privileges
-    echo.  2. Running  regsvr32.exe "%%VSINSTALLDIR%%\Common7\IDE\msdia120.dll"  
-    echo.
-    echo.This will only need to be done once for the lifetime of the machine.
-    echo.For more details see: https://github.com/dotnet/coreclr/issues/11305
-    exit /b 1
-)
-
 if defined VS150COMNTOOLS (
   set "__VSToolsRoot=%VS150COMNTOOLS%"
   set "__VCToolsRoot=%VS150COMNTOOLS%\..\..\VC\Auxiliary\Build"
@@ -39,6 +22,11 @@ if defined VS150COMNTOOLS (
   set __VSVersion=vs2015
 )
 
+:: Work around Jenkins CI + msbuild problem: Jenkins sometimes creates very large environment
+:: variables, and msbuild can't handle environment blocks with such large variables. So clear
+:: out the variables that might be too large.
+set ghprbCommentBody=
+
 :: Note that the msbuild project files (specifically, dir.proj) will use the following variables, if set:
 ::      __BuildArch         -- default: x64
 ::      __BuildType         -- default: Debug