COMException crash in pdb2pdb after linking System.Private.CoreLib

[eerhardt]

With dotnet/runtime#37615, we are introducing some ILLink.Substitutions.xml files that direct the linker to trim branches in IL that will never be called. For example, when building for x64, any if (AdvSimd.IsSupported) branches can be trimmed away, since ARM intrinsics will never be supported for x64.

However, Pdb2Pdb is failing in this PR:

  System.Private.CoreLib -> F:\workspace\_work\1\s\artifacts\bin\coreclr\Windows_NT.x64.Checked\IL\System.Private.CoreLib.dll
  PDB0004: token 0x06001C30: Method containing local variables has no local signature
  PDB0004: token 0x06001C30: Method containing local variables has no local signature
  PDB0004: token 0x06001C31: Method containing local variables has no local signature
  PDB0004: token 0x06001C31: Method containing local variables has no local signature
  Error HRESULT E_FAIL has been returned from a call to a COM component.
F:\workspace\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20280.1\tools\SymStore.targets(70,5): error MSB3073: The command ""F:\workspace\_work\1\s\.packages\microsoft.diasymreader.pdb2pdb\1.1.0-beta2-19575-01\tools\Pdb2Pdb.exe" "F:\workspace\_work\1\s\artifacts\bin\coreclr\Windows_NT.x64.Checked\IL\System.Private.CoreLib.dll" /out "F:\workspace\_work\1\s\artifacts\SymStore\Checked\System.Private.CoreLib\netcoreapp2.1\x64\System.Private.CoreLib.pdb" /srcsvrvar SRC_INDEX=public" exited with code 2. [F:\workspace\_work\1\s\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
##[error].packages\microsoft.dotnet.arcade.sdk\5.0.0-beta.20280.1\tools\SymStore.targets(70,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Build) The command ""F:\workspace\_work\1\s\.packages\microsoft.diasymreader.pdb2pdb\1.1.0-beta2-19575-01\tools\Pdb2Pdb.exe" "F:\workspace\_work\1\s\artifacts\bin\coreclr\Windows_NT.x64.Checked\IL\System.Private.CoreLib.dll" /out "F:\workspace\_work\1\s\artifacts\SymStore\Checked\System.Private.CoreLib\netcoreapp2.1\x64\System.Private.CoreLib.pdb" /srcsvrvar SRC_INDEX=public" exited with code 2.

Debugging into Pdb2Pdb.exe, I see the first problem is that we are trimming away locals from this method:

https://github.com/dotnet/runtime/blob/bd6cbe3642f51d70839912a6a666e5de747ad581/src/libraries/System.Private.CoreLib/src/System/Numerics/BitOperations.cs#L233-L251

The if (AdvSimd.Arm64.IsSupported) branch is getting trimmed, but it looks like the local variables are not getting removed from the portable .pdb. Here is the code after trimming:

        public static int PopCount(uint value)
        {
            if (Popcnt.IsSupported)
            {
                return (int)Popcnt.PopCount(value);
            }
            !AdvSimd.Arm64.IsSupported;
            return BitOperations.<PopCount>g__SoftwareFallback|9_0(value);
        }

That is issue dotnet/linker#1260.

After that, Pdb2Pdb is trying to work on this method:

https://github.com/dotnet/runtime/blob/bd6cbe3642f51d70839912a6a666e5de747ad581/src/libraries/System.Private.CoreLib/src/System/Globalization/CharUnicodeInfo.cs#L231-L248

But it is throwing a COMException when calling OpenScope:

>	Microsoft.DiaSymReader.Native.amd64.dll!SymWriter::OpenScope(unsigned int,unsigned int *)	Unknown
 	[Managed to Native Transition]	
 	Microsoft.DiaSymReader.dll!Microsoft.DiaSymReader.SymUnmanagedWriterImpl.OpenScope(int startOffset) Line 261	C#
 	Microsoft.DiaSymReader.Converter.dll!Microsoft.DiaSymReader.Tools.PdbConverterPortableToWindows.Convert.__OpenScope|6_6(int startOffset, int endOffset, ref Microsoft.DiaSymReader.Tools.PdbConverterPortableToWindows.<>c__DisplayClass6_0 value, ref Microsoft.DiaSymReader.Tools.PdbConverterPortableToWindows.<>c__DisplayClass6_1 value) Line 295	C#
 	Microsoft.DiaSymReader.Converter.dll!Microsoft.DiaSymReader.Tools.PdbConverterPortableToWindows.Convert(System.Reflection.PortableExecutable.PEReader peReader, System.Reflection.Metadata.MetadataReader pdbReader, Microsoft.DiaSymReader.SymUnmanagedWriter pdbWriter, Microsoft.DiaSymReader.Tools.PortablePdbConversionOptions options) Line 322	C#
 	Microsoft.DiaSymReader.Converter.dll!Microsoft.DiaSymReader.Tools.PdbConverter.ConvertPortableToWindows(System.Reflection.PortableExecutable.PEReader peReader, System.Reflection.Metadata.MetadataReader pdbReader, Microsoft.DiaSymReader.SymUnmanagedWriter pdbWriter, Microsoft.DiaSymReader.Tools.PortablePdbConversionOptions options) Line 189	C#
 	Microsoft.DiaSymReader.Converter.dll!Microsoft.DiaSymReader.Tools.PdbConverter.ConvertPortableToWindows(System.Reflection.PortableExecutable.PEReader peReader, System.Reflection.Metadata.MetadataReader pdbReader, System.IO.Stream targetPdbStream, Microsoft.DiaSymReader.Tools.PortablePdbConversionOptions options) Line 171	C#
 	Pdb2Pdb.exe!Microsoft.DiaSymReader.Tools.Pdb2Pdb.Convert(Microsoft.DiaSymReader.Tools.Pdb2Pdb.Args args) Line 268	C#
 	Pdb2Pdb.exe!Microsoft.DiaSymReader.Tools.Pdb2Pdb.Main(string[] args) Line 59	C#

From doing a little debugging, it appears that the top scope is getting closed in Microsoft.DiaSymReader.Native.amd64.dll, and the native code is returning E_FAIL when trying to open the scope again. It seems that Pdb2Pdb is getting in some invalid state right before this.

Here is the .dll and .pdb from the build that is causing the crash:

System.Private.CoreLib.zip

Unzip that, and run the command:

Pdb2Pdb.exe "C:\path\to\System.Private.CoreLib.dll" /out "C:\temp\System.Private.CoreLib.pdb" /srcsvrvar SRC_INDEX=public

cc @tmat

[vitek-karas]

The underlying issue is twofold. The input PDB is corrupted - I think it has the length of one of the local scope as -1. And on top of that System.Reflection.Metadata PDB reader has an inconsistency in it:
When accessing the GetEndOffset on the wrong section it returns a normal value (but it returns value as if length is -1) which is wrong - it reports the scope as starting at 41 and ending at 40.

When accessing the same scope via GetLength the reader will throw BadImageFormatException. Would be nice if the reader threw in both cases.

Not sure if this is an issue for this repo or for the runtime repo though.