Second source-only build of VMR introduces different reference packages

[mthalman]

There is something that is causing different reference packages to be added to the prereqs/packages/reference directory when a second source-only build occurs on the VMR. This doesn't happen on the first build, only on subsequent builds.

Repro Steps

1. Checkout the release/10.0.1xx-preview3 branch of the VMR
2. Make some source change in any repo (except the sdk repo) that will cause a build error.
3. Run ./prep-source-build.sh && ./build.sh --source-only --online. This will encounter the build error.
4. Undo your source change.
5. Run ./build.sh --source-only --online

This will cause the following error in the sdk repo:

    CSC : error CS0006: Metadata file '/repos/dotnet/src/sdk/artifacts/sb/package-cache/microsoft.aspnetcore.app.ref/8.0.0/analyzers/dotnet/cs/Microsoft.AspNetCore.App.Analyzers.dll' could not be found [/repos/dotnet/src/sdk/src/BuiltInTools/BrowserRefresh/Microsoft.AspNetCore.Watch.BrowserRefresh.csproj]
    CSC : error CS0006: Metadata file '/repos/dotnet/src/sdk/artifacts/sb/package-cache/microsoft.aspnetcore.app.ref/8.0.0/analyzers/dotnet/cs/Microsoft.AspNetCore.App.CodeFixes.dll' could not be found [/repos/dotnet/src/sdk/src/BuiltInTools/BrowserRefresh/Microsoft.AspNetCore.Watch.BrowserRefresh.csproj]
    CSC : error CS0006: Metadata file '/repos/dotnet/src/sdk/artifacts/sb/package-cache/microsoft.aspnetcore.app.ref/8.0.0/analyzers/dotnet/cs/Microsoft.AspNetCore.Components.Analyzers.dll' could not be found [/repos/dotnet/src/sdk/src/BuiltInTools/BrowserRefresh/Microsoft.AspNetCore.Watch.BrowserRefresh.csproj]
    CSC : error CS0006: Metadata file '/repos/dotnet/src/sdk/artifacts/sb/package-cache/microsoft.aspnetcore.app.ref/8.0.0/analyzers/dotnet/roslyn4.4/cs/Microsoft.Extensions.Logging.Generators.dll' could not be found [/repos/dotnet/src/sdk/src/BuiltInTools/BrowserRefresh/Microsoft.AspNetCore.Watch.BrowserRefresh.csproj]
    CSC : error CS0006: Metadata file '/repos/dotnet/src/sdk/artifacts/sb/package-cache/microsoft.aspnetcore.app.ref/8.0.0/analyzers/dotnet/roslyn4.4/cs/Microsoft.Extensions.Options.SourceGeneration.dll' could not be found [/repos/dotnet/src/sdk/src/BuiltInTools/BrowserRefresh/Microsoft.AspNetCore.Watch.BrowserRefresh.csproj]

This occurs because it's getting an incorrect version of the Microsoft.AspNetCore.App.Ref.8.0.0 package, one that doesn't contain the fix from dotnet/source-build-reference-packages#1199. Somehow this older version of Microsoft.AspNetCore.App.Ref.8.0.0 is being copied to the prereqs/packages/reference directory and picked up in the build. This is likely coming from the previously source built artifacts? It seems like when the build starts the prereqs/packages/reference directory is populated with the ref packages from PSB artifacts. Then SBRP builds and those get cleared out with the current ones. When the subsequent build occurs, it clears the current ones out and replaces them with the ones from the PSB artifacts. Then it skips SBRP build since that was already built and you're left with incorrect ref packages.

[MichaelSimons]

I hit this today so I am taking ownership to fix.

[MichaelSimons]

Here is a summary of what happens.

1. Start Build 1
2. init-source-only.props
   1. Extract PSB
   2. Move PSB/Ref packages to dotnet/prereqs/packages/reference
3. Build SBRP - special post build logic to copy resulting artifacts copies to dotnet/prereqs/packages/reference
4. Finish Build
5. Start Build 2
6. init-source-only.props runs the same targets again because it detects the output directories have changed. dotnet/prereqs/packages/reference is updated to the PSB version.
7. Finish Build but this time Build SBRP is skipped because the repo src hasn't changed. This results in the reference packages remaining as the PSB version.