scenario-tests isn't using the live local-built feeds when building source-only

[ViktorHofer]

See all the discussion in dotnet/sdk#46215 for more details. We should decide what to do here.

[mthalman]

As I described in dotnet/sdk#46215 (comment), I question whether using the live built feeds is the right thing to do for source build scenarios. Anything that gets loaded from a NuGet feed when using a source built SDK should be using Microsoft-build packages, since that is what users of a source built SDK would be getting. Using the live feed means we'd be loading source built packages. That is not validating what customers would be experiencing.

Related: dotnet/dotnet#1035

[tmds]

The tests fall into two categories (for a non-portable build):

• Those that only depend on the source-built SDK and which are expected to work without any additional sources.
• Those that depend on portable artifacts which are expected to come from additional sources.

The latter should be skippable. For example, there are no linux-ppc64le artifacts.

[ViktorHofer]

I wonder how not using the live local feeds would work when source-build starts having uniquely versioned builds.

[tmds]

I don't know what exactly "live local feeds" is composed of, is it all packages that are built, does it include prebuilts, or even CI NuGet feeds?

Anyway, there's are test that should only depend on the source-built SDK. One approach is to have these test check for themselves they are not using unexpected packages (cfr dotnet/dotnet#1035). Another is to configure the nuget feeds so these tests have no access to these packages.

I wonder how not using the live local feeds would work when source-build starts having uniquely versioned builds.

I think we should make it possible to skip tests that depend on packages that would not be available.

As a minimum we can run test that only depend on the SDK. If desired, this set could be made larger by including additional source-built packages or even pre-built packages.

I assume there's also going to be builds where the version numbers match up? For these builds, all tests can run and make use of external packages, for example to validate that a source-built SDK can leverage portable assets.

[ViktorHofer]

I don't know what exactly "live local feeds" is composed of, is it all packages that are built, does it include prebuilts, or even CI NuGet feeds?

When I say live local feeds, i mean the ones that point to the different inner repo artifacts that get live produced, i.e. <add key="source-built-runtime" value="artifacts\packages\Release\Shipping\runtime" />.

I think we should make it possible to skip tests that depend on packages that would not be available.

Agreed. That set of tests that depend on additional packages would need to be decorated (i.e. with an attribute) so that those have access to additional feeds while the SDK only tests shouldn't have access to any feeds.

I assume there's also going to be builds where the version numbers match up?

I don't think that would be possible as the unified-build and the source-build pipelines run in parallel. The source-build pipeline sometimes runs later but in the usual PR validation or CI builds, the assets from unified-build pipeline aren't yet available.

[tmds]

I don't think that would be possible as the unified-build and the source-build pipelines run in parallel. The source-build pipeline sometimes runs later but in the usual PR validation or CI builds, the assets from unified-build pipeline aren't yet available.

I see, this in a CI infrastructure issue.

For the version numbers themselves, I assume the ones in Microsoft.NETCoreSdk.BundledVersions.props are going to be the same when built from the same vmr commit?

[ViktorHofer]

@mmitche for that question. I think the idea is to depend less on the OfficialBuildId by doing stable branding more often (i.e. for previews) and for non stable branding, passing the official build id in when building source-only to match the unified-build pipeline one.

[mmitche]

Yes correct. The idea being that ideally, the source for our released products can be built multiple times by different parties and it will all work nicely together. Stable branding is one way we can achieve that. If we don't have that option yet, we can tell partners what the input was to our build (official build ID)

[MichaelSimons]

[Triage] As discussed, source-build should not be testing with any of the source-built live packages outside of what ships with the SDK as that is not how the product will be used by end customers. The tests should pull from nuget as well as the internal microsoft feeds. The changes to the OfficialBuildId will need to be reacted to but that is covered by #4855. Closing as not using the live source-built feeds is intentional.