Improve the UX around source build only dependencies in Version.Details.xml

[MichaelSimons]

To support source build, there are scenarios in which dependencies will be defined in the Version.Details.xml files that won't have a darc subscription. These entries can cause confusion when intertwined with normal darc dependencies. We should consider what patterns can help improve this situation. For example maybe there needs to be another dependency type added besides product and toolset dependencies to separate out these source-build only dependencies. In these cases, perhaps less metadata is needed. The version and sha are unused and often get out of sync with the version declared in versions.props.

[MichaelSimons]

cc @mmitche

[mmitche]

+10000 /cc @premun

There was some chatting about this last week. Where maybe we want to have 3 types of V.D.xml deps. Product, Toolset, and SourceBuiltProduct.

I'm not totally convinced of this yet. It's not super elegant, and so it says to me that maybe we're trying to solve the wrong problem. Or we don't understand the problem. Maybe the right answer is just that there should be no difference in deps.

[ViktorHofer]

A Microsoft build intentionally uses version X of i.e. System.Text.Json for VS compatibility vs source-build which upgrade to PSB to avoid the need for an SBRP package or when SBRPs simply can't be used (i.e. execution).

An example for Version.Details entries that only exist for PSB infra is https://github.com/dotnet/templating/blob/main/eng/Version.Details.xml#L15-L55. Those entries are useful for non-source-build as well but there is no incentive to define them as the Versions.props entries get manually updated. I think the reason for the manual update was that we don't want to cross release boundaries with Maestro?

[premun]

Why does it need to be in Version.Details.xml if it won't have a subscription? Maybe we keep that somewhere else so that Maestro does not need to learn about this only to ignore it? Or do I misunderstand it?

[mmitche]

Why does it need to be in Version.Details.xml if it won't have a subscription? Maybe we keep that somewhere else so that Maestro does not need to learn about this only to ignore it? Or do I misunderstand it?

Well, that whole premise of the V.D.xml file is that is defining the dependencies of the repo. This is the case for SB too. It's just that the set of dependencies is different between Microsoft's build and the distro partner build.

It's entirely possible to have dependencies that have no, or infrequent updates. Or, similarly, when we want to freeze deps after branching because we don't need them updated. For instance, we don't maintain the entire subscription structure of .NET 10 when we branch for a release. It's just not necessary and too unwieldy. We don't need runtime-assets flowing into runtime.

We don't special case SB's deps in some other place anymore because we want the SB infra to use the same logic as MSFTs build. If it's not in V.D.xml, it doesn't get updated during the VMR build.

A Microsoft build intentionally uses version X of i.e. System.Text.Json for VS compatibility vs source-build which upgrade to PSB to avoid the need for an SBRP package or when SBRPs simply can't be used (i.e. execution).

An example for Version.Details entries that only exist for PSB infra is https://github.com/dotnet/templating/blob/main/eng/Version.Details.xml#L15-L55. Those entries are useful for non-source-build as well but there is no incentive to define them as the Versions.props entries get manually updated. I think the reason for the manual update was that we don't want to cross release boundaries with Maestro?

I think in this case, we didn't want subscription updates because they don't really affect the product and we didn't want to add another coherency path. They can be updated now every build, if we wanted (aligning SB and MSFT) since there is no coherency complexity. See dotnet/templating#9060 (comment)

[ViktorHofer]

@MichaelSimons isn't this a source-build specific issue and should stay in dotnet/source-build?