Skip to content

Allow build targets to be invoked in batches #15277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 25, 2024
Merged

Allow build targets to be invoked in batches #15277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
391a83f
Allow build targets to be invoked in specific batches.
ViktorHofer Nov 25, 2024
1c3bbc4
Add docs
ViktorHofer Nov 25, 2024
45237fa
Fix target invocations
ViktorHofer Nov 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion Documentation/ArcadeSdk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,31 @@ optimizations by setting 'RestoreUsingNuGetTargets' to false.
</Project>
```

CoreFx does not use the default build projects in its repo - [example](https://github.com/dotnet/corefx/blob/66392f577c7852092f668876822b6385bcafbd44/eng/Build.props).
#### Example: batching projects together for more complex build ordering

Arcade builds all passed-in projects in parallel by default. While it's possible to set the `BuildInParallel` property or item metadata, more complex build order requirements might be necessary. When projects should be built in batches, the `BuildStep` item metadata can be used to express that.

Below, the build order is the following: `native1 -> native2 -> java & managed (parallel) -> installer1 & installer2 (parallel) -> cleanup -> optimization.
```xml
<!-- eng/Build.props -->
<Project>
<PropertyGroup>
<RestoreUsingNuGetTargets>false</RestoreUsingNuGetTargets>
</PropertyGroup>
<ItemGroup>
<ProjectToBuild Include="src\native1.proj" BuildStep="native" BuildInParallel="false" />
<ProjectToBuild Include="src\native2.proj" BuildStep="native" BuildInParallel="false" />
<ProjectToBuild Include="src\java.proj" BuildStep="managed" />
<ProjectToBuild Include="src\managed.proj" BuildStep="managed" />
<ProjectToBuild Include="src\installer1.proj" BuildStep="installers" />
<ProjectToBuild Include="src\installer2.proj" BuildStep="installers" />
<ProjectToBuild Include="src\cleanup.proj" BuildStep="finish" BuildInParallel="false" />
<ProjectToBuild Include="src\optimization.proj" BuildStep="finish" BuildInParallel="false" />
</ItemGroup>
</Project>
```

Runtime does not use the default build projects in its repo - [example](https://github.com/dotnet/runtime/blob/1e6311a9795556149b5a051c5f5b2159d5a9765c/eng/Build.props#L7).

### /eng/Versions.props: A single file listing component versions and used tools

Expand Down
113 changes: 58 additions & 55 deletions src/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
DotNetBuildPass While building the repo as part of the entire .NET stack, this parameter specifies which build pass the current build is part of.
DotNetBuildSourceOnly Build the repo as part of the entire .NET stack with no external dependencies.
DotNetPublishUsingPipelines Publish assets to Build Asset Registry.
DotNetSymbolServerTokenMsdl Personal access token for MSDL symbol server. Available from variable group DotNet-Symbol-Server-Pats.
DotNetSymbolServerTokenSymWeb Personal access token for SymWeb symbol server. Available from variable group DotNet-Symbol-Server-Pats.
DotNetSymbolServerTokenMsdl Personal access token for MSDL symbol server. Available from variable group DotNet-Symbol-Server-Pats.
DotNetSymbolServerTokenSymWeb Personal access token for SymWeb symbol server. Available from variable group DotNet-Symbol-Server-Pats.
DotNetSymbolExpirationInDays Symbol expiration time in days (defaults to 10 years).
DotNetSignType Specifies the signing type: 'real' (default), 'test'.
DotNetRuntimeSourceFeed Storage account to lookup for the .Net runtime files
Expand All @@ -44,6 +44,7 @@

<ItemDefinitionGroup>
<ProjectToBuild>
<BuildStep>1</BuildStep>
<RestoreInParallel>true</RestoreInParallel>
<BuildInParallel>true</BuildInParallel>
<DotNetBuildPass>1</DotNetBuildPass>
Expand Down Expand Up @@ -93,28 +94,7 @@
<ProjectToBuild Include="$(RepoRoot)*.sln" />
</ItemGroup>

<Target Name="Execute">
<Error Text="No projects were found to build. Either the 'Projects' property or 'ProjectToBuild' item group must be specified." Condition="'@(ProjectToBuild)' == ''"/>
<Error Text="Property 'RepoRoot' must be specified" Condition="'$(RepoRoot)' == ''"/>
<Error Text="File 'global.json' must exist in directory specified by RepoRoot: '$(RepoRoot)'" Condition="'$(RepoRoot)' != '' and !Exists('$(RepoRoot)global.json')"/>

<PropertyGroup>
<!-- 'IsRunningFromVisualStudio' may be true even when running msbuild.exe from command line. This generally means that MSBuild is from a Visual Studio installation and therefore we need to find NuGet.targets in a different location. -->
<_NuGetRestoreTargets Condition="Exists('$(MSBuildToolsPath)\NuGet.targets')" >$(MSBuildToolsPath)\NuGet.targets</_NuGetRestoreTargets>
<_NuGetRestoreTargets Condition="'$([MSBuild]::IsRunningFromVisualStudio())' == 'true' And Exists('$(MSBuildToolsPath32)\..\..\..\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets')">$(MSBuildToolsPath32)\..\..\..\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets</_NuGetRestoreTargets>

<!-- If `DotNetPublishUsingPipelines` is not set we default it to false. -->
<DotNetPublishUsingPipelines Condition="'$(DotNetPublishUsingPipelines)' == ''">false</DotNetPublishUsingPipelines>

<!--
If DotNetPublishUsingPipelines is set we don't publish symbols during the build stage,
only in the Maestro post-build stages. If DotNetPublishUsingPipelines is NOT set then
we publish symbols during the build stage.
-->
<PublishToSymbolServer>true</PublishToSymbolServer>
<PublishToSymbolServer Condition="'$(DotNetPublishUsingPipelines)' == 'true'">false</PublishToSymbolServer>
</PropertyGroup>

<Target Name="InitializeSolutionBuildInputs">
<ItemGroup>
<_SolutionBuildTargets Include="Rebuild" Condition="'$(Rebuild)' == 'true'" />
<_SolutionBuildTargets Include="Build" Condition="'$(Build)' == 'true' and '$(Rebuild)' != 'true'" />
Expand Down Expand Up @@ -149,12 +129,41 @@
<_CommonProps Include="VCTargetsPath=$([MSBuild]::ValueOrDefault('$(VCTargetsPath)', '$([MSBuild]::GetVsInstallRoot())\Common7\IDE\VC\VCTargets\'))" Condition="'$(MSBuildRuntimeType)' != 'Core'"/>
</ItemGroup>

<ItemGroup>
<_SolutionBuildProps Include="@(_CommonProps)"/>
<_SolutionBuildProps Include="__DeployProjectOutput=$(Deploy)" Condition="'$(Deploy)' != ''"/>
<_SolutionBuildProps Include="__ImportPackTargets=true" Condition="'$(Pack)' == 'true'" />
</ItemGroup>
</Target>

<Target Name="Execute" DependsOnTargets="InitializeSolutionBuildInputs">
<Error Text="No projects were found to build. Either the 'Projects' property or 'ProjectToBuild' item group must be specified." Condition="'@(ProjectToBuild)' == ''"/>
<Error Text="Property 'RepoRoot' must be specified" Condition="'$(RepoRoot)' == ''"/>
<Error Text="File 'global.json' must exist in directory specified by RepoRoot: '$(RepoRoot)'" Condition="'$(RepoRoot)' != '' and !Exists('$(RepoRoot)global.json')"/>

<PropertyGroup>
<!-- 'IsRunningFromVisualStudio' may be true even when running msbuild.exe from command line. This generally means that MSBuild is from a Visual Studio installation and therefore we need to find NuGet.targets in a different location. -->
<_NuGetRestoreTargets Condition="Exists('$(MSBuildToolsPath)\NuGet.targets')" >$(MSBuildToolsPath)\NuGet.targets</_NuGetRestoreTargets>
<_NuGetRestoreTargets Condition="'$([MSBuild]::IsRunningFromVisualStudio())' == 'true' And Exists('$(MSBuildToolsPath32)\..\..\..\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets')">$(MSBuildToolsPath32)\..\..\..\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets</_NuGetRestoreTargets>

<!-- If `DotNetPublishUsingPipelines` is not set we default it to false. -->
<DotNetPublishUsingPipelines Condition="'$(DotNetPublishUsingPipelines)' == ''">false</DotNetPublishUsingPipelines>

<!--
If DotNetPublishUsingPipelines is set we don't publish symbols during the build stage,
only in the Maestro post-build stages. If DotNetPublishUsingPipelines is NOT set then
we publish symbols during the build stage.
-->
<PublishToSymbolServer>true</PublishToSymbolServer>
<PublishToSymbolServer Condition="'$(DotNetPublishUsingPipelines)' == 'true'">false</PublishToSymbolServer>
</PropertyGroup>

<ItemGroup Condition="'$(Restore)' == 'true'">
<_RestoreToolsProps Include="@(_CommonProps)"/>
<_RestoreToolsProps Include="BaseIntermediateOutputPath=$(ArtifactsToolsetDir)Common"/>
<_RestoreToolsProps Include="ExcludeRestorePackageImports=true"/>
<_RestoreToolsProps Include="_NuGetRestoreTargets=$(_NuGetRestoreTargets)"/>

<!-- Used in the SDK (Tools.proj) to control whether Build.Tasks.Feed will be restored or not. -->
<_RestoreToolsProps Include="Publish=$(Publish)"/>

Expand All @@ -166,33 +175,14 @@
</ItemGroup>

<!--
Builds from the 'internal' project, and only those, can download the .net Runtime
Builds from the 'internal' project, and only those, can download the .net Runtime
from a private location.
-->
<ItemGroup Condition="'$(SYSTEM_TEAMPROJECT)' == 'internal'">
<_RestoreToolsProps Include="DotNetRuntimeSourceFeed=$(DotNetRuntimeSourceFeed)"/>
<_RestoreToolsProps Include="DotNetRuntimeSourceFeedKey=$(DotNetRuntimeSourceFeedKey)"/>
</ItemGroup>

<ItemGroup>
<_PublishProps Include="@(_CommonProps)"/>

<!-- Used in a few places in the stack to decide if publishing was enabled or not. -->
<_PublishProps Include="Publish=$(Publish)"/>

<!-- Used in the SDK (Publish.proj) to control whether in-build symbol publishing should be performed. -->
<_PublishProps Include="PublishToSymbolServer=$(PublishToSymbolServer)"/>

<!-- Forward this property because we can't assume it will be available globally. -->
<_PublishProps Include="DotNetPublishUsingPipelines=$(DotNetPublishUsingPipelines)"/>
</ItemGroup>

<ItemGroup>
<_SolutionBuildProps Include="@(_CommonProps)"/>
<_SolutionBuildProps Include="__DeployProjectOutput=$(Deploy)" Condition="'$(Deploy)' != ''"/>
<_SolutionBuildProps Include="__ImportPackTargets=true" Condition="'$(Pack)' == 'true'" />
</ItemGroup>

<!--
Restore built-in tools.
-->
Expand Down Expand Up @@ -222,7 +212,7 @@
<_SolutionRestoreProps Include="MSBuildRestoreSessionId=$([System.Guid]::NewGuid())" />
<_SolutionRestoreProps Include="RestoreUseStaticGraphEvaluation=$(RestoreUseStaticGraphEvaluation)" />
</ItemGroup>

<PropertyGroup>
<!-- This can be set to false as an optimization for repos that don't use NuGet. -->
<RestoreUsingNuGetTargets Condition="'$(RestoreUsingNuGetTargets)' == ''">true</RestoreUsingNuGetTargets>
Expand Down Expand Up @@ -277,15 +267,7 @@
BuildInParallel="%(_ProjectToRestore.RestoreInParallel)"
Condition="'$(Restore)' == 'true'"/>

<!--
Build solution.
-->
<MSBuild Projects="@(ProjectToBuild)"
Properties="@(_SolutionBuildProps);__BuildPhase=SolutionBuild;_NETCORE_ENGINEERING_TELEMETRY=Build"
RemoveProperties="$(_RemoveProps)"
Targets="@(_SolutionBuildTargets)"
BuildInParallel="%(ProjectToBuild.BuildInParallel)"
Condition="'@(_SolutionBuildTargets)' != ''" />
<CallTarget Targets="ExecuteBuildTargets" Condition="'@(_SolutionBuildTargets)' != ''" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentioanl that the build step is the only one that is executed for each, rather than for signing, AfterSolutionBuild.proj, etc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AfterSolutionBuild.proj, Sign.proj and Publish.proj run only once and not per ProjectToBuild item. So they don't need any batching. Restore is the only target that takes ProjectToBuild as an input but it's an anti-pattern to invoke the Restore target on every project which is why I intentionally don't support BuildStep there.


<MSBuild Projects="AfterSolutionBuild.proj"
Properties="@(_CommonProps);_NETCORE_ENGINEERING_TELEMETRY=Build"
Expand Down Expand Up @@ -333,11 +315,32 @@
<_ShouldRunPublish Condition="'$(DotNetBuildPhase)' != 'InnerRepo' and '$(DotNetBuildOrchestrator)' != 'true'">true</_ShouldRunPublish>
</PropertyGroup>

<ItemGroup>
<_PublishProps Include="@(_CommonProps)"/>
<!-- Used in a few places in the stack to decide if publishing was enabled or not. -->
<_PublishProps Include="Publish=$(Publish)"/>
<!-- Used in the SDK (Publish.proj) to control whether in-build symbol publishing should be performed. -->
<_PublishProps Include="PublishToSymbolServer=$(PublishToSymbolServer)"/>
<!-- Forward this property because we can't assume it will be available globally. -->
<_PublishProps Include="DotNetPublishUsingPipelines=$(DotNetPublishUsingPipelines)"/>
</ItemGroup>

<!-- Make sure we always publish in VMR build - working around runtime repo which sets Publish to false. -->
<MSBuild Projects="Publish.proj"
Properties="@(_PublishProps);_NETCORE_ENGINEERING_TELEMETRY=Publish"
Targets="Publish"
Condition="'$(Publish)' == 'true' and '$(_ShouldRunPublish)' == 'true'"/>
</Target>

<Target Name="ExecuteBuildTargets"
Inputs="%(ProjectToBuild.BuildStep)"
Outputs="unused"
DependsOnTargets="InitializeSolutionBuildInputs">
<MSBuild Projects="@(ProjectToBuild)"
Properties="@(_SolutionBuildProps);__BuildPhase=SolutionBuild;_NETCORE_ENGINEERING_TELEMETRY=Build"
RemoveProperties="$(_RemoveProps)"
Targets="@(_SolutionBuildTargets)"
BuildInParallel="%(ProjectToBuild.BuildInParallel)" />
</Target>

</Project>
Loading