From ae8bb9c8715545fab83493c03c1b50aa9f69919d Mon Sep 17 00:00:00 2001 From: Viktor Hofer Date: Tue, 20 May 2025 12:30:22 +0200 Subject: [PATCH 1/2] Update new-repo.md --- .../sourcebuild-in-repos/new-repo.md | 194 +----------------- 1 file changed, 6 insertions(+), 188 deletions(-) diff --git a/Documentation/sourcebuild-in-repos/new-repo.md b/Documentation/sourcebuild-in-repos/new-repo.md index 024600bccd..d478ea08f7 100644 --- a/Documentation/sourcebuild-in-repos/new-repo.md +++ b/Documentation/sourcebuild-in-repos/new-repo.md @@ -5,90 +5,11 @@ build. 1. [Source Build Configuration](#source-build-configuration) 1. [Setup CI](#setup-ci) -1. [Incorporate the new repo into the source build dependency - tree](#incorporate-the-new-repo-into-the-source-build-dependency-tree) 1. [Source build repos and the VMR](#source-build-repos-and-the-vmr) 1. [Validate](#validate) -1. [Additional resources](#additional-resources) ## Source Build Configuration -Configuring source build involves setting up files in `eng/` that determine the -behavior of source build in the repo. - -These changes are all needed before source build will work: - -* [`eng/DotNetBuild.props`](#engdotnetbuildprops) - Basic properties, such as - repo name. -* [`eng/SourceBuildPrebuiltBaseline.xml`](#engsourcebuildprebuiltbaselinexml) - - List of allowed prebuilts (approval required). -* [`eng/Version.Details.xml`](#engversiondetailsxml) - Already exists, but - modifications are needed to pull dependencies from upstream [intermediate - nupkgs](../planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs). - -See the following sections for details: - -### `eng/DotNetBuild.props` - -```xml - - - - this-repo - true - - - -``` - -This file contains basic configuration used to restore the correct dependencies -(managed-only or not) and produce the right name for the repo's [intermediate -nupkg]. - -* `this-repo` should be the same as the repo's name on GitHub, without the - GitHub organization name. -* `SourceBuildManagedOnly` defaults to false if omitted. `true` means that the - repo doesn't produce any RID-specific artifacts like - `Microsoft.NETCore.App.Runtime.linux-x64`, only managed code. - -These two properties determine the name of the [intermediate nupkg]: -`Microsoft.SourceBuild.Intermediate.$(GitHubRepositoryName)[.$(RidSuffix)]`. - -It's possible more configuration will be required for specific repos. -`eng/DotNetBuild.props`, similar to `eng/Build.props`, is a place to add extra -MSBuild code that can change the way source build behaves. - -### `eng/SourceBuildPrebuiltBaseline.xml` - -```xml - - - - - - - - -``` - -This defines which prebuilt NuGet packages are allowed to be used during the -build. Initially, only the source build intermediate packages are allowed. The -`*/*` glob means "any intermediate package, any version". All other prebuilts -require approval from the source build team. - -When a PR introduces an unexpected prebuilt, PR validation will fail and let us -resolve the source-buildability issue before the PR gets merged. - -### CODEOWNERS - -Add the source build team as the -[CODEOWNER](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) -of the source build infrastructure. - -``` text -/eng/SourceBuild* @dotnet/source-build -``` - ### Trying it out locally If a repo passes build options through to the common arcade build script, a @@ -98,28 +19,10 @@ source build can be triggered via the following command. ./build.sh -sb ``` -If a repo does not pass through build options through to the common arcade build -script, then you must specify the `/p:ArcadeBuildFromSource=true` property as -follows. - -```bash -./build.sh -c Release --restore --build --pack /p:ArcadeBuildFromSource=true -bl -``` - > Note: [source build is not supported on Windows](https://github.com/dotnet/source-build/issues/1190), only Linux and macOS. -After running the build, source build artifacts will be in -`artifacts/sb`, and the [intermediate nupkg] will be something like -`artifacts/packages/*/Microsoft.SourceBuild.Intermediate.*.nupkg`. - -The MSBuild binlog will be placed somewhere like: -`artifacts/log/Release/Build.binlog`. However, this "outer" binlog doesn't contain -the meat of the build: the "inner" build runs inside an `Exec` task. The inner -binlog will be written to somewhere like: -`artifacts/sb/src/artifacts/log/Release/source-inner-build.binlog`. - ### Excluding components It is not always necessary or correct to include all repo components in source @@ -127,68 +30,19 @@ build. Components should be excluded if they are not required for the platform being source-built. Including them expands the source build graph and may not be possible because of licensing. Examples include tests, Windows components (remember source build currently only supports Linux and macOS), etc. To exlcude -these components use the `DotNetBuildFromSource` msbuild property to +these components use the `DotNetBuildSourceOnly` msbuild property to conditionally exclude. ```code -Condition="'$(DotNetBuildFromSource)' != 'true'" +Condition="'$(DotNetBuildSourceOnly)' != 'true'" ``` See the [Unified Build Controls](https://github.com/dotnet/arcade/blob/main/Documentation/UnifiedBuild/Unified-Build-Controls.md) for more options on how to exclude components from source build. -### `eng/Version.Details.xml` - -```xml -... - - https://github.com/dotnet/runtime - a820ca1c4f9cb5892331e2624d3999c39161fe2a - - -... - - https://github.com/dotnet/source-build-reference-packages - def2e2c6dc5064319250e2868a041a3dc07f9579 - - -... -``` - -Dependency changes may include adding `SourceBuild` to existing dependency -elements, and adding a new `source-build-reference-packages` element. - -`SourceBuild` causes the source build targets in the Arcade SDK to download -[intermediate nupkg]s from the upstream repo's official build, rather than using -prebuilt binaries to fulfill the dependencies. Note that `RepoName` is used to -calculate the ID of the [intermediate -nupkg](../planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs): -the `Dependency` `Name` is ignored by source build. - -Building with the source-built versions of your dependencies also means that any -upstream repos will have been built in a source build context, including things -projects that are excluded from the source build. This can help you find issues -where your source build depends on an upstream component that isn't actually -built in source build. - -`ManagedOnly` determines whether a RID suffix is necessary on the [intermediate -nupkg](../planning/arcade-powered-source-build/README.md#intermediate-nupkg-outputsinputs) -ID. For example, running source build on `dotnet/installer` with `linux-x64` -with the above example configuration will restore: - -* `Microsoft.SourceBuild.Intermediate.runtime.linux-x64` - * `.linux-x64` because `ManagedOnly` is not `true`. -* `Microsoft.SourceBuild.Intermediate.source-build-reference-packages` - * Ends with the `RepoName` without a suffix because `ManagedOnly="true"`. - ## Setup CI -Source build needs to run during official builds to create source build -[intermediate nupkg]s for the downstream repos that will consume them. Source -build should also run in PR validation builds, to prevent regressions. +Source build should run in PR validation builds, to prevent regressions. Source build CI can be activated with a single flag in the ordinary Arcade SDK jobs template for easy consumption. If a repo can't simply use the Arcade SDK @@ -297,46 +151,19 @@ the most granular template, and may be useful if some repo-specific preamble or cleanup steps are required, or if the repo already has job matrix templates and this just happens to fit in nicely. -### Official build publishing - -Publishing [intermediate nupkg]s in the official build is handled by the -standard Arcade publish infrastructure, which should already be set up in the -repo. The source build steps handle uploading the [intermediate nupkg] to the -pipeline in the standard way that Arcade will detect and publish. - -[intermediate nupkg]: - https://github.com/dotnet/source-build/blob/main/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md - -## Incorporate the new repo into the source build dependency tree - -Once your repo can be source-built, it is time to register it into the source -build dependency tree. The graph of the product is defined by the -`eng/Version.Details.xml` files. This dependency graph starts at -[dotnet/sdk](https://github.com/dotnet/sdk/blob/main/eng/Version.Details.xml). -The dependendecies of repos declared in these files are walked and the first -copy/commit of each repo found in the dependency graph is used. - -Therefore, when adding new repositories, a dependency must be created within the -`eng/Version.Details.xml` graph. To do this, **go to the repo which depends on -the new repo and add a [new source build dependency](#engversiondetailsxml) to -the new source build repo**. - ## Source build repos and the VMR Another effect of adding a new source build repository is that its sources will be synchronized into the [Virtual Monolithic Repository of .NET](https://github.com/dotnet/dotnet). The VMR is then where the official -source build happens from. The sources are synchronized once the associated -commit/package flows into `dotnet/sdk`. +source build happens from. In order for the sources of the new repo to by synchronized into the VMR, the repo needs to be registered in the [`source-mappings.json` file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) which tells the tooling where from and which sources should be synchronized. Please -open a PR in [`dotnet/sdk`](https://github.com/dotnet/sdk) and add -your repository into `src/VirtualMonoRepo/source-mappings.json`. The name must -match the name declared in the `SourceBuild` tag in `Version.Details.xml` -created in the previous step. +open a PR in [`dotnet/sdk`](https://github.com/dotnet/dotnet) and add +your repository into `src/source-mappings.json`. ### Cloaking (filtering) the repository sources @@ -360,12 +187,3 @@ VMR will be built. This will validate that no prebuilts were added to the system and everything is functioning correctly. Please notify [@source-build](https://github.com/orgs/dotnet/teams/source-build) to be on the lookout for the new repo and they will validate as necessary. - -## Additional resources - -* For more details about how the build executes, see [Arcade's build - tools](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild). -* The source code for the build tasks that run for prebuilt validation and - intermediate nupkg dependency reading are maintained in - [Arcade](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.SourceBuild) - as well. From 54f79422f5b7c24e5b3e5a9d509a4c6525ecc9a9 Mon Sep 17 00:00:00 2001 From: Viktor Hofer Date: Thu, 22 May 2025 17:16:31 +0200 Subject: [PATCH 2/2] Update new-repo.md --- Documentation/sourcebuild-in-repos/new-repo.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/Documentation/sourcebuild-in-repos/new-repo.md b/Documentation/sourcebuild-in-repos/new-repo.md index d478ea08f7..436969e547 100644 --- a/Documentation/sourcebuild-in-repos/new-repo.md +++ b/Documentation/sourcebuild-in-repos/new-repo.md @@ -162,7 +162,7 @@ In order for the sources of the new repo to by synchronized into the VMR, the repo needs to be registered in the [`source-mappings.json` file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) which tells the tooling where from and which sources should be synchronized. Please -open a PR in [`dotnet/sdk`](https://github.com/dotnet/dotnet) and add +open a PR in [`dotnet/dotnet`](https://github.com/dotnet/dotnet) and add your repository into `src/source-mappings.json`. ### Cloaking (filtering) the repository sources @@ -180,10 +180,7 @@ is needed for the source-built .NET scenario. ## Validate -Once the downstream dependency(s) are added to the new repo and those changes -flow into `dotnet/sdk`, a complete .NET product can be built from source. -The repository will be synchronized into the VMR during the first build and the -VMR will be built. This will validate that no prebuilts were added to the system -and everything is functioning correctly. Please notify -[@source-build](https://github.com/orgs/dotnet/teams/source-build) -to be on the lookout for the new repo and they will validate as necessary. +Once the code flows to the VMR, the PR validation legs will ensure that no +[prebuilts](https://github.com/dotnet/source-build/blob/main/Documentation/eliminating-pre-builts.md#what-is-a-prebuilt) +were added to the system and everything is functioning correctly. +If you need help on addressing any prebuilds, reach out to @source-build.