Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Users referenced in this file will automatically be requested as reviewers for PRs that modify the given paths.
# See https://help.github.com/articles/about-code-owners/

* @MichaelSimons @mthalman
* @dotnet/source-build
2 changes: 1 addition & 1 deletion Documentation/sourcebuild-in-repos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ with and participate in source-build.
* [Updating dependencies](update-dependencies.md)
* [Considerations when adding features](adding-features.md)
* [General notes on the source-build build](build-info.md)
* [Adding new repositories](new-repo.md)
* [Onboarding repositories to source build](new-repo.md)
34 changes: 23 additions & 11 deletions Documentation/sourcebuild-in-repos/build-info.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,26 @@ produces multiple versions and these can be hard-coded or use a
`$(<PackageName>ReferenceVersion)` property or similar if you don't need
source-build to change them.

## Platform-specific packages

Packages that require components or packages only available on some other
operating system than the building OS cannot be built in source-build. These
should use `<ExcludeFromSourceBuild>true</ExcludeFromSourceBuild>` or other
options to be excluded from the source-build. For instance, if a project
depends on a package that can only be built on Windows, it will need to be
disabled or worked around in source-build. As an example, [Roslyn
removes](https://github.com/dotnet/roslyn/blob/b999a65c8b0feeccb2b58da3d7a6e80e5f08feab/src/Workspaces/Core/Portable/Storage/PersistentStorageExtensions.cs#L23)
a small performance improvement when building for source-build because it
requires a component that isn't available.
## Platform-specific components

It is not always necessary or correct to include all repo components in source build.
**Important philosophy: The Microsoft build and source build should be as close to the same as possible.**
Components should be excluded based on platform requirements rather than source-build specific reasons.
To exclude components, use the [VMR Controls](https://github.com/dotnet/dotnet/blob/main/docs/VMR-Controls.md) properties.

Common patterns include:

```xml
<!-- Exclude Windows-only components on non-Windows platforms -->
<ItemGroup Condition="'$(TargetOS)' == 'windows'">
<ProjectReference Include="WindowsSpecific.csproj" />
</ItemGroup>

<!-- Exclude components from source-only builds when truly necessary -->
<ItemGroup Condition="'$(DotNetBuildSourceOnly)' != 'true'">
<ProjectReference Include="MicrosoftProprietary.csproj" />
</ItemGroup>
```

**Avoid using `DotNetBuildSourceOnly` conditions unless absolutely necessary.**
**Prefer platform-based conditions** (like `TargetOS`) **to maintain alignment between Microsoft and source builds.**
143 changes: 36 additions & 107 deletions Documentation/sourcebuild-in-repos/new-repo.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,34 @@
# Adding a new repository to source build
# Onboarding a repository to source build

This document describes the overall process of onboarding new repos onto source
build.

1. [Source Build Configuration](#source-build-configuration)
1. [Setup CI](#setup-ci)
1. [Source build repos and the VMR](#source-build-repos-and-the-vmr)
1. [Validate](#validate)
This document describes the overall process of onboarding repos onto source build.

## Source Build Configuration

### Trying it out locally

If a repo passes build options through to the common arcade build script, a
source build can be triggered via the following command.
If a repo passes build options through to the common arcade build script, a source build can be triggered via the following command.

```bash
./build.sh -sb
```

> Note: [source build is not supported on
Windows](https://github.com/dotnet/source-build/issues/1190), only Linux and
macOS.

### Excluding components
> Note: [source build is only supported on Linux](https://github.com/dotnet/source-build/?tab=readme-ov-file#support).

It is not always necessary or correct to include all repo components in source
build. Components should be excluded if they are not required for the platform
being source-built. Including them expands the source build graph and may not
be possible because of licensing. Examples include tests, Windows components
(remember source build currently only supports Linux and macOS), etc. To exlcude
these components use the `DotNetBuildSourceOnly` msbuild property to
conditionally exclude.
### Excluding platform-specific components

```code
Condition="'$(DotNetBuildSourceOnly)' != 'true'"
```

See the [Unified Build Controls](https://github.com/dotnet/dotnet/blob/main/docs/VMR-Controls.md)
for more options on how to exclude components from source build.
See the [platform-specific components guidelines](./build-info.md#platform-specific-components) for guidance on excluding components based on platform requirements rather than source-build specific reasons.

## Setup CI

Source build should run in PR validation builds, to prevent regressions.
**Repository-level source build CI legs are optional.**
Consider adding them only if your repository has many source-build specific conditions or complex source-build scenarios.
If you have a lot of source-build specific conditions, this is when you should consider adding repo-level CI legs.

Source build CI can be activated with a single flag in the ordinary Arcade SDK
jobs template for easy consumption. If a repo can't simply use the Arcade SDK
jobs template, more granular templates are also available.
Source build CI can be activated with a single flag in the ordinary Arcade SDK jobs template for easy consumption.
If a repo can't simply use the Arcade SDK jobs template, more granular templates are also available.

See <https://github.com/dotnet/arcade/tree/main/eng/common/templates> for the
current template source code. The inline comments in the `parameters:` section
in those files are the most up to date docs, maintained with higher priority
than this general onboarding doc.
See <https://github.com/dotnet/arcade/tree/main/eng/common/templates> for the current template source code.
The inline comments in the `parameters:` section in those files are the most up to date docs, maintained with higher priority than this general onboarding doc.

### `eng/common/templates/jobs/jobs.yml` opt-in switch

Expand All @@ -60,46 +37,27 @@ The simplest way to onboard. This approach applies if the repo already uses the

To opt in:

1. Set `enableSourceBuild: true`

Set `enableSourceBuild: true` in the template parameters.

This should look something like [this sourcelink
implementation:](https://github.com/dotnet/sourcelink/blob/dfe619dc722be42d475595c755c958afe6177554/azure-pipelines.yml#L40)

```yaml
stages:
- stage: build
displayName: Build
jobs:
- template: /eng/common/templates/jobs/jobs.yml
parameters:
enablePublishUsingPipelines: true
enablePublishBuildArtifacts: true
enablePublishBuildAssets: true
enableSourceBuild: true
artifacts:
publish:
artifacts: true
manifests: true
```

1. Specify platforms (if repo is not managed-only)

A repo is managed-only if `eng/SourceBuild.props` contains
`<SourceBuildManagedOnly>true</SourceBuildManagedOnly>`. If this is true,
this step is not necessary. Otherwise, specify `sourceBuildParameters` in
the `jobs.yml` template's parameters like this:

```yaml
sourceBuildParameters:
platforms:
- name: 'CentosStream8_Portable'
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8'
- name: 'CentosStream8'
nonPortable: true
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8'
```
Set `enableSourceBuild: true` in the template parameters.

This should look something like [this sourcelink
implementation:](https://github.com/dotnet/sourcelink/blob/dfe619dc722be42d475595c755c958afe6177554/azure-pipelines.yml#L40)

```yaml
stages:
- stage: build
displayName: Build
jobs:
- template: /eng/common/templates/jobs/jobs.yml
parameters:
enablePublishUsingPipelines: true
enablePublishBuildArtifacts: true
enablePublishBuildAssets: true
enableSourceBuild: true
artifacts:
publish:
artifacts: true
manifests: true
```

#### End result

Expand Down Expand Up @@ -153,34 +111,5 @@ this just happens to fit in nicely.

## Source build repos and the VMR

Another effect of adding a new source build repository is that its sources will
be synchronized into the [Virtual Monolithic Repository of
.NET](https://github.com/dotnet/dotnet). The VMR is then where the official
source build happens from.

In order for the sources of the new repo to by synchronized into the VMR, the
repo needs to be registered in the [`source-mappings.json`
file](https://github.com/dotnet/dotnet/blob/main/src/source-mappings.json) which
tells the tooling where from and which sources should be synchronized. Please
open a PR in [`dotnet/dotnet`](https://github.com/dotnet/dotnet) and add
your repository into `src/source-mappings.json`.

### Cloaking (filtering) the repository sources

When creating the `source-mappings.json` record for the new repo, there is a
possibility of filtering which sources get synchronized into the VMR. The VMR
should only really contain plain text-based files as it is consumed by 3rd party
.NET distro builders who do not accept any non-text sources (e.g. binaries).
When registering the repository into the VMR, it is a good time to consider
which files are required for it to build and only synchronize those. Commonly,
repositories contain binaries that are required for testing or similar purposes.
Files like these should not be synchronized into the VMR. Another common
scenario is that the repo has multiple products/ship vehicles and only a subset
is needed for the source-built .NET scenario.

## Validate

Once the code flows to the VMR, the PR validation legs will ensure that no
[prebuilts](https://github.com/dotnet/source-build/blob/main/Documentation/eliminating-pre-builts.md#what-is-a-prebuilt)
were added to the system and everything is functioning correctly.
If you need help on addressing any prebuilds, reach out to @source-build.
In order to fully onboard a repo into source build, the repo must be included within the VMR.
The [Repository Onboarding Guide](https://github.com/dotnet/dotnet/blob/main/docs/Repository-Onboarding.md) contains the comprehensive VMR onboarding guidance which includes among several other topics, resolving source build prebuilts.
Loading