diff --git a/Documentation/planning/arcade-powered-source-build/README.md b/Documentation/planning/arcade-powered-source-build/README.md deleted file mode 100644 index 9b102a42b4..0000000000 --- a/Documentation/planning/arcade-powered-source-build/README.md +++ /dev/null @@ -1,213 +0,0 @@ -# ArPow (arcade-powered source-build) - -Source-build exists as a process outside the Microsoft build, taking place after -the Microsoft build is complete. This creates a significant amount of catch-up -work to keep .NET buildable from source. Integration into the normal build -process is necessary to make source-build sustainable. - -This is a user story in .NET 6.0: -[dotnet/core#5448](https://github.com/dotnet/core/issues/5448) - -> *Definitions*: -> -> * **Official builds** run in Azure DevOps and produce the signed bits released -> by Microsoft. These run on a daily or per-commit basis depending on repo and -> point in time. -> * **PR validation** builds run on each pull request submitted to a repository, -> and are typically required to succeed to merge. There are also **rolling -> builds** that run either periodically or after each merge that run -> lower-priority and/or slower tests. - -ArPow consists of two main goals with some key benefits: - -1. When the .NET Core SDK **official build** completes, its artifacts include - validated, ship-ready source-build outputs, in addition to the Microsoft - build outputs. - - * This way, we know that if a given SDK commit has a Microsoft build - available, it is also buildable from source. - - * If an error occurs during source-build, it is visible to the same - stakeholders as a Microsoft build failure, not just the source-build team. - - * Breaks in source-build can be detected and fixed immediately by repo - owners, just as any other build failure. This is opposed to the traditional - workflow: source-build maintainers detect the issue weeks to months after - the fact, and must fix it using patches. - - * With ArPow and sufficient automated testing, there is zero delay between - Microsoft build completion and source-build readiness. The traditional - delay no longer needs to be considered by the Microsoft release process - owners when figuring out a release timeline. - - * Rebuilding the SDK close to a release day doesn't cause setbacks to - source-build maintainers by throwing away manual build uptake work. - -2. Every repo involved in source-build validates source-buildability in its **PR - validation** build. - - * This allows developers and release owners to understand the source-build - impact of changes, reducing the frequency the source-build servicing team - has to root-cause and patch over problems. - - * Gives us a place to add additional source-build-specific tests for common - problems, such as prebuilt usage regressions. - -This doc is about where we can start, what incremental progress would look like, -and the vision. - -## Starting point: SBRP (dotnet/source-build-reference-packages) - -Status: Complete. - -We start by putting ArPow build infrastructure into SBRP, a repository owned by -the source-build team that has no upstream dependencies. This is done using the -Arcade SDK's extension points and custom Azure DevOps templates. Both are -integrated into Arcade after we see them working in SBRP. This lets us quickly -prototype the core ArPow functionality. - -There are a few key parts of the infrastructure to prototype: - -### Build integration - -The official build and PR validation build pipelines work, with a reasonable API -for other repositories to adopt. - -* [source-build-in-pipeline.md] - -### Intermediate nupkg outputs/inputs - -Every ArPow-enabled repository creates a NuGet package containing its build -outputs, for downstream repositories to use. On the other side, the Arcade SDK -infrastructure needs to be able to restore and use source-built intermediate -nupkg content. - -SBRP ultimately won't use the restore-side infrastructure because it has no -upstreams, however, it can be developed here as a prototype. - -* [intermediate-nupkg.md] -* [Create proof of concept: the source-build intermediate nupkg format - #1543](https://github.com/dotnet/source-build/issues/1543) -* [Use SBRP intermediate nupkg to build SBRP - #1636](https://github.com/dotnet/source-build/issues/1636) - -### Prebuilt usage tracking - -We aren't eliminating instances of prebuilt usage at this stage in the process, -but the tooling needs to work in the context of an Arcade-powered build. - -* [Include prebuilt reports in intermediate nupkgs - #1725](https://github.com/dotnet/source-build/issues/1725) - -### Managed-only and RID-specific builds - -Some .NET repositories only produce managed code outputs, and others produce -RID-specific outputs such as native binaries. To reduce the burden of ArPow on -small managed-only repositories, we build only one RID and use the same results -for all RIDs downstream. All repositories must be buildable under any RID, but -for simple managed-only repositories, this is a reasonable assumption. The -templates and Arcade SDK need to handle this. - -* [Add mechanism to restore RID-specific intermediate nupkgs - #1722](https://github.com/dotnet/source-build/issues/1722) - -### Integrating tooling into Arcade - -The ArPow tooling can be implemented inside the Arcade SDK's MSBuild extension -points, initially. Before onboarding any extra repos, it should be integrated -into the Arcade SDK proper to reuse the code. - -* [in-arcade.md] -* [Source-build-specific MSBuild tool source in Arcade SDK - (`Microsoft.DotNet.SourceBuild`)](https://github.com/dotnet/arcade/tree/main/src/Microsoft.DotNet.SourceBuild) - -## Incremental progress - -### Adding ArPow to the build graph - -When we have ArPow SBRP complete, we start adding ArPow to repositories in the -.NET SDK dependency graph. - -Working from the bottom (leaves) upward (towards dotnet/installer), we onboard -repos to ArPow: consume intermediate nupkgs from upstreams, run ArPow PR -validation, and produce intermediate nupkgs in the official build. For more -details about this process, see: - -* [incremental-official.md] - -It is possible to instead only implement official source-build in a handful of -repos. This segments the build into chunks, which must be coherent. This idea is -discussed in [incremental-official-chunked.md], and is not recommended. - -Some repos required to build .NET aren't maintained by Microsoft, and don't have -official builds that inject artifacts into the Arcade dependency flow system -(Darc/BAR/Maestro++). It isn't feasible to add ArPow to these repos in the same -way. These repos will continue being built by dotnet/source-build, and -dotnet/source-build becomes a leaf in the build graph instead of an -orchestration repo. - -### The speculative SDK - -It's difficult to validate that a PR won't break downstream repos. This problem -is shared by source-build and the Microsoft build. "Speculative builds" have -been proposed to try to help with this, but would be very difficult to implement -in the Microsoft build. It may be more reasonable in the context of -source-build: all builds happen on a single machine, so the problem is focused -on figuring out a build graph rather than organizing dozens of machines in a -build lab and flowing bits across a network. - -This is also necessary in source-build to validate several distro maintenance -scenarios: by making a PR, is it still possible to run a bootstrap build of the -.NET Core SDK? Can .NET Core SDK version N be built using SDK N-1? - -This can be developed in parallel to other efforts. See [speculative-build.md] -for more info about speculative builds. - -### Adding unit tests - -Unit tests are typically disabled in dotnet/source-build, because the test -infrastructure isn't built from source, and it increases build time. We should -run tests on the source-build product to catch bugs. However, this isn't -critical for the sustainability goals of this plan. - -## End result - -When all of this is working, the official Microsoft build of the .NET Core SDK -also produces tarballs that distro maintainers can use to build it from source. -Contributors in each repo use checks in PR validation to verify each change is -compatible with source-build requirements, and if validation runs into a -problem, they are able to reproduce the build locally using an Arcade build -command. - -[in-arcade.md]: in-arcade.md -[incremental-official-chunked.md]: incremental-official-chunked.md -[incremental-official.md]: incremental-official.md -[source-build-in-pipeline.md]: source-build-in-pipeline.md -[speculative-build.md]: speculative-build.md -[intermediate-nupkg.md]: intermediate-nupkg.md -[intermediate nupkgs]: intermediate-nupkg.md - ---- - -## Revisions - -**2021-02-17** dagood -Updated this doc to align with the current ArPow implementation plan. The old -implementation plan can be found at: -[2bf686a9](https://github.com/dotnet/source-build/tree/2bf686a91560911477885ae139909bbf10b7ca98/Documentation/planning/arcade-powered-source-build). -The plan has shifted towards purely incremental changes, with no end-to-end -ArPow source-build working until completion. - -**2021-02-11** dagood -Added a plan to use "supplemental intermediate nupkgs" to handle situations -where the [intermediate nupkgs] are too large to publish to Azure DevOps. - -**2020-07-15** dagood -Removed the plan to test every added intermediate nupkg all the way downstream -in dotnet/installer. Looking at it after some hands-on work has been done, we -don't think this end-to-end integration test is actually feasible. Not running -these tests creates some uncertainty, but it seems acceptable. We will likely -end up with a backlog of unknown issues to work through once we start building a -tarball in dotnet/installer. They *may* have been avoided with testing. We don't -expect this will be disruptive enough to make it worth trying to more -exhaustively find some way to get end-to-end testing feasible. diff --git a/Documentation/planning/arcade-powered-source-build/img/official-leaves.dot b/Documentation/planning/arcade-powered-source-build/img/official-leaves.dot deleted file mode 100644 index 4621734dd4..0000000000 --- a/Documentation/planning/arcade-powered-source-build/img/official-leaves.dot +++ /dev/null @@ -1,110 +0,0 @@ -digraph { -node [color="lightsteelblue1", style=filled]; -rankdir=BT; - -"sourcelink"[color="gold"]; -"xliff-tasks"[color="gold"]; -"newtonsoft-json901"[color="gold"]; -"linker"[color="gold"]; -"application-insights"[color="gold"]; -"netcorecli-fsc"[color="gold"]; - -"sourcelink" -> "arcade" -"arcade" -> "aspnet-extensions" -"core-setup" -> "aspnet-extensions" -"roslyn" -> "aspnet-extensions" -"arcade" -> "aspnet-xdt" -"arcade" -> "aspnetcore-tooling" -"aspnet-extensions" -> "aspnetcore-tooling" -"core-setup" -> "aspnetcore-tooling" -"msbuild" -> "aspnetcore-tooling" -"roslyn" -> "aspnetcore-tooling" -"arcade" -> "aspnetcore" -"aspnetcore-tooling" -> "aspnetcore" -"core-setup" -> "aspnetcore" -"msbuild" -> "aspnetcore" -"roslyn" -> "aspnetcore" -"arcade" -> "cli" -"application-insights" -> "cli" -"clicommandlineparser" -> "cli" -"core-setup" -> "cli" -"corefx" -> "cli" -"msbuild" -> "cli" -"nuget-client" -> "cli" -"newtonsoft-json" -> "cli" -"newtonsoft-json901" -> "cli" -"roslyn" -> "cli" -"sdk" -> "cli" -"standard" -> "cli" -"templating" -> "cli" -"websdk" -> "cli" -"xliff-tasks" -> "cli" -"arcade" -> "clicommandlineparser" -"newtonsoft-json" -> "common" -"aspnetcore" -> "core-sdk" -"newtonsoft-json" -> "core-sdk" -"newtonsoft-json901" -> "core-sdk" -"toolset" -> "core-sdk" -"coreclr" -> "core-setup-portable" -"corefx" -> "core-setup-portable" -"newtonsoft-json" -> "core-setup-portable" -"newtonsoft-json901" -> "core-setup-portable" -"roslyn" -> "core-setup-portable" -"standard" -> "core-setup-portable" -"coreclr" -> "core-setup" -"corefx" -> "core-setup" -"newtonsoft-json" -> "core-setup" -"newtonsoft-json901" -> "core-setup" -"roslyn" -> "core-setup" -"standard" -> "core-setup" -"linker" -> "coreclr-portable" -"roslyn" -> "coreclr-portable" -"linker" -> "coreclr" -"roslyn" -> "coreclr" -"arcade" -> "corefx-portable" -"linker" -> "corefx-portable" -"coreclr" -> "corefx-portable" -"standard" -> "corefx-portable" -"arcade" -> "corefx" -"linker" -> "corefx" -"coreclr" -> "corefx" -"standard" -> "corefx" -"arcade" -> "fsharp" -"newtonsoft-json901" -> "fsharp" -"newtonsoft-json" -> "fsharp" -"xliff-tasks" -> "fsharp" -"arcade" -> "msbuild" -"corefx" -> "msbuild" -"roslyn" -> "msbuild" -"arcade" -> "newtonsoft-json" -"newtonsoft-json" -> "nuget-client" -"common" -> "nuget-client" -"msbuild" -> "nuget-client" -"arcade" -> "roslyn" -"xliff-tasks" -> "roslyn" -"arcade" -> "sdk" -"xliff-tasks" -> "sdk" -"core-setup" -> "sdk" -"msbuild" -> "sdk" -"newtonsoft-json" -> "sdk" -"newtonsoft-json901" -> "sdk" -"nuget-client" -> "sdk" -"arcade" -> "standard" -"clicommandlineparser" -> "templating" -"newtonsoft-json" -> "templating" -"aspnetcore-tooling" -> "toolset" -"cli" -> "toolset" -"fsharp" -> "toolset" -"netcorecli-fsc" -> "toolset" -"newtonsoft-json" -> "toolset" -"newtonsoft-json901" -> "toolset" -"vstest" -> "toolset" -"websdk" -> "toolset" -"aspnet-extensions" -> "vstest" -"core-setup" -> "vstest" -"newtonsoft-json" -> "vstest" -"arcade" -> "websdk" -"aspnetcore" -> "websdk" -"aspnet-xdt" -> "websdk" -"newtonsoft-json" -> "websdk" -} diff --git a/Documentation/planning/arcade-powered-source-build/img/official-leaves.svg b/Documentation/planning/arcade-powered-source-build/img/official-leaves.svg deleted file mode 100644 index 9e81a9ce73..0000000000 --- a/Documentation/planning/arcade-powered-source-build/img/official-leaves.svg +++ /dev/null @@ -1,663 +0,0 @@ - - - - - - -%3 - - -sourcelink - -sourcelink - - -arcade - -arcade - - -sourcelink->arcade - - - - -xliff-tasks - -xliff-tasks - - -roslyn - -roslyn - - -xliff-tasks->roslyn - - - - -cli - -cli - - -xliff-tasks->cli - - - - -sdk - -sdk - - -xliff-tasks->sdk - - - - -fsharp - -fsharp - - -xliff-tasks->fsharp - - - - -newtonsoft-json901 - -newtonsoft-json901 - - -core-setup - -core-setup - - -newtonsoft-json901->core-setup - - - - -newtonsoft-json901->cli - - - - -newtonsoft-json901->sdk - - - - -core-sdk - -core-sdk - - -newtonsoft-json901->core-sdk - - - - -toolset - -toolset - - -newtonsoft-json901->toolset - - - - -core-setup-portable - -core-setup-portable - - -newtonsoft-json901->core-setup-portable - - - - -newtonsoft-json901->fsharp - - - - -linker - -linker - - -corefx - -corefx - - -linker->corefx - - - - -coreclr - -coreclr - - -linker->coreclr - - - - -coreclr-portable - -coreclr-portable - - -linker->coreclr-portable - - - - -corefx-portable - -corefx-portable - - -linker->corefx-portable - - - - -application-insights - -application-insights - - -application-insights->cli - - - - -netcorecli-fsc - -netcorecli-fsc - - -netcorecli-fsc->toolset - - - - -aspnet-extensions - -aspnet-extensions - - -arcade->aspnet-extensions - - - - -arcade->roslyn - - - - -aspnet-xdt - -aspnet-xdt - - -arcade->aspnet-xdt - - - - -aspnetcore-tooling - -aspnetcore-tooling - - -arcade->aspnetcore-tooling - - - - -msbuild - -msbuild - - -arcade->msbuild - - - - -aspnetcore - -aspnetcore - - -arcade->aspnetcore - - - - -arcade->cli - - - - -clicommandlineparser - -clicommandlineparser - - -arcade->clicommandlineparser - - - - -arcade->corefx - - - - -newtonsoft-json - -newtonsoft-json - - -arcade->newtonsoft-json - - - - -arcade->sdk - - - - -standard - -standard - - -arcade->standard - - - - -websdk - -websdk - - -arcade->websdk - - - - -arcade->corefx-portable - - - - -arcade->fsharp - - - - -aspnet-extensions->aspnetcore-tooling - - - - -vstest - -vstest - - -aspnet-extensions->vstest - - - - -core-setup->aspnet-extensions - - - - -core-setup->aspnetcore-tooling - - - - -core-setup->aspnetcore - - - - -core-setup->cli - - - - -core-setup->sdk - - - - -core-setup->vstest - - - - -roslyn->aspnet-extensions - - - - -roslyn->core-setup - - - - -roslyn->aspnetcore-tooling - - - - -roslyn->msbuild - - - - -roslyn->aspnetcore - - - - -roslyn->cli - - - - -roslyn->coreclr - - - - -roslyn->core-setup-portable - - - - -roslyn->coreclr-portable - - - - -aspnet-xdt->websdk - - - - -aspnetcore-tooling->aspnetcore - - - - -aspnetcore-tooling->toolset - - - - -msbuild->aspnetcore-tooling - - - - -msbuild->aspnetcore - - - - -msbuild->cli - - - - -nuget-client - -nuget-client - - -msbuild->nuget-client - - - - -msbuild->sdk - - - - -aspnetcore->websdk - - - - -aspnetcore->core-sdk - - - - -cli->toolset - - - - -clicommandlineparser->cli - - - - -templating - -templating - - -clicommandlineparser->templating - - - - -corefx->core-setup - - - - -corefx->msbuild - - - - -corefx->cli - - - - -corefx->core-setup-portable - - - - -nuget-client->cli - - - - -nuget-client->sdk - - - - -newtonsoft-json->core-setup - - - - -newtonsoft-json->cli - - - - -newtonsoft-json->nuget-client - - - - -newtonsoft-json->sdk - - - - -newtonsoft-json->templating - - - - -newtonsoft-json->websdk - - - - -common - -common - - -newtonsoft-json->common - - - - -newtonsoft-json->core-sdk - - - - -newtonsoft-json->toolset - - - - -newtonsoft-json->core-setup-portable - - - - -newtonsoft-json->fsharp - - - - -newtonsoft-json->vstest - - - - -sdk->cli - - - - -standard->core-setup - - - - -standard->cli - - - - -standard->corefx - - - - -standard->core-setup-portable - - - - -standard->corefx-portable - - - - -templating->cli - - - - -websdk->cli - - - - -websdk->toolset - - - - -common->nuget-client - - - - -toolset->core-sdk - - - - -coreclr->core-setup - - - - -coreclr->corefx - - - - -coreclr->core-setup-portable - - - - -coreclr->corefx-portable - - - - -fsharp->toolset - - - - -vstest->toolset - - - - - diff --git a/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-official.png b/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-official.png deleted file mode 100644 index d7074b35b1..0000000000 Binary files a/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-official.png and /dev/null differ diff --git a/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-validate.png b/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-validate.png deleted file mode 100644 index 6b79e044f0..0000000000 Binary files a/Documentation/planning/arcade-powered-source-build/img/rough-pipeline-validate.png and /dev/null differ diff --git a/Documentation/planning/arcade-powered-source-build/img/spec-build-1-general.png b/Documentation/planning/arcade-powered-source-build/img/spec-build-1-general.png deleted file mode 100644 index 57ebd68645..0000000000 Binary files a/Documentation/planning/arcade-powered-source-build/img/spec-build-1-general.png and /dev/null differ diff --git a/Documentation/planning/arcade-powered-source-build/img/spec-build-2-bootstrap.png b/Documentation/planning/arcade-powered-source-build/img/spec-build-2-bootstrap.png deleted file mode 100644 index 01c80d7964..0000000000 Binary files a/Documentation/planning/arcade-powered-source-build/img/spec-build-2-bootstrap.png and /dev/null differ diff --git a/Documentation/planning/arcade-powered-source-build/img/spec-build-3-n-1.png b/Documentation/planning/arcade-powered-source-build/img/spec-build-3-n-1.png deleted file mode 100644 index 81aca09562..0000000000 Binary files a/Documentation/planning/arcade-powered-source-build/img/spec-build-3-n-1.png and /dev/null differ diff --git a/Documentation/planning/arcade-powered-source-build/in-arcade.md b/Documentation/planning/arcade-powered-source-build/in-arcade.md deleted file mode 100644 index 2576fb70a6..0000000000 --- a/Documentation/planning/arcade-powered-source-build/in-arcade.md +++ /dev/null @@ -1,138 +0,0 @@ -# Source-build as a part of the Arcade build - -Source-build does a lot of stuff. A sampler: - -1. Clones the upstream repos to form a layout. -1. Sets up package cache and source-built intermediate output directories. -1. Orchestrates build order. -1. Provides a number of source-build-specific args to build commands. -1. Runs extra targets before and after each build to fill in missing - source-build compatibility, if necessary. -1. Builds tarballs--`.tar.gz` file containing everything necessary to build a - specific version of the SDK from source without prebuilts. -1. Maintains scripts specific to the tarballs to enable scenario like - bootstrapping. -1. Track prebuilt usage to ensure repos don't pull in dependencies not built - from source. -1. Automatically decompile and recompile certain dependencies used as prebuilts - so that they can be maintained as source code even if we don't have the - original source code. - -The main doc discusses why this needs to be done in Arcade. We should also aim -at all of this being doable via a single `./build.sh [...]` command. This allows -source-build to be easily reproducible with a local build, something the project -has been sorely missing and bars all but the most committed and awesome -contributors. - -## Additions to the `build.sh` procedure - -It seems that source-build can fit into the existing Arcade extension points. - -### Tool Restore - -This point is normally used to restore tooling that may impact the ordinary -restore by adding MSBuild targets. Source-build can add a step to get -source-build dependencies, with several options: - -1. Use transport package. Restores the [intermediate nupkg] and sets it up as a - restore source. -2. Bring-your-own dependencies. A set of args allow the user (or automation) to - point to existing intermediates on disk. -3. Build dependencies from source. Instead of downloading intermediates, clone - the repos and build them all. - -Any of those options allows the source-build of the current repo to continue -afterwards with source-built intermediates. Then, source-build sets up -RestoreSources, a Package Version Props, a blob root, and any repo-specific -special requirements. - -At this point in the build, we should also have all we need to create a -source-buildable tarball. We should aim to allow the user to create it as early -as possible and cancel the rest of the build, as they may be a distro maintainer -who doesn't actually want to build the repo right now. Unfortunately, there are -some known issues blocking us from creating a source-buildable tarball without -building the product first, tracked by -[source-build#831](https://github.com/dotnet/source-build/issues/831). Until we -find solutions, tarball creation must be done in Publish instead. - -### Restore - -We need to be careful that no machine-wide caches are used here. Caching a -source-built nupkg may cause unexpected and incredibly hard to detect errors -later on. - -### Build/Pack - -The slew of source-build args normally passed to the build command should be -moved to Arcade as much as possible. Many of these apply to the Build and Pack -parts of the build process. - -### Publish - -We should add a simple target to Publish that ensures no artifacts are published -directly, only the [intermediate nupkg] wrapper. This prevents NuGet feed -pollution/conflict vs. the non-source-built packages. - -### New "AfterSourceBuild" project - -We need a place to put [intermediate nupkg] creation and prebuilt usage -detection. The intermediate nupkg should contain the prebuilt report, so these -steps are related. - -It's important to put prebuilt usage detection as the very last step, to be sure -no prebuilts were used, even by Publish. Not all package restores and artifact -downloads happen during the Arcade SDK's Restore phase. - -Creating an intermediate nupkg involves gathering together all shipped artifacts -(nupkgs and tarballs) into a nupkg. The nupkg the needs to be uploaded to the -build pipeline along with a manifest to indicate to Arcade how to publish it. - -## Supporting source-built SDKs - -We need to build using a source-built Arcade SDK. For bootstrap and N-1 flows, -we also need to use a source-built .NET Core SDK. We can get a source-built SDK -in an [intermediate nupkg]. The problem is that by the time we've restored the -intermediate nupkg, we're already inside a process using a prebuilt SDK, so we -need to launch a new build. A few alternative ways how to do this (best first): - -1. Reclone the current repo somewhere in `artifacts/`, then run `build.sh` with - extra parameters pointing to the restored SDK. - * This seems like a complex way to do it, however it ensures a clean repo and - keeps the logic inside MSBuild code. - * We already need to be able to clone upstreams in `artifacts` for the "build - dependencies from source" option. This can leverage that infrastructure. - The `darc clone` command provided by Arcade should fulfill this role. - * It may not be necessary to create a fresh clone: this can be investigated - as a perf optimization. -2. Have the user first call `./build.sh [...]` with some args that bring down - the new SDK. Then the user calls a second command to use that SDK. - * Not bad in official builds. However, harder to reproduce locally. - * We could have an additional script to automate the two calls. This is - better. We may still need to worry about cleanup logic between the first - and second call. It is still a departure from the norm of just calling - build.sh once. -3. Add functionality to `/eng/common/tool.sh` directly to support intermediate - nupkgs before entering any SDK code. - * Reinvents a lot of NuGet caching/download/restore logic. - * Needs to be implemented in tool.sh and tool.ps1 for parity. - -Option (1) has been implemented in the Arcade SDK, using `git` commands. -[dotnet/arcade#5116](https://github.com/dotnet/arcade/issues/5116) - -## Special per-repo handling - -Some repos may have some infra in source-build that is incompatible with arcade. -Any functionality like this should be added to `eng/SourceBuild.targets`. - -## Non-Microsoft-maintained repositories - -Repos not maintained by Microsoft or the .NET Foundation likely don't use the -Arcade toolset. The simple approach is to fork the repo and add the Arcade -wrapper to allow it to seamlessly take part in the .NET Core SDK source-build. - -A "soft" way to fork is to maintain a repo with a single submodule pointing to -the non-Microsoft repo. This wraps it without forking the original repo's -history. The intent would be very clear. However, it would be more difficult to -maintain patches, if any are necessary to add Arcade. - -[intermediate nupkg]: intermediate-nupkg.md diff --git a/Documentation/planning/arcade-powered-source-build/incremental-official-chunked.md b/Documentation/planning/arcade-powered-source-build/incremental-official-chunked.md deleted file mode 100644 index 64bf081fbe..0000000000 --- a/Documentation/planning/arcade-powered-source-build/incremental-official-chunked.md +++ /dev/null @@ -1,27 +0,0 @@ -# Chunked approach to including source-build in official builds - -Instead of adding source-build to all official builds from the bottom up, it is -possible to separate the repos into chunks. Each chunk is several repos with one -"builder" repo that builds the entire chunk from source. This has some -significant restrictions: - -* The builder must use all repos in the chunk as upstreams. - * The builder needs to know which version to build, so it needs a dependency. -* The repos in a chunk need to be coherent. - * The builder won't build multiple versions of the same repo, so all - references to repos within the chunk need to be on the same version. -* When a downstream repo takes a dependency on any repo in the chunk, it must - have a coherent parent dependency on the builder. - * If the downstream repo takes a dependency on a repo that isn't part of a - coherent build, source-build intermediates won't be available. - -These coherency and upstream restrictions apply to the Microsoft build as well, -because it shares the same dependency graph. This will slow down dependency flow -in some cases. For example, a new package may need to flow through several repos -to maintain coherency rather than being uptaken immediately by a downstream -repo, due to the fake dependencies put in place to make the chunks work. - -## Suggestion: avoid it - -A chunked approach *could* be a backup if it ends up being extremely costly to -implement source-build in every official build. This is not expected. diff --git a/Documentation/planning/arcade-powered-source-build/incremental-official.md b/Documentation/planning/arcade-powered-source-build/incremental-official.md deleted file mode 100644 index 0880edf76d..0000000000 --- a/Documentation/planning/arcade-powered-source-build/incremental-official.md +++ /dev/null @@ -1,47 +0,0 @@ -# Incremental improvements to source-build in official builds - -We can add official source-build to repos one at a time. This incrementally -creates a tree of cached builds. - -## Dependency leaves - -The following diagram is the source-build intermediate output flow in 3.1, with -leaf nodes colored gold and the Core-SDK (dotnet/installer) product at the top: - -> ![Source build leaves](img/official-leaves.svg) -> [source (img/official-leaves.dot)](img/official-leaves.dot) - -To build up the tree of cached builds, source-build must be distributed amongst -the repos. We should do this by repeatedly adding source-build functionality to -leaf repos (removing completed repos from the graph) until every repo runs -source-build. - -> Note: the .NET Core build does have circular dependencies, so technically no -> repos are pure leaves. For example, compiling C# in the leaves above requires -> a .NET Core SDK toolset. Source-build cuts the dependencies as shown above and -> uses a bootstrap flow to avoid prebuilts. The bootstrap flow is out of the -> scope of this incremental official build doc. See ["The speculative SDK" in -> README.md](README.md#the-speculative-sdk). - -## Leaf first - -Reasons to implement ArPow in leaves first: - -* This most closely resembles an actual source-build. Either the repo has no - dependencies, or it gets all its dependencies from upstream ArPow builds. -* If there are differences in the source-built artifacts from the upstreams, - they are exercised. -* Fewer or no prebuilts are used. - -## Exceptions to leaf-first - -Some repos have very high impact to source-build and may introduce unexpected -problems, such as dotnet/runtime. It is useful to implement ArPow in these repos -even if upstreams aren't ready yet. For example, working on dotnet/runtime -before it was a leaf exposed issues with artifact size. - -This makes sense for dotnet/runtime, but not for dotnet/aspnetcore. The behavior -of aspnetcore depends a lot on the non-portable source-build RID, and needs -artifacts to be available specifically for that RID. Building aspnetcore for -`centos.7-x64` requires runtime artifacts with that non-portable RID, which are -only built by source-build. diff --git a/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md b/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md deleted file mode 100644 index cad69f1c75..0000000000 --- a/Documentation/planning/arcade-powered-source-build/intermediate-nupkg.md +++ /dev/null @@ -1,159 +0,0 @@ -# The source-build intermediate NuGet package - -To transfer source-built prebuilts from one repo to the next, we need a -transport format. NuGet packages are a proven method to do so, including -built-in caching, secure restore and push supported by Arcade, and versioning. - -## The monster package - -We can't publish individual source-built NuGet packages (nupkgs) to the feed -directly. They would conflict with Microsoft-built packages with the same IDs -and versions. These packages also have a reasonable risk of accidentally being -used by devs not familiar with source-build, poisoning the NuGet caches on their -systems. - -To address both of these, source-build should assemble a monster nupkg that -contains all the nupkgs and other artifacts produced by the build. - -* `source-build.dotnet-runtime.linux-x64.5.0.0-alpha.1.12345.nupkg/` - * `Microsoft.NETCore.App.Ref.5.0.0-alpha.1.12345.nupkg` - * `Microsoft.NETCore.App.Runtime.linux-x64.5.0.0-alpha.1.12345.nupkg` - * `dotnet-runtime-5.0.0-alpha.1.12345-linux-x64.tar.gz` - * ... - -The version number here is the same as the official build's version number. This -makes the official build produce consistent artifacts and prevents overlap in -the same way it's prevented in existing official builds. The nupkg should always -have an unstable version because it's a transport package, helping to avoid -complicated feed management allowing mutation for stabilized rebuilds. - -Official builds publish the monster nupkg, and downstream repos restore the -monster nupkg then use the contents. Note that if an official build fails to -produce its Microsoft build outputs *or* the source-build monster nupkg, it must -fail altogether. This ensures downstream repos have uninterrupted access to the -source-built intermediates. - -## Too large? - -Some repositories produce artifacts that would be too large for Azure DevOps -feeds to handle if put into a single intermediate package. To solve this, some -repos need to produce "supplemental intermediate nupkgs". - -A "supplemental intermediate nupkg" uses the same naming convention as -intermediate nupkgs, but inserts an extra name between the repo name and the -RID. - -* `Microsoft.SourceBuild.Intermediate.runtime.Libraries.linux.x64/6.0.0-foo` - * Contains the dotnet/runtime platform extensions libraries. - * E.g. `System.IO.Pipelines.6.0.0-foo.nupkg`, - `System.Numerics.Tensors.6.0.0-foo.nupkg`, ... -* `Microsoft.SourceBuild.Intermediate.runtime.Crossgen2Pack.linux.x64/6.0.0-foo` - * Contains the crossgen2 framework pack: - * `Microsoft.NETCore.App.Crossgen2.linux-x64.6.0.0-foo.nupkg` -* `Microsoft.SourceBuild.Intermediate.runtime.Crossgen2Pack.Archive.linux.x64/6.0.0-foo` - * This stores the `tar.gz` version of the crossgen2 framework pack: - * `dotnet-crossgen2-6.0.0-foo-linux-x64.tar.gz` - * Note: Calling it `.Archive` rather than `.Tarball` gives us room to handle - `zip` files, to allow for Windows support in the future. - * Crossgen2 is huge (~167 MB), so we need one supplemental nupkg for each - artifact: one for the nupkg, another for the archive. - -The names of the supplemental categories are based on the contents, but the -exact names are arbitrary. The goal is that it's clear enough that downstream -users will know what it contains. - -For example, it would be fine to give the `Crossgen2Pack.Archive` supplemental -nupkg a name based on the artifact's filename, say, `dotnet-crossgen2`. -Arcade-powered source-build doesn't require a specific naming style. In this -case, making `Crossgen2Pack` and `Crossgen2Pack.Archive` similar seems -reasonable, since these are simply the same bits packaged different ways. Naming -is ultimately up to the project owner. - -In general, the name should be concise: long NuGet package IDs can cause path -length problems. - -### Contents - -Each nupkg should either be < 20 MB, or contain only one artifact. This is a -heuristic to use to decide when to produce supplemental nupkgs and how to split -them up. - -The non-supplemental intermediate nupkg contains: - -* The prebuilt report. -* Any artifacts that are < 20 MB total and that all downstreams use. -* [New!] A list of supplemental intermediate nupkg names produced by the repo. - * This shouldn't be read by any tooling. (Avoid complicated implementation.) - * This makes it easier to figure out what supplemental nupkgs to include when - working on a downstream repo. Just restore the main one and look at the list - to see what to add. - -The supplemental intermediate nupkgs only contain one or more artifacts--no -reports, and no list of supplemental nupkg names. The single *non-supplemental* -nupkg is always restored if *any supplemental* nupkg is restored, so the -report/metadata is always available to someone investigating a downstream build -that uses supplemental nupkgs. There is no need to deliver redundant data. - -The supplemental packages of dotnet/runtime will have approximately these -download sizes: - -Name | Download size (approx) --- | -- -RefPack | 1 MB -RefPack.Archive | 1 MB -RuntimePack | 107 MB -RuntimePack.Archive | 93 MB -HostPack | 47 MB -HostPack.Archive | 47 MB -Crossgen2Pack | 167 MB -Crossgen2Pack.Archive | 167 MB -Libraries | 6 MB -CoreCLR (ilasm/ildasm/testhost) | 6 MB - -For context, some other repositories' intermediate nupkg total download size -that do not need any supplemental nupkgs: - -* Arcade's intermediate nupkg is ~12 MB. -* SourceLink's intermediate nupkg is < 1 MB. -* xliff-tasks's intermediate nupkg is < 100 kB. - -### Alternatives to manually split supplemental nupkgs - -There are a few alternate solutions that aren't suitable: - -* Split just enough for the nupkg to fit on Azure DevOps, and always restore - every supplemental package. - * This is direct workaround for the immediate problem: Azure DevOps has a 500 - MB limit on nupkg size (as of writing), and our intermediate nupkg for - dotnet/runtime is ~1 GB. - * This simplifies the design: the supplemental packages don't need meaningful - names, and there are no extra lines needed in `eng/Version.Details.xml`. - * This creates significant concern for CI performance, because every build - downstream of dotnet/runtime will have to download all of that data, whether - or not it's needed. - * We have an opportunity to avoid a significant perf regression by rejecting - this bare minimum approach. - -* Split artifacts into one artifact per package. - * This absolutely minimizes downloading artifacts that aren't needed. - * This will either: - * Significantly bloat the `eng/Version.Details.xml` file, specifying every - single required artifact. This makes it less maintainable and more brittle - to upstream changes. - * Require much more complicated intermediate nupkg restore logic to figure - out which supplemental intermediate nupkgs are necessary for the build. - * This doesn't seem like it has major benefits vs. establishing a reasonable - splitting heuristic. This is rejected to keep arcade-powered source-build - simple. - -## Addressability with different build args? - -With a generic build cache, it's typically important for the build inputs to be -captured and incorporated in how the cache is addressed later. The build and its -artifacts must be deterministic, or else the distro maintainer ends up with -something different and we've failed to do useful validation. - -However, the source-build intermediate packages are not a generic build cache. A -given official build version number is *never* built twice by Microsoft, let -alone with unpredictably varying arguments. That immutability is automatically -shared by the source-build official build. diff --git a/Documentation/planning/arcade-powered-source-build/source-build-in-pipeline.md b/Documentation/planning/arcade-powered-source-build/source-build-in-pipeline.md deleted file mode 100644 index 246df6e6c4..0000000000 --- a/Documentation/planning/arcade-powered-source-build/source-build-in-pipeline.md +++ /dev/null @@ -1,47 +0,0 @@ -# Source-build's place in a build pipeline - -For each repo, source-build must occur in the same official build pipeline that -produces the Microsoft product. This is needed to guarantee the version is the -same and source-built intermediates are available for uptake by downstream -repos. - -## Official build - -> ![Official builds](img/rough-pipeline-official.png) -> Blue: Existing -> Black: Source-build additions. - -In official builds, it is important for source-build to be reliable, and not -extend the build time. This means only the necessary steps are performed to -produce the source-build intermediates: minimal to no validation. - -The source-build job produces artifacts that are published by the Arcade stages, -like any other artifacts. This will be facilitated by yaml templates in Arcade, -but may require action for each repo to add. - -The official build also generates source-buildable tarballs for distro -maintainers, for quick and easy access. These are only needed for the Core-SDK -repo in particular. A full SDK is necessary to be buildable from source via the -bootstrap flow, so there is no significant benefit to producing tarballs in -other official builds. - -## PR validation and rolling builds - -> ![Pipeline validation](img/rough-pipeline-validate.png) -> Blue: Existing. -> Black: Basic source-build. Always validate. -> Green: Extra validation steps (necessary coverage per repo varies). - -There are a variety of extra ways to validate source-build. For example, -speculatively building an SDK with the current PR's changes (see -[speculative-build.md](speculative-build.md)), testing the bootstrap flow, etc. - -Depending on the value each type of coverage adds to a specific repo, the extra -validation steps need to be configured. They may change depending on the risk of -certain breaks per repo, the state of the release cycle, the set of changed -files, or manual contributor request. For example, a `master` branch is likely -to hit API breaks frequently and not work speculatively, but a servicing branch -may find value in always confirming there are no such breaks. - -For rolling builds and PR validation, there may be extra pipelines to run these -steps, to avoid interfering with dev workflows. diff --git a/Documentation/planning/arcade-powered-source-build/speculative-build.md b/Documentation/planning/arcade-powered-source-build/speculative-build.md deleted file mode 100644 index 7c198954ff..0000000000 --- a/Documentation/planning/arcade-powered-source-build/speculative-build.md +++ /dev/null @@ -1,56 +0,0 @@ -# Speculative build: what would an SDK look like with my changes? - -As a dev working on a repo upstream from the SDK, it would be desirable to be -able to build a full SDK that incorporates my changes to see if the SDK's -behavior is changed as expected. It is also convenient to test changes in a full -end-to-end environment. - -This has additional uses in the context of source-build. In order to build from -source, we need to be able to bootstrap: build a .NET Core SDK, then build the -entire product again using that .NET Core SDK. Without speculatively building an -SDK, bootstrapping can only be tested in the Core-SDK repository. In some cases, -this is too late to actually fix any problems that are encountered. We need to -push validation for this scenario as far upstream as we can. - -Speculative builds are complex. They have been considered before for the -Microsoft build, but haven't been done. It may be significantly easier to do in -source-build, because the entire build can happen on a single machine, rather -than every supported platform (requiring transporting artifacts across -machines). - -## How to orchestrate a speculative build - -> ![Speculative build](img/spec-build-1-general.png) - -The red circle is the current build. For example, dotnet/aspnetcore on `master`. -The black circles are the builds that were used for intermediates. - -First, we need to choose a speculative root build. We choose Core-SDK because it -produces the SDK, and the latest version because it's most likely what we want -to test against. This results in the blue graph. Note that because we have to -make a choice here, there are many possible speculative builds we could perform, -and they might not all work! - -Then, we replace all the nodes in the speculative root graph with our local -build graph nodes, based on repository identity. This forms the third graph in -the diagram. - -Finally, we rebuild the remaining nodes in the speculative graph, using our -local build outputs. This gives us a speculative SDK as an output. - -## Validation: speculative bootstrap - -> ![Speculative bootstrap](img/spec-build-2-bootstrap.png) -> (The big arrow means *from* and/or *to* many nodes.) - -We can use the output SDK and intermediates from our bootstrap build to try to -build the product again, using the same set of repositories/commits. If this is -successful, we have validated the bootstrap build scenario. - -## Validation: speculative N-1 - -> ![Speculative N-1](img/spec-build-3-n-1.png) -> (The big arrow means *from* and/or *to* many nodes.) - -If we instead use the output SDK and intermediates from an N-1 build, instead of -building the local speculative graph, we validate the N-1 scenario.