|
| 1 | +# .NET Core Release Checklist |
| 2 | + |
| 3 | +<!-- |
| 4 | + To start the checklist for a new release: |
| 5 | + - If it's an internal release, open a new issue in dotnet/core-eng (private repo). |
| 6 | + - Otherwise, open a new issue in dotnet/source-build. |
| 7 | + - Paste one copy of the below checklist per release. (Once for 2.1, again for 3.1, if both are being serviced) |
| 8 | + - Delete lines that only apply to other releases. (E.g. delete [2.1] if doing a 3.1 release.) |
| 9 | + - Delete lines starting with [Internal] if running a non-internal release. |
| 10 | + - Delete lines starting with [Non-Internal] if running an internal release. |
| 11 | +--> |
| 12 | + |
| 13 | +<!-- |
| 14 | + Enter the version tracked by this checklist. |
| 15 | +--> |
| 16 | +# {runtime-version} / {sdk-version} |
| 17 | + |
| 18 | +1. - [ ] Create a page at ".NET Core A&D" OneNote -> "Servicing" section -> "Retrospective" -> "{Month} {YYYY}" to take ongoing notes on problems with the process, workarounds, and fixes. |
| 19 | + - This is useful to make sure context is available to review later. It may end up blank if the release goes very smoothly. |
| 20 | + - There are other notes on servicing in this OneNote. It may be useful to review if something goes wrong to see if it's been fixed before. |
| 21 | + - [Non-Internal] File GitHub issues as you encounter problems, and link to them from the notes. Provide info in the issue rather than in the notes. |
| 22 | +1. - [ ] [Internal] Ensure internal/release/X.Y branch is up to date with mirrored release/X.Y branch. |
| 23 | +1. - [ ] Update to new version. |
| 24 | + - [2.1] [/Documentation/servicing/update-2.1.md](https://github.com/dotnet/source-build/tree/release/3.1/Documentation/servicing/update-2.1.md) |
| 25 | + - [3.1] [/Documentation/servicing/update-3.1.md](https://github.com/dotnet/source-build/tree/release/3.1/Documentation/servicing/update-3.1.md) |
| 26 | + - Start a new working branch and PR for WIP so others can easily look if needed and CI starts running. |
| 27 | + - [Internal] For internal builds, use <https://dev.azure.com/dnceng/internal/_git/dotnet-source-build>, branch name `internal/release/<SDK-version>-<month name><YYYY>`. |
| 28 | +1. - [ ] Version update cleanup and confirmations |
| 29 | + - [2.1] |
| 30 | + - [ ] Verify that `source-build/ProdConFeed.txt` contains the latest feed required for the release. |
| 31 | + - Build number should match ProdCon build number. |
| 32 | + - Ensure auth details are **not** included in this file. |
| 33 | + - [ ] Verify/update the ASP.NET versions (`MicrosoftAspNetCoreAllPackageVersion` and `MicrosoftAspNetCoreAppPackageVersion`) in dependencies.props. |
| 34 | + - [ ] Verify/update the `OfficialBuildId`s in updated repos (usually coreclr, corefx, core-setup). |
| 35 | + - [ ] Update the `DotnetCLIVersion.txt` SDK version to N-1: the latest released "2.1.XYY" SDK version, where "2.1.X" matches the SDK we're building. |
| 36 | + - [3.1] |
| 37 | + - [ ] Verify that the `PrivateSourceBuiltArtifactsPackageVersion` in `eng/Versions.props` matches N-1 release artifacts. |
| 38 | + - [ ] Update the `global.json` SDK version to N-1: the latest released "3.1.XYY" SDK version, where "3.1.X" matches the SDK we're building. |
| 39 | +1. - [ ] [2.1] Fetch internal git data for submodules. |
| 40 | + 1. Run `git submodule update --init --recursive` and see errors due to missing commits. |
| 41 | + 1. Run `fetch-vsts-commits.sh` with a PAT to automatically fetch internal refs. |
| 42 | + 1. Run `git submodule update --init --recursive` again to update failed submodules. |
| 43 | + * If some refs are still missing on unusual repos, those repos may need to be set up in `fetch-vsts-commits.sh`. Note: Repos in `fetch-vsts-commits.sh` don't need to be cleaned up. |
| 44 | +1. - [ ] Regenerate patches as necessary. |
| 45 | + - Run `./build.sh` to begin clone and patch application and see what, if anything, needs to be regenerated. Issue to track re-patching workflow: [#1468](https://github.com/dotnet/source-build/issues/1468) |
| 46 | + - [3.1] ASP.NET will always need to be regenerated because it contains a version number that needs to be updated. The version should be updated to the SDK version being built. See https://github.com/dotnet/source-build/blob/release/3.1/patches/aspnetcore/0007-Fix-version-number.patch |
| 47 | +1. - [ ] [Internal] [3.1] Update internal feed list in `.\build.sh` |
| 48 | + - Internal feeds are listed under `VSS_NUGET_EXTERNAL_FEED_ENDPOINTS` |
| 49 | + - Make sure to keep `dotnet3.1-internal-transport` in the list |
| 50 | + - Issue tracking this: [#1674](https://github.com/dotnet/source-build/issues/1674) |
| 51 | +1. - [ ] Start building locally (for quick diagnosis) and in CI (for super clean build) and iterate towards a green build. |
| 52 | + - [Internal] [3.1] Set environment variable `internalPackageFeedPat` to dnceng internal PAT with package read and code read access. |
| 53 | + - E.g. run `export internalPackageFeedPat;read internalPackageFeedPat` and paste the PAT. This sets up your running shell for an authenticated build. Passing `/p:` isn't sufficient. |
| 54 | + - [Internal] [3.1] Also pass arg `/p:AzDoPat=$internalPackageFeedPat` for darc clone in particular. |
| 55 | + - [Internal] [2.1] Pass auth arg `/p:ProdConBlobFeedUrlPrefix=<url up to and not including 'orchestrated-release-2-1/' from feed.txt>` |
| 56 | + - [Internal] [2.1] When running smoke-test locally, also pass `/p:ProdConBlobFeedUrlPrefix=<...>` |
| 57 | + - The smoke-test command looks like `./build.sh /t:RunSmokeTest /p:Configuration=Release /p:ProdConBlobFeedUrlPrefix=***`. |
| 58 | + - Update baseline from the CI job, using the `centos71 Offline` artifacts. |
| 59 | + - `Logs centos71 Offline/source-build/logs/artifacts/prebuilt-report/generated-new-baseline.xml` |
| 60 | + - => `tools-local/prebuilt-baseline-online.xml` |
| 61 | + - `Logs centos71 Offline/tarball/logs/artifacts/prebuilt-report/generated-new-baseline.xml` |
| 62 | + - => `tools-local/prebuilt-baseline-offline.xml` |
| 63 | +1. - [ ] Review prebuilt baseline diff and iterate builds to drive to zero. |
| 64 | + - Find an explanation for each change in each (online/offline) baseline. |
| 65 | + - If possible, figure out the cause and fix. |
| 66 | + - Discuss with repo or code owner if the fix is a new patch. |
| 67 | + - This may require adding packages to dotnet/source-build-reference-packages. |
| 68 | + - If the baseline change is due to the change in patch version, make sure there's an issue tracking fixing the instability, or start tracking it. |
| 69 | + - If in doubt, talk to the team. Someone might have already encountered a similar diff in the past. |
| 70 | +1. - [ ] Complete manual validation steps |
| 71 | + - [ ] Ask for confirmation on poison report. |
| 72 | + - Confirm with team how to check this for a specific build/release. |
| 73 | + - [ ] [3.1] Verify the packs included in the SDK are from source-build-reference-packages, not source-build. |
| 74 | + - E.g. in `testing-smoke/builtCli`, run `find . -iname System.IO.Pipelines.dll -exec sha256sum {} \; | sort` and look at file origin based on checksum of the file in the SDK. |
| 75 | + - [ ] [3.1] Verify the packs included in the SDK have XML documentation comments. Presence of XML files in the SDK dir is adequate. Issue tracking automation of this: [#1579](https://github.com/dotnet/source-build/issues/1579) |
| 76 | + - [ ] Re-validate the SHA1s in Versions file with the manifest in VSU share dir. |
| 77 | + - [ ] [3.1] Review the file list diff between Microsoft-built SDK and source-built SDK in smoke-test output. |
| 78 | + - [Internal] [3.1] For internal builds, smoke-test fails to acquire the Microsoft-built SDK. Manually acquire the Microsoft-built SDK from the build email you should have received, then run `scripts/compare-builds.sh` yourself to get the output to review. |
| 79 | + - [#1631](https://github.com/dotnet/source-build/issues/1631) tracks a fix. |
| 80 | + - Use <https://gist.github.com/omajid/9111a9310d452b3594b770d4ca8a3d87> as a loose baseline. |
| 81 | + - [#1630](https://github.com/dotnet/source-build/issues/1630) tracks a checked-in baseline. |
| 82 | +1. - [ ] Add team members as reviewers to the PR. Get acceptance. |
| 83 | + - Note: The PR is internal for internal builds and on GitHub for non-internal builds. |
| 84 | + - [Non-Internal] Merge the PR when green and accepted. |
| 85 | + - [Internal] Don't merge the PR. |
| 86 | +1. - [ ] Coordinate with team and complete manual distributed smoke-testing. Suggested assignment below, but it's flexible. Send email with direct links to artifacts from green PR. |
| 87 | + - [2.1] |
| 88 | + - [ ] RHEL7 and RHEL8 VMs - crummel |
| 89 | + - [ ] RHEL docker - dleeapho |
| 90 | + - [ ] Fedora30 - dseefeld |
| 91 | + - [ ] Fedora31 - dagood |
| 92 | + - [ ] Debian9 - crummel |
| 93 | + - [3.1] |
| 94 | + - [ ] RHEL 7 & 8 VM - crummel |
| 95 | + - [ ] RHEL Docker - dleeapho |
| 96 | + - [ ] Fedora30 - dseefeld |
| 97 | + - [ ] CentOS 7 - dagood |
| 98 | + - [ ] CentOS 8 - dseefeld |
| 99 | + - [ ] Debian - dagood |
| 100 | +1. - [ ] [Internal] Download the artifacts from the green PR validation build onto a machine with Bash. |
| 101 | + - We may require only a subset of the artifacts for certain releases: refer to the previous release email thread corresponding to 2.1 vs. 3.1 build. |
| 102 | + - Can use <https://github.com/dotnet/source-build/blob/release/3.1/scripts/fetch-azdo-artifact-tarball.sh> as a library. |
| 103 | + - E.g. download the tarballs with `download_tarball`, and fix them up with `fix_azdo_tarball` if necessary. |
| 104 | +1. - [ ] [Internal] Rename tarball and smoke-test prereqs to human-friendly names. (Can use `rename_tarball_inner_dir`.) |
| 105 | + - Refer to previous release's email threads for naming pattern. |
| 106 | +1. - [ ] [Internal] Upload tarballs and smoke-test prereqs to blob storage. |
| 107 | +1. - [ ] [Internal] Send the tarball to partners. Include info about how certain we are that this will be the final Microsoft build. |
| 108 | + - Never overwrite a tarball. At least change the blob storage virtual dir to represent a new build. This can help avoid timing issues and make it more obvious if stale links were accidentally re-sent rather than new ones. |
| 109 | +1. - [ ] SYNC POINT: Wait for Microsoft build release. |
| 110 | + - Wait for dotnet/announcements post. <kbd>Watch</kbd> the repo to get emails so you don't have to poll. |
| 111 | + - Example: <https://github.com/dotnet/announcements/issues/160>. |
| 112 | +1. - [ ] [Internal] [3.1] In a dev branch on your GitHub fork of source-build, clean up. |
| 113 | + - [ ] [Internal] [3.1] Convert internal URIs to public in `eng/Version.Details.xml`. |
| 114 | + - `https://dev.azure.com/dnceng/internal/_git/<org>-<repo>` => `https://github.com/<org>/<repo>` |
| 115 | + - [ ] [Internal] [3.1] Remove internal package feeds from source-build. |
| 116 | + - `/NuGet.Config` |
| 117 | + - `/smoke-testNuGet.Config` |
| 118 | + - Do not remove from subrepos. The build infra removes these sources automatically while modifying each subrepo nuget.config file. |
| 119 | +1. - [ ] [Internal] Push the servicing branch to your GitHub fork and submit a source-build PR. |
| 120 | +1. - [ ] [Internal] If a build hits `fatal: reference is not a tree: cb5f173b`, [make the commit available on a fork and switch to it temporarily](https://github.com/dotnet/source-build/tree/release/3.1/Documentation/servicing/make-commit-available.md). |
| 121 | +1. - [ ] [Internal] Add source-build team as reviewers. |
| 122 | +1. - [ ] [Internal] When CI is green and two reviewers approve, merge. |
| 123 | + - Avoid squash/rebase: nice to preserve commit hashes. However, there are no known dependencies on *source-build* commits being preserved. |
| 124 | +1. - [ ] Create and push tags on the post-merge commit. [/Documentation/servicing/tagging.md](https://github.com/dotnet/source-build/tree/release/3.1/Documentation/servicing/tagging.md) |
| 125 | +1. - [ ] Create a GitHub release on the tag with SDK versioning. |
| 126 | + - Go to <https://github.com/dotnet/source-build/tags> |
| 127 | + - Expand the SDK-versioned tag's message and copy it |
| 128 | + - Click <kbd>...</kbd> on the right on the SDK-versioned tag and press "Create release" |
| 129 | + - Paste into the release title |
| 130 | + - Click submit. |
| 131 | +1. - [ ] Notify distro maintainers and partners about the release tags |
| 132 | +1. - [ ] [3.1] Download merged Private.SourceBuilt.Artifacts.XX.tar.gz files from the prepare-artifacts leg in source-build-rolling-CI. |
| 133 | +1. - [ ] [3.1] Upload merged Private.SourceBuilt.Artifacts.XX.tar.gz file to dotnetcli account source-built-artifacts blob container. |
| 134 | + - Never overwrite a tarball. Use a sub-patch version if one already exists, e.g. `0.1.0-3.1.105.1`. |
| 135 | +1. - [ ] If there's a respin (for any reason: source-build infra fixes or upstream repo problems): |
| 136 | + - Go back in this process checklist as far as necessary. |
| 137 | + - When tagging, make sure to create a new tag. A sub-patch version tag for SDK and runtime must be generated. E.g - v2.1.300.1-SDK |
| 138 | +1. - [ ] Clean up retrospective notes if necessary. |
0 commit comments