Start migrating Git for Windows' snapshot builds to GitHub Actions #109
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In 6224fe2 (git-artifacts: force-rebuilding the v2.48.0-rc1 package, 2024-12-31), I had to special-case the v2.48.0-rc1 release. This release has been published long ago, therefore we can drop that ugly hack. Signed-off-by: Johannes Schindelin <[email protected]>
It makes little sense to run these GitHub workflows anywhere but in the `git-for-windows` org; In fact, it may cause mayhem if attempting to run them elsewhere, so let's prevent that footgun. Signed-off-by: Johannes Schindelin <[email protected]>
Logically, `downloadAndUnZip()` does three things: - generate a temporary file path - download a file to that path - unzip the downloaded file and delete it Let's refactor this function to make this separation explicit, and individually reusable. Signed-off-by: Johannes Schindelin <[email protected]>
Some GitHub REST API calls will strangely think that you prefer a JSON when you want to get, say, a release asset... unless you specifically ask for a binary download. Tsk, tsk. Signed-off-by: Johannes Schindelin <[email protected]>
It requires an amazingly unintuitive process to do something as commonplace as downloading release assets. That's why I originally preferred to let the GitHub CLI deal with it an be done with it. However, I want to avoid the necessity to have that GitHub CLI installed on self-hosted runners, so let's reimplement this using the plain (or actually, not so plain, as indicated above) REST API. Note that I still punt on replacing the `curl` call; It strikes me as not so bad, given that a `curl` version that seems good enough to handle my request is installed on all-but-ancient Windows versions by default (see https://curl.se/windows/microsoft.html for details). Signed-off-by: Johannes Schindelin <[email protected]>
This topic branch contains preparatory patches for the huge task of
adapting Git for Windows' automation to build embargoed releases.
So what _are_ embargoed releases?
Every once in a while (seemingly on a semi-annual cadence),
security-relevant bugs are discovered in Git, reported responsibly to
the git-security mailing list, and if the core Git developers deem it
important enough of an issue, patches are developed in secret ("under
embargo"). These patches are then reviewed (typically even less
enthusiastically than on the Git mailing list because much more is at
stake than the question whether the lines are indented correctly or
whether the lines have trailing whitespace), and eventually they are
tagged (at least if consensus is reached) and then the task begins for
Git for Windows to provide installers, portable Gits and MinGit versions
well in advance of the publication date ("the lifting of the embargo"),
a date that has to be negotiated between all affected parties (not a
small feat, I can tell ya).
In Git for Windows, I call these artifacts "embargoed build". They are
distributed to the stakeholders such as GitHub Desktop and Visual Studio
in advance of the publication date so that they can test, and integrate
them into their own embargoed builds.
Previously, embargoed builds were built in an Azure Pipeline, in a
private Azure DevOps project. This has become a bit of a problem in the
meantime, for multiple reasons:
- The upstream Git project (which enjoyed very active contributions from
up to 8 GitHub engineers at its peak) shifted CI builds from Azure
Pipelines to GitHub Actions a long time ago. Even though investment
into GitHub Actions seems to have been drastically reduced recently,
there is too much momentum and the last remnants of Azure Pipelines
support has been removed, all but forcing Git for Windows to invest in
a private org on GitHub to continue building confidence in embargoed
builds by running the CI builds there.
This has implications also on _building_ the embargoed versions, as it
is better to build where you test.
- Even though the hoped-for Windows/ARM64 runners have not materialized
in GitHub Actions yet (Windows/ARM64 runners exist, but they are not
free, not even for open source), Git for Windows invested substantial
efforts (with serious, serious time committment of volunteers, most of
all the never-tiring Dennis Ameling) to side-step this short-coming
and build a custom infrastructure based on creating custom Azure VMs
and registering them as Windows/ARM64 runners.
This support is noticeably lacking from Azure Pipelines, too, and Git
for Windows would have to invest the same time and work to gain the
ability to build Windows/ARM64 binaries in Azure Pipelines.
- It is always cumbersome to maintain two totally different things that
do virtually the same, and Azure Pipelines and GitHub Actions are just
very different even though they started out using the same technology
(it does not seem as if a lot of synergy has happened there as a
consequence, which is a shame).
- For historical reasons, the Azure Pipeline we used was not YAML-based,
and is therefore very hard to transport to other
repositories/projects. In contrast, the GitHub workflows we
established in the Git for Windows projects are very easy to transport
to forks or even transmogrify them into totally different projects as
needed. To allow for the same mobility with the Azure Pipeline, we
would have had to rewrite it in YAML anyway, and it seems a smarter
investment to extend the already-existing GitHub workflows to that
end.
The journey to adapt the GitHub workflows to allow for building and
publishing embargoed artifacts is split into these topic branches (with
the current one being the first one):
- embargoed-builds-preamble:
Some preparatory patches, refactoring, that kind of stuff. This is
what is actually merged in this here merge commit.
- embargoed-branches:
Developing a GitHub workflow that initializes "time-traveling"
branches, i.e. branches that branch off at the time when the version
preceding the embargoed one were done, in the following repositories
(which are all used to build Git for Windows' installers, portable
Gits, MinGits, etc):
- git
- build-extra
- MINGW-packages
- git-sdk-64
- git-sdk-arm64
- git-sdk-32
- embargoed-git-artifacts:
Adapting the `git-artifacts` workflow to allow for running it in a
private GitHub org. This requires some special care because access
tokens are required to access the relevant repositories
- embargoed-tag-git:
Adapting the `tag-git` workflow to allow for running in a private
GitHub org. This workflow is actually the one triggered by the
corresponding slash command, as its build artifacts are required to
run the `git-artifacts` workflow for all supported CPU architectures.
- embargoed-releases:
While the regular, non-embargoed release process of Git for Windows
calls for the build artifacts of the `git-artifacts` workflow runs to
be gathered and released e.g. to a GitHub release in
`git-for-windows/git`, the same is not possible in embargoed releases.
The problem is not even that the build artifacts would have to be
retrieved from a different org, using different access tokens than the
GitHub release to which they need to be uploaded, the biggest problem
is the time factor: While regular releases have a very short time
window between building the artifacts and releasing them, in embargoed
releases sometimes months have to pass before the build artifacts can
be released, and GitHub's retention policies do not allow for such
long time windows.
As a consequence, a lot of mechanics are necessary to refactor some of
the processing that would historically be done during the `release`
workflow to allow for it to be run waaaaay before the actual release.
And to counter the retention rules, all embargoed build artifacts are
uploaded to a GitHub release in the private org, where retention rules
do not apply.
One step is missing: Actually releasing the embargoed artifacts to a
_public_ GitHub Release in git-for-windows/git, to NuGet.org and to Git
for Windows' Pacman repositories, as well as updating the `build-extra`
and `git-for-windows.github.io` repositories with the information that
is only available once that GitHub Release is available. The reason that
this is missing is that I ran out of steam doing it during the (quite
stressful) release process of v2.47.2 & friends.
Signed-off-by: Johannes Schindelin <[email protected]>
…indows versions When embargoed Git for Windows versions (or backport versions) need to be prepared, a couple of branches need to be created (internally nicknamed "time-traveling branches"), to pick back up at the precise revisions from where the preceding version was built. That way, the embargoed version will only differ in the fixed Git artifacts, and avoid shipping with updates in other components that might cause risk of regressions. This new workflow prepares these branches. Signed-off-by: Johannes Schindelin <[email protected]>
Since v2.47.1, Git for Windows supports Windows/ARM64. Signed-off-by: Johannes Schindelin <[email protected]>
Sometimes, Git for Windows has to go alone, and release several versions based on the same upstream version. In these instances, a version format like v2.47.0(2) is used. Let's support those, too. Signed-off-by: Johannes Schindelin <[email protected]>
MinGit backports are similar to embargoed Git for Windows versions, but more limited in scope, and quite often, not even embargoed: Git for Windows is only ever released for the latest major Git version and its bugfix versions, whereas MinGit caters to 3rd party applications that often choose to stay behind a major version or three. Signed-off-by: Johannes Schindelin <[email protected]>
This topic branch introduces a GitHub workflow that initializes the branches in the relevant repositories which are needed to build embargoed artifacts for a security release of Git for Windows. Signed-off-by: Johannes Schindelin <[email protected]>
As part of the snapshot builds via the `git-artifacts` workflow, we try to import the tag from a bundle. This uncovered a bug that is over 17 years old, and which I am trying to fix via gitgitgadget/git#1857. The symptom is that just after fetching the refs, when looking through the fetched revisions whether a recursive fetch is needed, Git fails to `mmap()` the newly-imported packfile. The symptom looks like this: From [...]/git.bundle * tag <tag> -> FETCH_HEAD * [new tag] <tag> -> <tag> fatal: mmap: could not determine filesize Curiously, this only happens on 32-bit Windows, not on 64-bit Windows, nor on Linux or macOS. The explanation is most likely to be found in how quickly file descriptor values are used again after closing them, which would appear to be _really_ quickly on i686 Windows. Be that as it may, to work around this issue, we simply avoid any operation that would need to access the just-imported packfile in `git fetch <bundle>`. Signed-off-by: Johannes Schindelin <[email protected]>
Signed-off-by: Johannes Schindelin <[email protected]>
This is another step in the migration from the Azure Pipeline that produced Git for Windows' snapshot builds and the Azure Release Pipeline that then uploaded them to Azure Blobs. With this new GitHub workflow, all we need are the successful `git-artifacts` runs (one per CPU architecture), and the workflow will upload them to the new snapshot location reachable via https://gitforwindows.org/git-snapshots. Signed-off-by: Johannes Schindelin <[email protected]>
mjcheetham
approved these changes
Jan 28, 2025
Member
mjcheetham
left a comment
mjcheetham
left a comment
There was a problem hiding this comment.
Thanks for the thorough description and splitting the commits into small pieces :)
dscho
added a commit
to dscho/git-for-windows-automation
that referenced
this pull request
Feb 5, 2025
For almost eight years, Git for Windows' snapshots were hosted at https://wingit.blob.core.windows.net/files/index.html, i.e. on Azure Blobs. As of a combination of PRs [*1*], [*2*], snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted on GitHub [*3*], with the main page being hosted on GitHub Pages [*4*]. Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. References: *1*: git-for-windows#109 *2*: git-for-windows/gfw-helper-github-app#117 *3*: https://github.com/git-for-windows/git-snapshots/releases/ *4*: https://github.com/git-for-windows/git-snapshots/commits/gh-pages Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to dscho/git-for-windows-automation
that referenced
this pull request
Feb 5, 2025
For almost eight years, Git for Windows' snapshots were hosted at https://wingit.blob.core.windows.net/files/index.html, i.e. on Azure Blobs. As of a combination of PRs [*1*], [*2*], snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted on GitHub [*3*], with the main page being hosted on GitHub Pages [*4*]. Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. References: *1*: git-for-windows#109 *2*: git-for-windows/gfw-helper-github-app#117 *3*: https://github.com/git-for-windows/git-snapshots/releases/ *4*: https://github.com/git-for-windows/git-snapshots/commits/gh-pages Signed-off-by: Johannes Schindelin <[email protected]>
4 tasks
dscho
added a commit
to dscho/build-extra
that referenced
this pull request
Feb 5, 2025
For almost eight years, Git for Windows' snapshots were hosted at https://wingit.blob.core.windows.net/files/index.html, i.e. on Azure Blobs. As of a combination of PRs [*1*], [*2*], snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted on GitHub [*3*], with the main page being hosted on GitHub Pages [*4*]. Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. References: *1*: git-for-windows/git-for-windows-automation#109 *2*: git-for-windows/gfw-helper-github-app#117 *3*: https://github.com/git-for-windows/git-snapshots/releases/ *4*: https://github.com/git-for-windows/git-snapshots/commits/gh-pages Note that technically, we could alternatively delete the `render_release_notes_and_mail` function from `please.sh`, as it was only used in the Azure Pipeline that is no longer triggered to build the snapshots. Let's leave the code there for a bit longer, though. Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to git-for-windows/git-for-windows.github.io
that referenced
this pull request
Feb 5, 2025
For almost eight years, Git for Windows' snapshots were hosted at https://wingit.blob.core.windows.net/files/index.html, i.e. on Azure Blobs. As of a combination of PRs [*1*], [*2*], snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted on GitHub [*3*], with the main page being hosted on GitHub Pages [*4*]. Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. References: *1*: git-for-windows/git-for-windows-automation#109 *2*: git-for-windows/gfw-helper-github-app#117 *3*: https://github.com/git-for-windows/git-snapshots/releases/ *4*: https://github.com/git-for-windows/git-snapshots/commits/gh-pages Signed-off-by: Johannes Schindelin <[email protected]>
dscho
added a commit
to dscho/git
that referenced
this pull request
Feb 5, 2025
The `SECURITY.md` document mentions Git for Windows' snapshots and helpfully provides a link. For almost eight years, these snapshots were hosted at https://wingit.blob.core.windows.net/files/index.html, i.e. on Azure Blobs. As of a combination of PRs [*1*], [*2*], snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted on GitHub [*3*], with the main page being hosted on GitHub Pages [*4*]. Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. References: *1*: git-for-windows/git-for-windows-automation#109 *2*: git-for-windows/gfw-helper-github-app#117 *3*: https://github.com/git-for-windows/git-snapshots/releases/ *4*: https://github.com/git-for-windows/git-snapshots/commits/gh-pages Signed-off-by: Johannes Schindelin <[email protected]>
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Feb 6, 2025
As described in commit 8d456a7 of PR git-lfs#5879, we currently depend on the Git for Windows SDK to install a version of the "prove" command from Perl Test::Harness distribution, because we use that command to run our suite of shell tests. However, in our CI jobs on Windows, the SDK does not at present install the "prove" command into a location listed in the PATH environment variable, so these CI jobs having begun failing. This change may be related to the recent migration of the SDK's release artifacts to GitHub from Azure Blobs, as outlined in PR git-for-windows/git-for-windows-automation#109, and is possibly a consequence of PR git-for-windows/build-extra#588. Regardless, the "prove" command is fortunately still included in the "minimal" flavour of the SDK, since it is listed in the manifest for that flavour: https://github.com/git-for-windows/git-sdk-64/blob/f845bb725e56e058a0fbf93aba18c70906a1068e/.sparse/minimal-sdk#L188 We simply need to ensure that the directory in which the command is now located, namely /usr/bin/core_perl, is added to the set of paths in the PATH variable for the Git Bash environment. Note that we only need to add this extra path to PATH when we are running our script/cibuild script, as that runs the default recipe from our t/Makefile file, which in turn executes the "test" recipe, which uses the "prove" command to run and summarize all our shell test scripts.
chrisd8088
added a commit
to chrisd8088/git-lfs
that referenced
this pull request
Feb 6, 2025
As described in commit 8d456a7 of PR git-lfs#5879, we currently depend on the Git for Windows SDK to install a version of the "prove" command from the Perl Test::Harness distribution, because we use that command to run our suite of shell tests. However, in our CI jobs on Windows, the SDK does not at present install the "prove" command into a location listed in the PATH environment variable, so these CI jobs having begun failing. This change may be related to the recent migration of the SDK's release artifacts to GitHub from Azure Blobs, as outlined in PR git-for-windows/git-for-windows-automation#109, and is possibly a consequence of PR git-for-windows/build-extra#588. Regardless, the "prove" command is fortunately still included in the "minimal" flavour of the SDK, since it is listed in the manifest for that flavour: https://github.com/git-for-windows/git-sdk-64/blob/f845bb725e56e058a0fbf93aba18c70906a1068e/.sparse/minimal-sdk#L188 We simply need to ensure that the directory in which the command is now located, namely /usr/bin/core_perl, is added to the set of paths in the PATH variable for the Git Bash environment. Note that we only need to add this extra path to PATH when we are running our script/cibuild script, as that runs the default recipe from our t/Makefile file, which in turn executes the "test" recipe, which uses the "prove" command to run and summarize all our shell test scripts.
dscho
added a commit
to git-for-windows/git
that referenced
this pull request
Feb 6, 2025
The `SECURITY.md` document mentions Git for Windows' snapshots and helpfully provides a link. For almost eight years, Git for Windows' snapshots were hosted [on Azure Blobs](https://wingit.blob.core.windows.net/files/index.html). As of a combination of PRs (git-for-windows/git-for-windows-automation#109, git-for-windows/gfw-helper-github-app#117), snapshots are not only now built and deployed via GitHub Actions instead of Azure Pipelines (and ARM64 artifacts are now included, too), they are also hosted [on GitHub](https://github.com/git-for-windows/git-snapshots/releases/), with the main page being hosted [on GitHub Pages](https://github.com/git-for-windows/git-snapshots/commits/gh-pages). Therefore, the original link now redirects to a new location (which is also a lot easier to remember): http://gitforwindows.org/git-snapshots. Let's adjust the link to link there directly. This is a companion PR of git-for-windows/git-for-windows-automation#111 and of git-for-windows/build-extra#589.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In 2017, I started working on the
GitArtifactsAzure Pipeline to not only produce Git for Windows' artifacts (such as installer, portable Git, MinGit, etc) for full releases, but basically on every push. The resulting artifacts were uploaded to Azure Blobs and for convenience, an index of all of those snapshots was maintained at https://wingit.blob.core.windows.net/files/index.html.This all depends on Azure Blob storage that is hosted in my personal Azure account.
I recently stumbled over yet another excellent letter to open-source maintainers that had this great piece of advice:
In an attempt to start remedying this, I have transferred all of the snapshot artifacts to releases at https://github.com/git-for-windows/git-snapshots/releases/, and moved the index to https://gitforwindows.org/git-snapshots/.
Then, I triggered a
tag-gitrun, which upon completion (thanks to the GitForWindowsHelper GitHub App) triggered a new set ofgit-artifactsruns (i686, x86_64 and aarch64). There was a little bit of a hiccup in the previously un-tested snapshots-only part of the i686 run that uncovered a really old bug (which I am trying to fix in gitgitgadget/git#1857) which I had to work around and then trigger a new i686 run.After that, I finally could start working on a new GitHub workflow called
upload-snapshot, which is presented here. It takes the three run IDs of the i686, x86_64 and aarch64 workflow runs as input and takes it from there.As a bonus of moving this process over from Azure Pipelines, where we enjoy even less support for Windows/ARM64 runners than in GitHub Actions, snapshots now also have ARM64 artifacts.
Note that I used some functionality that I had to develop during the recent embargoed Git release phase (because a lot of functionality is missing from GitHub Actions and from GitHub repositories that are needed to do embargoed releases, for example, you cannot keep workflow runs private until a given publication date, the only way to do private builds is to have them done in private repositories, Actions do not work in the private forks associated with security advisories either, and it is an unfortunate consequence that there is no way to move the involved private builds to the public repository; Also, most techniques used in public repositories won't Just Work™️ in private repositories because you have to pass around tokens all of a sudden. Just to name a few complications), and I used the opportunity to sneak them into this here repository (i.e. the public one, not used for embargoed releases, but I definitely want the code here for transparency). There are still quite some patches left, but I have to polish them first so that they don't break non-embargoed releases.
Do also note that this PR is only half of what is required to fully port the Git for Windows snapshots process to GitHub Actions. Since there is no way to trigger workflow runs on pushes to a different repository than where the workflow runs live (yet another thing I miss from Azure Pipelines), the GitForWindowsHelper GitHub App will need to be taught to run the
tag-gitworkflow onpushunless there is already atag-gitcheck run attached to thegit-for-windows/gitcommit. (Note that the difference between workflow run and check run is quite important in this context.) If there already is one, then theupload-snapshotworkflow needs to be triggered (uploading already-released Git for Windows versions, which is the scenario where atag-gitcheck run already exists). And GitForWindowsHelper also needs to learn to react to completedgit-artifactssnapshot check runs ongit-for-windows/git, triggering theupload-snapshotworkflow also in that case. It needs to be careful, though, to verify that this is actually wanted, maybe by having a peek atgit-for-windows/git-artifacts' releases to see whether a corresponding tag already exists, and maybe by doing anaheadcheck to see whether the corresponding commit is actually reachable fromgit-for-windows/git'smainbranch.