Skip to content

Commit 144535a

Browse files
committed
Avoid using Pipeline Artifacts, except for the minimal flavor
We ran into trouble with the `build-installers` flavor, where the Action will try to download the Azure Pipeline Artifact, get only an incomplete file, try again, and again, until the workflow run times out, see #336 This presented an excellent opportunity to re-think the entire strategy. Recap: so far, we use an Azure Pipeline that is triggered whenever anything was pushed to the Git for Windows SDK's mirror repositories (i.e. Git repositories capturing snapshots of Git for Windows' MSYS2 fork at https://github.com/git-for-windows/git-sdk-64 and .../git-sdk-32 for x86_64 and i686, respectively). This builds the `makepkg-git`, `build-installers` and `full` artifacts, representing various subsets of the files. This here `setup-git-for-windows-sdk` Action, then, finds the latest successful run and downloads the respective build artifact from there. Building these artifacts is not cheap, and downloading them isn't, either, because a `.zip` file is constructed on the fly every time this happens. For this reason, the `setup-git-for-windows-sdk` tries to cache the artifact using GitHub Actions' native @actions/cache mechanism. When this scheme was concocted, partial clones were not really a thing, and GitHub's server side struggled with serving large repositories, which is the reason why Git for Windows historically leaned a bit on Azure Repos (which historically support large repositories better). Fast-forward to 2022, where partial clone _is_ a thing, and performance of shallow and partial clones is quite nice, and parallel checkout also exists, and all of a sudden the original design decisions start making a lot less sense. In a semi-scientific test, then, a different design was tested: Instead of relying on Azure Pipeline artifacts, a bare clone of the respective SDK repository was downloaded, then the `build-extra` repository was cloned, and the SDK artifacts were constructed from scratch. The clones were all shallow, the SDK clone even bare, and to weigh our best options, both partial and full clones of the SDK repositories were used. The timings of one such run can be seen at https://github.com/git-for-windows/build-extra/actions/runs/2475342609: partial full minimal 48s 2m 08s makepkg-git 1m 42s 2m 40s build-installers 4m 35s 4m 52s full 6m 8s 3m 02s build-installers(i686) 2m 56s 3m 28s full(i686) 3m 44s 2m 03s Note: the non-i686 versions include the i686 versions, which is the reason why the i686 versions look so much quicker. These numbers strongly suggest to use the full clones for the full flavor, and the partial clones for all other flavors. These numbers _also_ suggest that the performance of this strategy is pretty much on-par or even better than the current strategy. At https://dev.azure.com/git-for-windows/git/_build?definitionId=34 we can see that the latest Git artifacts Pipeline (which does something equivalent to the current strategy to download the `build-installers` artifact, but does not loop until it times out) takes about seven minutes. The 2m19s illustrated above beat that hands-down. So let's switch strategies to make everything quicker: use partial/full, shallow, bare clones of the SDK, and a shallow clone of https://github.com/git-for-windows/build-extra, and then run the script that generates the SDK artifacts. Note: we do _not_ do the same for the `minimal` flavor. The reason is that there is more to the `git-sdk-64-minimal` Pipeline at https://dev.azure.com/Git-for-Windows/git/_build?definitionId=22 than just building that artifact: Since this artifact is used in all Git for Windows _and also_ all Git CI/PR builds, we *must* avoid introducing breakages due to changes in MSYS2 (this happened a couple of times in the past). To avoid that, this Pipeline not only builds the artifact but then also verifies that it can build a current Git revision and passes the test suite. Therefore, we leave the `minimal` flavor alone, still preferring to download that Pipeline artifact over the (otherwise quite quick!) partial clone/sparse checkout strategy. Signed-off-by: Johannes Schindelin <[email protected]>
1 parent 2b8b6e9 commit 144535a

File tree

3 files changed

+146
-1
lines changed

3 files changed

+146
-1
lines changed

.vscode/settings.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,13 @@
1313
"artifactsize",
1414
"autodrain",
1515
"Backoff",
16+
"bitness",
1617
"mqueue",
1718
"MSYSTEM",
1819
"Pacman",
1920
"unzipper",
2021
"vercel",
22+
"WINDIR",
2123
"winsymlinks"
2224
],
2325
"git.ignoreLimitWarning": true

main.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import {get, mkdirp} from './src/downloader'
33
import {restoreCache, saveCache} from '@actions/cache'
44
import process from 'process'
55
import {spawnSync} from 'child_process'
6+
import {getViaGit} from './src/git'
67

78
async function run(): Promise<void> {
89
try {
@@ -17,7 +18,10 @@ async function run(): Promise<void> {
1718
const verbose = core.getInput('verbose')
1819
const msysMode = core.getInput('msys') === 'true'
1920

20-
const {artifactName, download, id} = await get(flavor, architecture)
21+
const {artifactName, download, id} =
22+
flavor === 'minimal'
23+
? await get(flavor, architecture)
24+
: await getViaGit(flavor, architecture)
2125
const outputDirectory = core.getInput('path') || `C:/${artifactName}`
2226
let useCache: boolean
2327
switch (core.getInput('cache')) {

src/git.ts

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
import * as core from '@actions/core'
2+
import {ChildProcess, spawn} from 'child_process'
3+
import {Octokit} from '@octokit/rest'
4+
import {delimiter} from 'path'
5+
6+
const gitForWindowsUsrBinPath = 'C:/Program Files/Git/usr/bin'
7+
8+
async function clone(
9+
url: string,
10+
destination: string,
11+
verbose: number | boolean,
12+
cloneExtraOptions: string[] = []
13+
): Promise<void> {
14+
if (verbose) core.notice(`Cloning ${url} to ${destination}`)
15+
const child = spawn(
16+
'git.exe',
17+
[
18+
'clone',
19+
'--depth=1',
20+
'--single-branch',
21+
'--branch=main',
22+
...cloneExtraOptions,
23+
url,
24+
destination
25+
],
26+
{
27+
stdio: [undefined, 'inherit', 'inherit']
28+
}
29+
)
30+
return new Promise<void>((resolve, reject) => {
31+
child.on('close', code => {
32+
if (code === 0) {
33+
resolve()
34+
} else {
35+
reject(new Error(`tar: exited with code ${code}`))
36+
}
37+
})
38+
})
39+
}
40+
41+
export async function getViaGit(
42+
flavor: string,
43+
architecture: string
44+
): Promise<{
45+
artifactName: string
46+
id: string
47+
download: (
48+
outputDirectory: string,
49+
verbose?: number | boolean
50+
) => Promise<void>
51+
}> {
52+
const bitness = architecture === 'i686' ? '32' : '64'
53+
const owner = 'git-for-windows'
54+
const repo = `git-sdk-${bitness}`
55+
const artifactName = `${repo}-${flavor}`
56+
57+
const octokit = new Octokit()
58+
const info = await octokit.repos.getBranch({
59+
owner,
60+
repo,
61+
branch: 'main'
62+
})
63+
const id = info.data.commit.sha
64+
core.notice(`Got ID ${id} for ${repo}`)
65+
66+
return {
67+
artifactName,
68+
id,
69+
download: async (
70+
outputDirectory: string,
71+
verbose: number | boolean = false
72+
): Promise<void> => {
73+
core.startGroup(`Cloning ${repo}`)
74+
const partialCloneArg = flavor === 'full' ? [] : ['--filter=blob:none']
75+
await clone(`https://github.com/${owner}/${repo}`, `.tmp`, verbose, [
76+
'--bare',
77+
...partialCloneArg
78+
])
79+
core.endGroup()
80+
81+
let child: ChildProcess
82+
if (flavor === 'full') {
83+
core.startGroup(`Checking out ${repo}`)
84+
child = spawn(
85+
'git.exe',
86+
[`--git-dir=.tmp`, 'worktree', 'add', outputDirectory, id],
87+
{
88+
stdio: [undefined, 'inherit', 'inherit']
89+
}
90+
)
91+
} else {
92+
core.startGroup('Cloning build-extra')
93+
await clone(
94+
`https://github.com/${owner}/build-extra`,
95+
'.tmp/build-extra',
96+
verbose
97+
)
98+
core.endGroup()
99+
100+
core.startGroup(`Creating ${flavor} artifact`)
101+
const traceArg = verbose ? ['-x'] : []
102+
child = spawn(
103+
`${gitForWindowsUsrBinPath}/bash.exe`,
104+
[
105+
...traceArg,
106+
'.tmp/build-extra/please.sh',
107+
'create-sdk-artifact',
108+
`--bitness=${bitness}`,
109+
`--out=${outputDirectory}`,
110+
'--sdk=.tmp',
111+
flavor
112+
],
113+
{
114+
env: {
115+
COMSPEC:
116+
process.env.COMSPEC ||
117+
`${process.env.WINDIR}\\system32\\cmd.exe`,
118+
LC_CTYPE: 'C.UTF-8',
119+
CHERE_INVOKING: '1',
120+
MSYSTEM: 'MINGW64',
121+
PATH: `${gitForWindowsUsrBinPath}${delimiter}${process.env.PATH}`
122+
},
123+
stdio: [undefined, 'inherit', 'inherit']
124+
}
125+
)
126+
}
127+
return new Promise<void>((resolve, reject) => {
128+
child.on('close', code => {
129+
core.endGroup()
130+
if (code === 0) {
131+
resolve()
132+
} else {
133+
reject(new Error(`process exited with code ${code}`))
134+
}
135+
})
136+
})
137+
}
138+
}
139+
}

0 commit comments

Comments
 (0)