Setting up an SDK/Pacman repo for ARM64

[dennisameling]

Now that we can build pretty much all Git for Windows artifacts for ARM64, I'd like to suggest that we set up an ARM64 SDK and/or Pacman repo as a next step.

There's a few places that are currently a bit painful to maintain due to the lack of an SDK/Pacman repo:

• The setup-git-for-windows-sdk GitHub Action is significantly slower on ARM64 because it needs to install a bunch of clangarm64 dependencies every time it runs on ARM64. Caching might be a solution, but having a proper SDK would be even better.
• The git-artifacts CI pipeline needs to download the pre-built dependencies below as they would take a long time to build.

We'll at least need the following packages in this SDK and/or Pacman repo:

• mingw-w64-clang-aarch64-openssl (MINGW-packages)
• mingw-w64-clang-aarch64-curl (MINGW-packages)
• mingw-w64-clang-aarch64-wintoast (build-extra)
• mingw-w64-clang-aarch64-xpdf-tools (MINGW-packages)
• mingw-w64-clang-aarch64-git-credential-manager (build-extra)
• mingw-w64-clang-aarch64-git-lfs (build-extra)
• git-extra (build-extra)

Then the Pacman repo needs to be added to /etc/pacman.conf (like here).

Curious to hear what's needed to set up all of this!

[dscho]

I already added an aarch64/ container at https://wingit.blob.core.windows.net/aarch64, but since it is not a regular web server you do not see anything there because it requires an index.html file there (which we do not have in https://wingit.blob.core.windows.net/x86-64/ either).

But before we can populate that, we need to teach our branch-deploy-based automation to build those packages.

To do that, we need to set up self-hosted runners (most likely using a separate workflow using your excellent Azure Resource Management template combined with the azure/arm-deploy GitHub Action, setting up ephemeral runners) and we will want to deploy/tear down these Virtual Machines in response to webhook events most likely via the GitForWindowsHelper GitHub App.

To address the slowness of setup-git-sdk-64-sdk, I think we will want to fork git-sdk-64, naming the fork git-sdk-arm64 and adding all the mingw-w64-clang-aarch64-* packages we need to build mingw-w64-clang-aarch64-git, and removing probably all of the mingw-w64-x86_64-* packages.

The sync workflow of git-sdk-64 should be easily adapted to the needs of git-sdk-arm64, as it can run using a regular windows-latest agent: as far as I can tell, none of the required clang-aarch64 package need to run post-install steps that actually require an ARM64 processor.

The biggest obstacle for the sync workflow is actually that git-extra is incorrectly named: it should be called mingw-w64-<arch>-git-extra instead.

Then the Pacman repo needs to be added to /etc/pacman.conf (like here).

Indeed, it should probably look somewhat like this:

[git-for-windows-clang64]
Server = https://wingit.blob.core.windows.net/aarch64

[dscho]

FWIW I created a Windows/ARM64 VM on Azure and it took 3m20 to provision (but that was without installing Git for Windows or the GitHub Actions runner). It's a bit on the long-ish side, but definitely could be done on-demand in git-for-windows-automation, it's not like we're running many jobs per day there, it's more like a job or two, maybe even as many as five, per week.

[dscho]

Even if someone opens a PR targeting the main branch from their fork, they won't be able to access the self-hosted runner as their workflow file is not in the main branch of the git-for-windows/git repo in the example below.

Is that really the case? I thought that the target branch would qualify... I guess we'll need to verify this in a test.

[dennisameling]

FWIW I created a Windows/ARM64 VM on Azure and it took 3m20 to provision (but that was without installing Git for Windows or the GitHub Actions runner). It's a bit on the long-ish side, but definitely could be done on-demand in git-for-windows-automation, it's not like we're running many jobs per day there, it's more like a job or two, maybe even as many as five, per week.

In the PR I created to create VMs and immediately deallocate them, it took ~6 mins to provision the VM and install GfW + the Actions Runner, plus an additional ~1 min to deallocate it. The good news though is that it can simply be started at any given time (but within 14 days after creation, as GH Actions runner registrations expire after that), which should be much faster than creating a VM from scratch.

[dscho]

we will want to deploy/tear down these Virtual Machines in response to webhook events most likely via the GitForWindowsHelper GitHub App.

I did some more research on this. The webhook event we receive is a bit unhelpful as-is:

Headers

Request method: POST
Accept: */*
content-type: application/json
User-Agent: GitHub-Hookshot/2179efe
X-GitHub-Delivery: d9e22840-8ae3-11ed-996f-fb8fd9c03449
X-GitHub-Event: check_run
X-GitHub-Hook-ID: 392499201
X-GitHub-Hook-Installation-Target-ID: 246505
X-GitHub-Hook-Installation-Target-Type: integration

Payload

{
  "action": "completed",
  "check_run": {
    "id": 10401792578,
    "name": "build",
    "node_id": "CR_kwDOIhGD8s8AAAACa_7CQg",
    "head_sha": "e941aa8e7a860e54bc4b78b9feda23cc999f1a93",
    "external_id": "ca395085-040a-526b-2ce8-bdc85f692774",
    "url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/10401792578",
    "html_url": "https://github.com/git-for-windows/git-for-windows-automation/actions/runs/3825211124/jobs/6508006925",
    "details_url": "https://github.com/git-for-windows/git-for-windows-automation/actions/runs/3825211124/jobs/6508006925",
    "status": "completed",
    "conclusion": "failure",
    "started_at": "2023-01-02T21:22:38Z",
    "completed_at": "2023-01-02T21:24:31Z",
    "output": {
      "title": null,
      "summary": null,
      "text": null,
      "annotations_count": 1,
      "annotations_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/10401792578/annotations"
    },
    "check_suite": {
      "id": 10147442373,
      "node_id": "CS_kwDOIhGD8s8AAAACXNWuxQ",
      "head_branch": "git-extra-arm64",
      "head_sha": "e941aa8e7a860e54bc4b78b9feda23cc999f1a93",
      "status": "completed",
      "conclusion": "failure",
      "url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-suites/10147442373",
      "before": "188915318903864fe902f715da3274cb5fb1e2ad",
      "after": "e941aa8e7a860e54bc4b78b9feda23cc999f1a93",
      "pull_requests": [

      ],
      "app": {
        "id": 15368,
        "slug": "github-actions",
        "node_id": "MDM6QXBwMTUzNjg=",
        "owner": {
          "login": "github",
          "id": 9919,
          "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
          "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/github",
          "html_url": "https://github.com/github",
          "followers_url": "https://api.github.com/users/github/followers",
          "following_url": "https://api.github.com/users/github/following{/other_user}",
          "gists_url": "https://api.github.com/users/github/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/github/subscriptions",
          "organizations_url": "https://api.github.com/users/github/orgs",
          "repos_url": "https://api.github.com/users/github/repos",
          "events_url": "https://api.github.com/users/github/events{/privacy}",
          "received_events_url": "https://api.github.com/users/github/received_events",
          "type": "Organization",
          "site_admin": false
        },
        "name": "GitHub Actions",
        "description": "Automate your workflow from idea to production",
        "external_url": "https://help.github.com/en/actions",
        "html_url": "https://github.com/apps/github-actions",
        "created_at": "2018-07-30T09:30:17Z",
        "updated_at": "2019-12-10T19:04:12Z",
        "permissions": {
          "actions": "write",
          "administration": "read",
          "checks": "write",
          "contents": "write",
          "deployments": "write",
          "discussions": "write",
          "issues": "write",
          "merge_queues": "write",
          "metadata": "read",
          "packages": "write",
          "pages": "write",
          "pull_requests": "write",
          "repository_hooks": "write",
          "repository_projects": "write",
          "security_events": "write",
          "statuses": "write",
          "vulnerability_alerts": "read"
        },
        "events": [
          "branch_protection_rule",
          "check_run",
          "check_suite",
          "create",
          "delete",
          "deployment",
          "deployment_status",
          "discussion",
          "discussion_comment",
          "fork",
          "gollum",
          "issues",
          "issue_comment",
          "label",
          "merge_group",
          "milestone",
          "page_build",
          "project",
          "project_card",
          "project_column",
          "public",
          "pull_request",
          "pull_request_review",
          "pull_request_review_comment",
          "push",
          "registry_package",
          "release",
          "repository",
          "repository_dispatch",
          "status",
          "watch",
          "workflow_dispatch",
          "workflow_run"
        ]
      },
      "created_at": "2023-01-02T21:22:31Z",
      "updated_at": "2023-01-02T21:24:32Z"
    },
    "app": {
      "id": 15368,
      "slug": "github-actions",
      "node_id": "MDM6QXBwMTUzNjg=",
      "owner": {
        "login": "github",
        "id": 9919,
        "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
        "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/github",
        "html_url": "https://github.com/github",
        "followers_url": "https://api.github.com/users/github/followers",
        "following_url": "https://api.github.com/users/github/following{/other_user}",
        "gists_url": "https://api.github.com/users/github/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/github/subscriptions",
        "organizations_url": "https://api.github.com/users/github/orgs",
        "repos_url": "https://api.github.com/users/github/repos",
        "events_url": "https://api.github.com/users/github/events{/privacy}",
        "received_events_url": "https://api.github.com/users/github/received_events",
        "type": "Organization",
        "site_admin": false
      },
      "name": "GitHub Actions",
      "description": "Automate your workflow from idea to production",
      "external_url": "https://help.github.com/en/actions",
      "html_url": "https://github.com/apps/github-actions",
      "created_at": "2018-07-30T09:30:17Z",
      "updated_at": "2019-12-10T19:04:12Z",
      "permissions": {
        "actions": "write",
        "administration": "read",
        "checks": "write",
        "contents": "write",
        "deployments": "write",
        "discussions": "write",
        "issues": "write",
        "merge_queues": "write",
        "metadata": "read",
        "packages": "write",
        "pages": "write",
        "pull_requests": "write",
        "repository_hooks": "write",
        "repository_projects": "write",
        "security_events": "write",
        "statuses": "write",
        "vulnerability_alerts": "read"
      },
      "events": [
        "branch_protection_rule",
        "check_run",
        "check_suite",
        "create",
        "delete",
        "deployment",
        "deployment_status",
        "discussion",
        "discussion_comment",
        "fork",
        "gollum",
        "issues",
        "issue_comment",
        "label",
        "merge_group",
        "milestone",
        "page_build",
        "project",
        "project_card",
        "project_column",
        "public",
        "pull_request",
        "pull_request_review",
        "pull_request_review_comment",
        "push",
        "registry_package",
        "release",
        "repository",
        "repository_dispatch",
        "status",
        "watch",
        "workflow_dispatch",
        "workflow_run"
      ]
    },
    "pull_requests": [

    ]
  },
  "repository": {
    "id": 571573234,
    "node_id": "R_kgDOIhGD8g",
    "name": "git-for-windows-automation",
    "full_name": "git-for-windows/git-for-windows-automation",
    "private": false,
    "owner": {
      "login": "git-for-windows",
      "id": 4571183,
      "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
      "avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/git-for-windows",
      "html_url": "https://github.com/git-for-windows",
      "followers_url": "https://api.github.com/users/git-for-windows/followers",
      "following_url": "https://api.github.com/users/git-for-windows/following{/other_user}",
      "gists_url": "https://api.github.com/users/git-for-windows/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/git-for-windows/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/git-for-windows/subscriptions",
      "organizations_url": "https://api.github.com/users/git-for-windows/orgs",
      "repos_url": "https://api.github.com/users/git-for-windows/repos",
      "events_url": "https://api.github.com/users/git-for-windows/events{/privacy}",
      "received_events_url": "https://api.github.com/users/git-for-windows/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/git-for-windows/git-for-windows-automation",
    "description": "A few GitHub workflows and support code to help with Git for Windows' day-to-day tasks",
    "fork": false,
    "url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation",
    "forks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/forks",
    "keys_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/teams",
    "hooks_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/hooks",
    "issue_events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/events{/number}",
    "events_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/events",
    "assignees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/assignees{/user}",
    "branches_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/branches{/branch}",
    "tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/tags",
    "blobs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/languages",
    "stargazers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/stargazers",
    "contributors_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contributors",
    "subscribers_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscribers",
    "subscription_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/subscription",
    "commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/contents/{+path}",
    "compare_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/merges",
    "archive_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/downloads",
    "issues_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/issues{/number}",
    "pulls_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/labels{/name}",
    "releases_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/releases{/id}",
    "deployments_url": "https://api.github.com/repos/git-for-windows/git-for-windows-automation/deployments",
    "created_at": "2022-11-28T12:30:12Z",
    "updated_at": "2022-12-21T04:50:16Z",
    "pushed_at": "2023-01-02T21:22:29Z",
    "git_url": "git://github.com/git-for-windows/git-for-windows-automation.git",
    "ssh_url": "[email protected]:git-for-windows/git-for-windows-automation.git",
    "clone_url": "https://github.com/git-for-windows/git-for-windows-automation.git",
    "svn_url": "https://github.com/git-for-windows/git-for-windows-automation",
    "homepage": null,
    "size": 38,
    "stargazers_count": 1,
    "watchers_count": 1,
    "language": "JavaScript",
    "has_issues": true,
    "has_projects": false,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "has_discussions": true,
    "forks_count": 5,
    "mirror_url": null,
    "archived": false,
    "disabled": false,
    "open_issues_count": 4,
    "license": null,
    "allow_forking": true,
    "is_template": false,
    "web_commit_signoff_required": false,
    "topics": [

    ],
    "visibility": "public",
    "forks": 5,
    "open_issues": 4,
    "watchers": 1,
    "default_branch": "main"
  },
  "organization": {
    "login": "git-for-windows",
    "id": 4571183,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ1NzExODM=",
    "url": "https://api.github.com/orgs/git-for-windows",
    "repos_url": "https://api.github.com/orgs/git-for-windows/repos",
    "events_url": "https://api.github.com/orgs/git-for-windows/events",
    "hooks_url": "https://api.github.com/orgs/git-for-windows/hooks",
    "issues_url": "https://api.github.com/orgs/git-for-windows/issues",
    "members_url": "https://api.github.com/orgs/git-for-windows/members{/member}",
    "public_members_url": "https://api.github.com/orgs/git-for-windows/public_members{/member}",
    "avatar_url": "https://avatars.githubusercontent.com/u/4571183?v=4",
    "description": ""
  },
  "sender": {
    "login": "dscho",
    "id": 127790,
    "node_id": "MDQ6VXNlcjEyNzc5MA==",
    "avatar_url": "https://avatars.githubusercontent.com/u/127790?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/dscho",
    "html_url": "https://github.com/dscho",
    "followers_url": "https://api.github.com/users/dscho/followers",
    "following_url": "https://api.github.com/users/dscho/following{/other_user}",
    "gists_url": "https://api.github.com/users/dscho/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/dscho/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/dscho/subscriptions",
    "organizations_url": "https://api.github.com/users/dscho/orgs",
    "repos_url": "https://api.github.com/users/dscho/repos",
    "events_url": "https://api.github.com/users/dscho/events{/privacy}",
    "received_events_url": "https://api.github.com/users/dscho/received_events",
    "type": "User",
    "site_admin": true
  },
  "installation": {
    "id": 31544731,
    "node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMzE1NDQ3MzE="
  }
}

If you follow the check_run.url https://api.github.com/repos/git-for-windows/git-for-windows-automation/check-runs/10401792578, you do not see much of anything useful. But if you replace the check-runs part with actions/jobs, it gets a lot more useful: https://api.github.com/repos/git-for-windows/git-for-windows-automation/actions/jobs/10401792578 contains this:

labels:
  0: "Windows"
  1: "ARM64"
runner_id: 25
runner_name: "actions-runner-20221231123310813147989"
runner_group_id: 1
runner_group_name: "Default"

Which should be enough information to pass onto a separate workflow run that should be able to figure out what Azure resources to tear down.

[dscho]

See git-for-windows/gfw-helper-github-app#20 for the final bit to automate this.

[dennisameling]

But before we can populate that, we need to teach our branch-deploy-based automation to build those packages.

Here's a PR that teaches the pipeline to build aarch64 packages: git-for-windows/git-for-windows-automation#4

/usr/src/build-extra/pacman-helper.sh quick_add will also need to be updated so support aarch64. May I ask you to take on that part as I don't have access to the Pacman repo nor Azure Storage? 😊

[dscho]

Will do.