[Tooling] Improve Prototype (aka Installable) Build comment and metadata

[AliSoftware]

What?

Improve Prototype (aka Installable) Builds in a couple of ways:

• Make the PR comment announcing the Prototype build QR code and information nicer
  • In particular, the QR code and URLs now directly links to the appropriate build on App Center, instead of defaulting to show the latest build in App Center as before
• Change the wording used throughout Fastfile/Buildkite from "Installable Build" to "Prototype Build".
  • Every build is "installable" in some way, so that wording didn't make much sense. We're already using the term "Prototype Builds" for those in other apps, which feels like a better fit already, so we've decided to slowly adopt this new wording for those from now on to us make the term less ambiguous and more descriptive.
• Attach Buildkite metadata to the CI build with relevant infos
• Add a Buildkite annotation to the CI build with relevant infos
• Add more info to the App Center "Release Notes" field of each Prototype Builds being created

To test:

• Check that the PR got two PR comments, providing the information for the WordPress and Jetpack Prototype builds respectively, unfold the triangle/spoiler blocks, and verify that the information displayed for those builds is accurate
• Scan the QR code (and/or follow the URL) and validate that it directly redirects you to the page in App Center that corresponds to the right build, even if new builds have been uploaded to App Center since and this is not the latest one anymore
• Verify that the "Release Notes" field in App Center for that build contains the appropriate information about the Commit and PR it corresponds to
• Open the Buildkite build for this PR, and check that the annotation shows some metadata about the App Center build (short/long version, app center id, …)
• Add /meta-data to the URL of the Buildkite build and validate that it includes the correct metadata
• Use the ?meta_data query parameter on the Buildkite URL, e.g. ?meta_data[build_type]=Prototype to search for all CI builds that correspond to / include Prototype builds

[wpmobilebot]

📲 You can test the changes from this Pull Request in WordPress Alpha by scanning the QR code below to install the corresponding build.

	App Name	WordPress Alpha
	Configuration	Release-Alpha
	Build Number	pr20071-59a3f8e
	Version	21.9
	Bundle ID	org.wordpress.alpha
	Commit	59a3f8e
	App Center Build	WPiOS - One-Offs #5146

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

[wpmobilebot]

📲 You can test the changes from this Pull Request in Jetpack Alpha by scanning the QR code below to install the corresponding build.

	App Name	Jetpack Alpha
	Configuration	Release-Alpha
	Build Number	pr20071-59a3f8e
	Version	21.9
	Bundle ID	com.jetpack.alpha
	Commit	59a3f8e
	App Center Build	jetpack-installable-builds #4168

Automatticians: You can use our internal self-serve MC tool to give yourself access to App Center if needed.

[mokagio]

I was about to comment in regards to the output_app_name CI failure, but I saw your force push should address it 👍

[AliSoftware]

👌

Note: I cancelled the "Build for Testing" step on CI to avoid spawning Mac workers for it needlessly (as this PR is still Draft and will have more force-pushes to come—especially to update the reference in the Gemfile once the new toolkit is released—and our Mac queue is already under enough stress as it is recently)

[crazytonyli]

I love how git got confused about the renaming and ended up producing this diff.

[crazytonyli]

Occasionally I noticed there are "Test Failures" annotation in the build page, but all test jobs are successful. I think the root cause is the "Test Failures" annotation isn't removed after a successful retry.

I think a similar issue may occur here too: this "build info" annotation won't be removed if the latest build fails.

[AliSoftware]

That fastlane action is called at the very end of the script and buildkite step doing the Prototype build. So an annotation will only be added if the Prototype build step ran all the way to the end once. But if it went all the way to the end without an issue, the CI step will be green and Buildkite won't even show a button to retry the step, so it's impossible for that step to succeed first (and add metadata) then fail later.

(and if the user chooses "Rebuild" on the whole CI build, that creates an entirely separate build and Buildkite page for it anyway)

The only case where I can see the situation that you describes happening is if the script and lane called by that CI step goes all the way to the end, after successfully pushing the build to App Center and adding the annotation, but fail at some meta CI level (eg failure to terminate and unregister the VM properly or whatnot)… and then you retry the step, and then this time if fails before getting to upload a new App Center build and adding the annotation… quite the corner case imho 😅

---

In contrast, the reason why annotations for tests stay around even after build success is a bit different:

• The test failure annotations are generated on step failure, in case where we can (and are likely to) retry the step. While App Center metadata info annotations are added on success
• The annotation for test failure annotation is handled by a shell script from our a8c-ci-toolkit-buildkite-plugin (annotate_test_failures action iirc?) not by the fastlane action from our release-toolkit
• One of the reasons for why we sometimes still have those annotations reporting test failures even when the step ended up green in the end… might not be because we forgot to remove the annotation, but because the test was flaky but Xcode was set up to do 3 retries before marking it as failed, but still logged every attempt in the report.junit (with one XML node for the initial failure and another for the ultimate success for the same test) but our annotate_test_failures a8c-ci-toolkit command is not smart enough to pick up on that (we'll have to debug and implement a fix for all that at some point)