Skip to content

Commit 83063d7

Browse files
ci: [NGOv2.X] Automation of NGO releases (#3602)
## Purpose of this PR This PR automates the releases of NGO package (which can be extended for other packages since it's quite modular). The job will operate on the branch from which it will be triggered and by default it will be on the `develop` branch (job trigger). For more background note that Netcode team sprint last 4 weeks, starting and ending on Wednesday and our goal is to kickstart the release process after last full week of the sprint and preferably release until next Wednesday (end of the sprint). So the goal would be to run everything on Saturday (after the week concluded) and have everything ready from Monday morning to Playtest/investigate failures. As a separate part of this automation, as per **[THIS PR](#3557 we are able to trigger builds of our samples so those will also be triggered on the weekend to be ready for playtesting on Monday. Now with this in mind the process will work in the following way: 1. It runs on weekly schedule but we are interesting on executing it every 4th Saturday (after last full week of the Netcode team sprint. Our sprint last 4 weeks). Because of that the first action that this job will do is to run a check to see if the following requirements are satisfied: - Is this the 4th Saturday since we last released? - Is NGO CHANGELOG not empty? (no point of making the release is there is nothing to release) - Is the release branch not yet created? Before looking at the next steps, please note that as per **[THIS PR](#3549 the package version of Nshould always correspond to the current state of the package. 2. If the above is satisfied the next step will be to create the release branch (based on the known package version). The already existing **release.py** script will be executed there in order to regenerate recipes, cleanup changelog and generally prepare our package for releasing (see result **[HERE](c9f3f204fc4058df8781dd577b187a117216b544)**) 3. The next step will be to trigger `all_promotion_related_jobs_promotiontrigger` job. In that way when we will set-up packageworks release stream on Monday morning we will already have the results from this job and we can address any potential failures. 4. As a last step the job will create a commit to the local branch (normally `develop` but for the sake of playtesting I used a test branch). The goal with changelog here is we want to avoid changelog divergence if release will take longer then expected. The script will clean the empty sections, assign correct release version and date and then add back the [Unreleased] section template for the purpose of next entries. The goal with package version is that we want to indicate that now the package reflects further state (next patch). After this the job will commit this to the branch. I created Netcode-Team-Bot with appropriate tokens and marked committer to point to the bot (when for example making a direct commit to `develop` with changelog and package version update) <img width="1782" height="292" alt="image" src="https://github.com/user-attachments/assets/81e76d5a-b8f3-48f2-985f-40126e6511b8" /> The bot will make commits like <img width="1804" height="164" alt="image" src="https://github.com/user-attachments/assets/728eb89c-2c59-4b81-8a7e-7f858470b979" /> ## Jira ticket https://jira.unity3d.com/browse/MTT-12841 ## Documentation After this gets merged I will add approperiate description of Netcode release process in https://docs.google.com/document/d/16g9B5jxXeV0zG44Fhax-6Md_gtLoqGVthKQ7LkDvdqM/edit?tab=t.0#heading=h.t1guku6agyev ## Testing & QA Since this automation runs on the same branch on which it's triggered (and just because of job trigger it will be `develop-2.0.0` branch) I tested it locally (each script) and entire automation was tested on `testing-automated-netcode-releases` branch, the effects of which you can see as follows (note that instead of `develop-2.0.0` I just used local branch): 1. I tested for the normal release schedule date (result **[HERE](https://unity-ci.cds.internal.unity3d.com/job/56127622)**) 2. I tested with empty changelog (result **[HERE](https://unity-ci.cds.internal.unity3d.com/job/56123485)**) 3. The job itself was green (**[LINK](https://unity-ci.cds.internal.unity3d.com/job/56127745)**). 4. `release/2.5.1` branch was created with proper initial setup (**[LINK](https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/tree/release/2.5.1)**), see latest commits there 5. Changelog and package versions were updated on `develop-2.0.0` branch (replaced for testing with `testing-automated-netcode-releases`). Notice that this simulates what will happen on develop branch after this script runs. (**[LINK](https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/tree/testing-automated-netcode-releases-2.x)**) 6. Wrench jobs were correctly triggered on the created release branch (**[LINK](https://unity-ci.cds.internal.unity3d.com/project/1201/branch/release%2F2.5.1/jobDefinition/.yamato%2Fwrench%2Fpromotion-jobs.yml%23publish_dry_run_netcode_gameobjects/recent-jobs?nav=none)**) 7. Builds were triggered from release branch (**[example build LINK](https://unity-ci.cds.internal.unity3d.com/project/1201/branch/release%2F2.5.1/jobDefinition/.yamato%2Fproject-builders%2Fproject-builders.yml%23build_BossRoom_project/recent-jobs?nav=jobDefinitions)**) ## Things to note - Updating package version to next patch won't be immediately valid (vetting test will still compare it to latest released package version) but it serves the purpose of automating those steps and not needing to remember about it after package is released - **[email protected]** was created to assign NETCODE_YAMATO_API_KEY to it. Additionally I created tokens for both GitHubs and when making commit I'm just pointing to this bot as committer and author. ## Backports It's backported to #3603
1 parent 12f3c79 commit 83063d7

10 files changed

+812
-78
lines changed

.yamato/ngo-publish.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
ngo_release_preparation:
2+
name: "NGO release preparation"
3+
agent: { type: Unity::VM, flavor: b1.small, image: package-ci/ubuntu-22.04:v4 }
4+
triggers:
5+
recurring:
6+
- branch: develop-2.0.0 # We make new releases from this branch
7+
frequency: weekly # Run at some point every Saturday. Note that it's restricted to every 4th Saturday inside the script
8+
rerun: always
9+
commands:
10+
- pip install PyGithub
11+
- pip install GitPython
12+
- python Tools/scripts/ReleaseAutomation/run_release_preparation.py
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
"""Netcode configuration for the release process automation."""
2+
3+
import datetime
4+
import sys
5+
import os
6+
from github import Github
7+
from github import GithubException
8+
9+
PARENT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))
10+
sys.path.insert(0, PARENT_DIR)
11+
12+
from Utils.general_utils import get_package_version_from_manifest
13+
from release import make_package_release_ready
14+
15+
class GithubUtils:
16+
def __init__(self, access_token, repo):
17+
self.github = Github(base_url="https://api.github.com",
18+
login_or_token=access_token)
19+
self.repo = self.github.get_repo(repo)
20+
21+
def is_branch_present(self, branch_name):
22+
try:
23+
self.repo.get_branch(branch_name)
24+
return True # Branch exists
25+
26+
except GithubException as ghe:
27+
if ghe.status == 404:
28+
return False # Branch does not exist
29+
raise Exception(f"An error occurred with the GitHub API: {ghe.status}", data=ghe.data)
30+
31+
class ReleaseConfig:
32+
"""A simple class to hold all shared configuration."""
33+
def __init__(self):
34+
self.manifest_path = 'com.unity.netcode.gameobjects/package.json'
35+
self.changelog_path = 'com.unity.netcode.gameobjects/CHANGELOG.md'
36+
self.validation_exceptions_path = './ValidationExceptions.json'
37+
self.github_repo = 'Unity-Technologies/com.unity.netcode.gameobjects'
38+
self.default_repo_branch = 'develop-2.0.0' # Changelog and package version change will be pushed to this branch
39+
self.yamato_project_id = '1201'
40+
self.command_to_run_on_release_branch = make_package_release_ready
41+
42+
self.release_weekday = 5 # Saturday
43+
self.release_week_cycle = 4 # Release every 4 weeks
44+
self.anchor_date = datetime.date(2025, 7, 19) # Anchor date for the release cycle (previous release Saturday)
45+
46+
self.package_version = get_package_version_from_manifest(self.manifest_path)
47+
self.release_branch_name = f"release/{self.package_version}" # Branch from which we want to release
48+
self.commit_message = f"Updated changelog and package version for Netcode in anticipation of v{self.package_version} release"
49+
50+
GITHUB_TOKEN_NAME = "NETCODE_GITHUB_TOKEN"
51+
YAMATO_API_KEY_NAME = "NETCODE_YAMATO_API_KEY"
52+
self.github_token = os.environ.get(GITHUB_TOKEN_NAME)
53+
self.yamato_api_token = os.environ.get(YAMATO_API_KEY_NAME)
54+
self.commiter_name = "netcode-automation"
55+
self.commiter_email = "[email protected]"
56+
57+
self.yamato_samples_to_build = [
58+
{
59+
"name": "BossRoom",
60+
"jobDefinition": f".yamato%2Fproject-builders%2Fproject-builders.yml%23build_BossRoom_project",
61+
},
62+
{
63+
"name": "Asteroids",
64+
"jobDefinition": f".yamato%2Fproject-builders%2Fproject-builders.yml%23build_Asteroids_project",
65+
},
66+
{
67+
"name": "SocialHub",
68+
"jobDefinition": f".yamato%2Fproject-builders%2Fproject-builders.yml%23build_SocialHub_project",
69+
}
70+
]
71+
72+
self.yamato_build_automation_configs = [
73+
{
74+
"job_name": "Build Sample for Windows with minimal supported editor (2022.3), burst ON, IL2CPP",
75+
"variables": [
76+
{ "key": "BURST_ON_OFF", "value": "on" },
77+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "win64" },
78+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
79+
{ "key": "UNITY_VERSION", "value": "2022.3" } # Minimal supported editor
80+
]
81+
},
82+
{
83+
"job_name": "Build Sample for Windows with latest functional editor (6000.2), burst ON, IL2CPP",
84+
"variables": [
85+
{ "key": "BURST_ON_OFF", "value": "on" },
86+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "win64" },
87+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
88+
{ "key": "UNITY_VERSION", "value": "6000.2" } # Editor that most our users will use (not alpha). Sometimes when testing on trunk we have weird editor issues not caused by us so the preference will be to test on latest editor that our users will use.
89+
]
90+
},
91+
{
92+
"job_name": "Build Sample for Windows with latest editor (trunk), burst ON, IL2CPP",
93+
"variables": [
94+
{ "key": "BURST_ON_OFF", "value": "on" },
95+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "win64" },
96+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
97+
{ "key": "UNITY_VERSION", "value": "trunk" } # latest editor
98+
]
99+
},
100+
{
101+
"job_name": "Build Sample for MacOS with minimal supported editor (2022.3), burst OFF, Mono",
102+
"variables": [
103+
{ "key": "BURST_ON_OFF", "value": "off" },
104+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "mac" },
105+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "mono" },
106+
{ "key": "UNITY_VERSION", "value": "2022.3" } # Minimal supported editor
107+
]
108+
},
109+
{
110+
"job_name": "Build Sample for MacOS with latest functional editor (6000.2), burst OFF, Mono",
111+
"variables": [
112+
{ "key": "BURST_ON_OFF", "value": "off" },
113+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "mac" },
114+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "mono" },
115+
{ "key": "UNITY_VERSION", "value": "6000.2" } # Editor that most our users will use (not alpha). Sometimes when testing on trunk we have weird editor issues not caused by us so the preference will be to test on latest editor that our users will use.
116+
]
117+
},
118+
{
119+
"job_name": "Build Sample for MacOS with latest editor (trunk), burst OFF, Mono",
120+
"variables": [
121+
{ "key": "BURST_ON_OFF", "value": "off" },
122+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "mac" },
123+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "mono" },
124+
{ "key": "UNITY_VERSION", "value": "trunk" } # latest editor
125+
]
126+
},
127+
{
128+
"job_name": "Build Sample for Android with minimal supported editor (2022.3), burst ON, IL2CPP",
129+
"variables": [
130+
{ "key": "BURST_ON_OFF", "value": "on" },
131+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "android" },
132+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
133+
{ "key": "UNITY_VERSION", "value": "2022.3" } # Minimal supported editor
134+
]
135+
},
136+
{
137+
"job_name": "Build Sample for Android with latest functional editor (6000.2), burst ON, IL2CPP",
138+
"variables": [
139+
{ "key": "BURST_ON_OFF", "value": "on" },
140+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "android" },
141+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
142+
{ "key": "UNITY_VERSION", "value": "6000.2" } # Editor that most our users will use (not alpha). Sometimes when testing on trunk we have weird editor issues not caused by us so the preference will be to test on latest editor that our users will use.
143+
]
144+
},
145+
{
146+
"job_name": "Build Sample for Android with latest editor (trunk), burst ON, IL2CPP",
147+
"variables": [
148+
{ "key": "BURST_ON_OFF", "value": "on" },
149+
{ "key": "PLATFORM_WIN64_MAC_ANDROID", "value": "android" },
150+
{ "key": "SCRIPTING_BACKEND_IL2CPP_MONO", "value": "il2cpp" },
151+
{ "key": "UNITY_VERSION", "value": "trunk" } # latest editor
152+
]
153+
}
154+
]
155+
156+
error_messages = []
157+
if not os.path.exists(self.manifest_path):
158+
error_messages.append(f" Path does not exist: {self.manifest_path}")
159+
160+
if not os.path.exists(self.changelog_path):
161+
error_messages.append(f" Path does not exist: {self.changelog_path}")
162+
163+
if not os.path.exists(self.validation_exceptions_path):
164+
error_messages.append(f" Path does not exist: {self.validation_exceptions_path}")
165+
166+
if not callable(self.command_to_run_on_release_branch):
167+
error_messages.append("command_to_run_on_release_branch is not a function! Actual value:", self.command_to_run_on_release_branch)
168+
169+
if self.package_version is None:
170+
error_messages.append(f"Package version not found at {self.manifest_path}")
171+
172+
if not self.github_token:
173+
error_messages.append(f"Error: {GITHUB_TOKEN_NAME} environment variable not set.")
174+
175+
if not self.yamato_api_token:
176+
error_messages.append(f"Error: {YAMATO_API_KEY_NAME} environment variable not set.")
177+
178+
# Initialize PyGithub and get the repository object
179+
self.github_manager = GithubUtils(self.github_token, self.github_repo)
180+
181+
if not self.github_manager.is_branch_present(self.default_repo_branch):
182+
error_messages.append(f"Branch '{self.default_repo_branch}' does not exist.")
183+
184+
if self.github_manager.is_branch_present(self.release_branch_name):
185+
error_messages.append(f"Branch '{self.release_branch_name}' is already present in the repo.")
186+
187+
if error_messages:
188+
summary = "Failed to initialize NetcodeReleaseConfig due to invalid setup:\n" + "\n".join(f"- {msg}" for msg in error_messages)
189+
raise ValueError(summary)
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
"""Automation for package release process."""
2+
3+
import sys
4+
import os
5+
6+
PARENT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))
7+
sys.path.insert(0, PARENT_DIR)
8+
9+
from ReleaseAutomation.release_config import ReleaseConfig
10+
from Utils.git_utils import create_branch_execute_commands_and_push
11+
from Utils.verifyReleaseConditions import verifyReleaseConditions
12+
from Utils.commitChangelogAndPackageVersionUpdates import commitChangelogAndPackageVersionUpdates
13+
from Utils.triggerYamatoJobsForReleasePreparation import trigger_release_preparation_jobs
14+
15+
def PrepareNetcodePackageForRelease():
16+
try:
17+
config = ReleaseConfig()
18+
19+
print("\nStep 1: Verifying release conditions...")
20+
verifyReleaseConditions(config)
21+
22+
print("\nStep 2: Creating release branch...")
23+
create_branch_execute_commands_and_push(config)
24+
25+
print("\nStep 3: Triggering Yamato validation jobs...")
26+
trigger_release_preparation_jobs(config)
27+
28+
print("\nStep 4: Committing changelog and version updates...")
29+
commitChangelogAndPackageVersionUpdates(config)
30+
31+
except Exception as e:
32+
print("\n--- ERROR: Netcode release process failed ---", file=sys.stderr)
33+
print(f"Reason: {e}", file=sys.stderr)
34+
sys.exit(1)
35+
36+
if __name__ == "__main__":
37+
PrepareNetcodePackageForRelease()
Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
"""
2+
Creates a direct commit to specified branch (in the config) to update the changelog, package version and validation exceptions for a new release using the GitHub API.
3+
Quite often the changelog gets distorted between the time we branch for the release and the time we will branch back.
4+
To mitigate this we want to create changelog update PR straight away and merge it fast while proceeding with the release.
5+
6+
This will also allow us to skip any PRs after releasing, unless, we made some changes on this branch.
7+
8+
"""
9+
#!/usr/bin/env python3
10+
import os
11+
import sys
12+
from github import GithubException
13+
from git import Actor
14+
15+
PARENT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))
16+
sys.path.insert(0, PARENT_DIR)
17+
18+
from ReleaseAutomation.release_config import ReleaseConfig
19+
from Utils.general_utils import get_package_version_from_manifest, update_changelog, update_package_version_by_patch, update_validation_exceptions
20+
from Utils.git_utils import get_local_repo
21+
22+
def commitChangelogAndPackageVersionUpdates(config: ReleaseConfig):
23+
"""
24+
The function updates the changelog and package version of the package in anticipation of a new release.
25+
This means that it will
26+
1) Clean and update the changelog for the current package version.
27+
2) Add new Unreleased section template at the top.
28+
3) Update the package version in the package.json file by incrementing the patch version to signify the current state of the package.
29+
4) Update package version in the validation exceptions to match the new package version.
30+
31+
This assumes that at the same time you already branched off for the release. Otherwise it may be confusing
32+
"""
33+
34+
try:
35+
if not config.github_manager.is_branch_present(config.default_repo_branch):
36+
print(f"Branch '{config.default_repo_branch}' does not exist. Exiting.")
37+
sys.exit(1)
38+
39+
repo = get_local_repo()
40+
repo.git.fetch('--prune', '--prune-tags')
41+
repo.git.checkout(config.default_repo_branch)
42+
repo.git.pull("origin", config.default_repo_branch)
43+
44+
# Update the changelog file with adding new [Unreleased] section
45+
update_changelog(config.changelog_path, config.package_version, add_unreleased_template=True)
46+
# Update the package version by patch to represent the "current package state" after release
47+
updated_package_version = update_package_version_by_patch(config.manifest_path)
48+
update_validation_exceptions(config.validation_exceptions_path, updated_package_version)
49+
50+
repo.git.add(config.changelog_path)
51+
repo.git.add(config.manifest_path)
52+
repo.git.add(config.validation_exceptions_path)
53+
54+
author = Actor(config.commiter_name, config.commiter_email)
55+
committer = Actor(config.commiter_name, config.commiter_email)
56+
57+
repo.index.commit(config.commit_message, author=author, committer=committer, skip_hooks=True)
58+
repo.git.push("origin", config.default_repo_branch)
59+
60+
print(f"Successfully updated and pushed the changelog on branch: {config.default_repo_branch}")
61+
62+
except GithubException as e:
63+
print(f"An error occurred with the GitHub API: {e.status}", file=sys.stderr)
64+
print(f"Error details: {e.data}", file=sys.stderr)
65+
sys.exit(1)
66+
except Exception as e:
67+
print(f"An unexpected error occurred: {e}", file=sys.stderr)
68+
sys.exit(1)

0 commit comments

Comments
 (0)