Skip to content

Commit 98b2b9f

Browse files
ci: Automated project builders (#3557)
## What This PR is an initial introduction of Yamato jobs which build our samples and the result can be found in the artifacts of the job under **players**. Currently there are 3 samples that we are able to build (I'm open to add more) The idea here is to run the job from the branch from which we want to test packages (let's say `release/1.4.1`) and the samples will be build with their manifest pointing to local packages from that branch. The steps are included here are: - cloning the repo - modifying project manifest. **If the Tools dependency is present I'm ensuring it's the latest one** - Validating Yamato variable inputs (described at the end of this section) - Connecting the project to use Services (needed in NGO case) - Disabling or Enabling Burst compilation - Copying Builder and assembly definition scripts to the project so we can execute them Hopefully the scripts are annotated with enough comments that entire setup is clear. Note that when executing the job you have an option to modify Yamato variables to set - Editor version which you want to use for the build - Decide if you want burst compilation to be enabled or disabled - Choose the platform for which you want to build (for now available options are macOS, Win64 and Android) - Which scripting backend you want to use (il2cpp or mono) When the job finishes running you can inspect the job artifacts to get the build (players path) or detailed build logs in case of failure or the build not appearing (logs path) ## WHY The reason for this job is to facilitate/automate Playtesting and improving developers ability to easily test their changes. This PR will help with: 1. When building bigger samples it can somehow take a lot of time (depending on computer resources) and for Playtesting we usually want to have several combinations (windows, mac, android etc) which makes our machine busy for sometimes several hours during which everything is slow and our work is affected. After this PR it will be possible to **use our CI resources to build the project which will allow us to spend our time on some other work during that time** 4. This will create **more standardized way** of making builds which will hopefully ensure that **no steps are missed** when creating the build (for example connecting to services in case of NGO) 5. This may help our developers to validate their changes by facilitating build making process The main desire behind those changes is the scenario when you trigger appropriate builds when finishing the day and by next morning you have them ready so you can start playtesting right away ## Need to know/Additional considerations - It is possible that executing such job will take **MORE TIME** than building the sample on your machine since you need to add the distribution time - Keep in mind that as for now the job uses windows machine to make the builds and I'm planning to add mac version/more platforms in another pass but that was my initial setup to not overcomplicate it too much. Note that because of this when opening macOS build you need to follow some additional steps to open the project (namely **xattr -c <path/to/application.app>** and **chmod -R +x <path/to/application.app>**). Also you can't build mac with il2cpp (only mono) - Some parameters I included as secrets so are not publicly visible ## Backport #3558
1 parent ef2fbe8 commit 98b2b9f

File tree

9 files changed

+663
-0
lines changed

9 files changed

+663
-0
lines changed
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# https://internaldocs.unity.com/yamato_continuous_integration/usage/templating/
2+
3+
NetcodeProjects:
4+
# Note that we are using internal Unity repo. This means that we may test with newest changes that are not yet released to our users (there are also public versions)
5+
# Note that for BossRoom 'main' branch supports NGOv1.X and 'develop' branch supports NGOv2.X
6+
BossRoom:
7+
GithubRepo: "https://github.com/Unity-Technologies/com.unity.multiplayer.samples.coop.git"
8+
branch: develop
9+
manifestPath: Packages/manifest.json
10+
projectPath: '.'
11+
Asteroids:
12+
GithubRepo: "https://github.cds.internal.unity3d.com/unity/Asteroids-CMB-NGO-Sample.git"
13+
branch: main
14+
manifestPath: Packages/manifest.json
15+
projectPath: '.'
16+
SocialHub:
17+
GithubRepo: "https://github.com/Unity-Technologies/com.unity.multiplayer.samples.bitesize.git"
18+
branch: main
19+
manifestPath: Basic/DistributedAuthoritySocialHub/Packages/manifest.json
20+
projectPath: 'Basic/DistributedAuthoritySocialHub'
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
{% metadata_file .yamato/project-builders/builder.metafile %}
2+
---
3+
# The above line corresponds to https://internaldocs.unity.com/yamato_continuous_integration/usage/templating/
4+
5+
# Yamato paths disclaimer:
6+
# Note that artifacts can get only local paths and since cloned project is in different location, you need to be careful with paths
7+
# All artifact paths are defined relative to the Yamato source directory (YAMATO_SOURCE_DIR) and are case-sensitive.
8+
# !!! important “Artifact directory locations” All artifact globs are defined relative to the source directory (YAMATO_SOURCE_DIR). Yamato can’t capture artifacts from outside this directory, so if you need files from elsewhere, you should copy them into your source directory as part of job commands.
9+
10+
# Those jobs were created in order to (in most cases) speed up playtesting and development time.
11+
# The aim is to collect all possible projects that use Netcode for Entities and create a job that will build the project with the given version of N4E package.
12+
# The package is taken directly from the branch from which the job was triggered. So image triggering the job from release/2.0.0 and release/2.1.0 branch to compare differences
13+
# Example use case would be to trigger the build job on Sunday so by Monday morning all builds are ready for playtesting. This limits the time a dev/QA has to spend building projects for different configurations (win, mac, Android, scripting backends, burst etc) or simply time necessary for building huge projects (Megacity).
14+
15+
# This job takes parameters as scriptable backend configuration, burst compilation, unity editor version and platform.
16+
# Since Yamato variables can't be restricted to only specific values, the job will validate the parameters passed to it and fail quickly if those are incorrect. In order to see what options are available, please look at the variable name.
17+
18+
# Note that for now all of those builds are being made on Windows machine (so for example combination of macOS + il2cpp is expected to fail)
19+
# TODO: for now all builds are being made on Windows machine, but it would be nice to have a Mac build as well.
20+
# TODO: add iOS support
21+
{% for netcodeProject in NetcodeProjects -%}
22+
build_{{ netcodeProject[0] }}_project:
23+
name: {{ netcodeProject[0] }}
24+
agent:
25+
type: Unity::VM
26+
image: package-ci/win10:v4
27+
flavor: b1.xlarge
28+
variables:
29+
UNITY_VERSION: trunk
30+
SCRIPTING_BACKEND_IL2CPP_MONO: il2cpp
31+
BURST_ON_OFF: on
32+
PLATFORM_WIN64_MAC_ANDROID: win64
33+
commands:
34+
# Validate inputs passed via Yamato variables
35+
- python Tools/CI/scripts/BuildAutomation/validate_params.py
36+
- echo Building {{ netcodeProject[0] }} project with Unity version of %UNITY_VERSION%, Scripting backend %SCRIPTING_BACKEND_IL2CPP_MONO%, Burst %BURST_ON_OFF% for platform %PLATFORM_WIN64_MAC_ANDROID%
37+
38+
# Clone the external project repository into a specific directory. Notice that branch is also specified.
39+
- git clone --single-branch --branch {{ netcodeProject[1].branch }} {{ netcodeProject[1].GithubRepo }} C:/ClonedProject
40+
41+
# Modify the external project's manifest to use the local N4E package from current branch on which this Yamato job is running. (requires python that should be preinstalled in the image)
42+
- python Tools/CI/scripts/BuildAutomation/manifest_update.py --manifest-path C:/ClonedProject/{{ netcodeProject[1].manifestPath }} --local-package-path %YAMATO_SOURCE_DIR%/com.unity.netcode.gameobjects
43+
44+
# Run python script to update ProjectSettings.asset in order to connect the project to Unity Services/set proper values.
45+
# Notice that if a project has this already set up then in theory we don't need to run this script.
46+
- python Tools/CI/scripts/BuildAutomation/connect_services.py --project-settings-path C:/ClonedProject/{{ netcodeProject[1].projectPath }}/ProjectSettings/ProjectSettings.asset
47+
48+
# Enable or disable Burst compilation. This step is specific to Netcode package (or any package that uses Burst)
49+
- IF "%BURST_ON_OFF%"=="on" (python Tools/CI/scripts/BuildAutomation/disable-enable-burst.py --enable-burst --project-path C:/ClonedProject/{{ netcodeProject[1].projectPath }})
50+
ELSE (python Tools/CI/scripts/BuildAutomation/disable-enable-burst.py --disable-burst --project-path C:/ClonedProject/{{ netcodeProject[1].projectPath }})
51+
52+
# Download the Unity Editor version specified in the UNITY_VERSION variable. Il2cpp component is downloaded only if the SCRIPTING_BACKEND_IL2CPP_MONO is set to "il2cpp".
53+
# TODO: we could download components only if needed
54+
- unity-downloader-cli --fast --wait -u %UNITY_VERSION% -p C:/TestingEditor -c Editor -c il2cpp -c Android -c macOS
55+
56+
# Add BuilderScript.cs to the project so we can modify and build the project using Unity Editor.
57+
# This is a bit tricky step, notice that we also need to include proper assembly definition in order for those scripts to compile properly.
58+
# TODO: the asmdef file can be simplified
59+
- python Tools/CI/scripts/BuildAutomation/FileCopy.py "C:/ClonedProject/{{ netcodeProject[1].projectPath }}"
60+
61+
# Build the project using Unity Editor. This will call the given static BuilderScripts method.
62+
# Ideally, it would be nice to parametrize the BuilderScripts (for example to pass scripting backend as parameter) but -executeMethod only calls static methods without parameters so for now we will have multiple configurations
63+
# Notice that for Android platform even if mono is selected, il2cpp will be used since mono is not supported for Android builds.
64+
- IF "%PLATFORM_WIN64_MAC_ANDROID%"=="win64" (
65+
IF "%SCRIPTING_BACKEND_IL2CPP_MONO%"=="il2cpp" (
66+
C:/TestingEditor/Unity.exe -projectPath C:/ClonedProject/{{ netcodeProject[1].projectPath }} -buildTarget win64 -executeMethod BuilderScripts.BuildWinIl2cpp -batchmode -logFile ./artifacts/UnityLog.txt -automated -crash-report-folder ./artifacts/CrashArtifacts -quit
67+
) ELSE (
68+
C:/TestingEditor/Unity.exe -projectPath C:/ClonedProject/{{ netcodeProject[1].projectPath }} -buildTarget win64 -executeMethod BuilderScripts.BuildWinMono -batchmode -logFile ./artifacts/UnityLog.txt -automated -crash-report-folder ./artifacts/CrashArtifacts -quit
69+
)
70+
)
71+
ELSE IF "%PLATFORM_WIN64_MAC_ANDROID%"=="mac" (
72+
IF "%SCRIPTING_BACKEND_IL2CPP_MONO%"=="il2cpp" (
73+
C:/TestingEditor/Unity.exe -projectPath C:/ClonedProject/{{ netcodeProject[1].projectPath }} -buildTarget osx -executeMethod BuilderScripts.BuildMacIl2cpp -batchmode -logFile ./artifacts/UnityLog.txt -automated -crash-report-folder ./artifacts/CrashArtifacts -quit
74+
) ELSE (
75+
C:/TestingEditor/Unity.exe -projectPath C:/ClonedProject/{{ netcodeProject[1].projectPath }} -buildTarget osx -executeMethod BuilderScripts.BuildMacMono -batchmode -logFile ./artifacts/UnityLog.txt -automated -crash-report-folder ./artifacts/CrashArtifacts -quit
76+
)
77+
)
78+
ELSE IF "%PLATFORM_WIN64_MAC_ANDROID%"=="android" (
79+
C:/TestingEditor/Unity.exe -projectPath C:/ClonedProject/{{ netcodeProject[1].projectPath }} -buildTarget android -executeMethod BuilderScripts.BuildAndroidIl2cpp -batchmode -logFile ./artifacts/UnityLog.txt -automated -crash-report-folder ./artifacts/CrashArtifacts -quit
80+
)
81+
82+
# Because of this we need to ensure that all files are copied to the source directory.
83+
# TODO: this can be omitted if I can somehow build the project in the source directory (YAMATO_SOURCE_DIR) instead of C:/CompetitiveAction
84+
- python -c "import os; os.makedirs('./build', exist_ok=True)" # --> Create the build directory if it doesn't exist
85+
- python -c "import shutil; shutil.copytree('C:/ClonedProject/{{ netcodeProject[1].projectPath }}/build', './build', dirs_exist_ok=True)" # --> Copy the build directory to the source directory (YAMATO_SOURCE_DIR). Remember to copy entire directory and not only exe file
86+
87+
artifacts:
88+
logs:
89+
paths:
90+
- '*.log'
91+
- '*.xml'
92+
- artifacts/**/*
93+
players:
94+
paths:
95+
- build/**/*
96+
{% endfor -%}
Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
using System.Collections.Generic;
2+
using UnityEditor;
3+
using UnityEditor.Build;
4+
using UnityEngine;
5+
using System;
6+
using System.Collections;
7+
using UnityEngine.Rendering;
8+
using UnityEngine.SceneManagement;
9+
10+
// This script is used to automate the build process for different platforms.
11+
// When included in the project, it can be triggered from the script the game for teh given configuration.
12+
// Note that it's possible to have those as a button in the editor (see https://github.cds.internal.unity3d.com/unity/Megacity-Metro/blob/c3b1b16ff1f04f96fbfbcc3267696679ad4e8396/Megacity-Metro/Assets/Scripts/Utils/Editor/BuilderScript.cs)
13+
// Ideally we would like to pass scripting backend and platform as parameters instead of having different script per each combintation but the nature of calling this nmethod via script (via -executeMethod) is that it needs to be static and the parameters are not passed in a way that we can use them.
14+
// TODO: add iOS support
15+
public class BuilderScripts : MonoBehaviour
16+
{
17+
[MenuItem("Tools/Builder/Build Development Windows Il2cpp")]
18+
static void BuildWinIl2cpp()
19+
{
20+
// This part modifies Player Settings. We only use it (for our case) to modify scripting backend
21+
PlayerSettings.SetScriptingBackend(NamedBuildTarget.Standalone, ScriptingImplementation.IL2CPP);
22+
PlayerSettings.SetUseDefaultGraphicsAPIs(BuildTarget.StandaloneWindows64, false); // disable auto graphic to use our own custom list
23+
PlayerSettings.SetGraphicsAPIs(BuildTarget.StandaloneWindows64, new []{GraphicsDeviceType.Vulkan, GraphicsDeviceType.Direct3D11, GraphicsDeviceType.Direct3D12}); // We need to specify the graphics API for Android builds to ensure proper shader compilation. Vulkan is recommended for modern devices.
24+
25+
// Below you can see additional settings that you can apply to the build:
26+
//PlayerSettings.SetGraphicsAPIs(BuildTarget.StandaloneWindows64, new []{GraphicsDeviceType.Direct3D12});
27+
//PlayerSettings.SetArchitecture(NamedBuildTarget.Standalone,0);
28+
29+
// The settings that we applied above need to be saved.
30+
AssetDatabase.SaveAssets();
31+
32+
// We want to build all scenes in build settings, so we collect them here.
33+
// If you want to build only specific scenes, then you could just use something like buildPlayerOptions.scenes = new[] { "Assets/Scenes/Menu.unity","Assets/Scenes/Main.unity" }; below.
34+
List<string> scenesToAdd = new List<string>();
35+
foreach (var scene in EditorBuildSettings.scenes)
36+
{
37+
if (scene.enabled)
38+
{
39+
Debug.Log("Adding scene to build: " + scene.path);
40+
scenesToAdd.Add(scene.path);
41+
}
42+
}
43+
44+
// This is an equivalent of BuildPlayerOptions in the Unity Editor.
45+
// We want to choose development build, what platform are we targetting, where to save the build and which scenes to include.
46+
// Some of those options can be omitted when triggering this script from withing GUI since more implicit context is provided (targetGroup, subtarget)
47+
// subtarget = (int)StandaloneBuildSubtarget.Player
48+
// targetGroup = BuildTargetGroup.Standalone
49+
// extraScriptingDefines = new[] { "NETCODE_DEBUG", "UNITY_CLIENT" },
50+
var buildPlayerOptions = new BuildPlayerOptions
51+
{
52+
locationPathName = "./build/Windows_Il2cpp/PlaytestBuild.exe",
53+
target = BuildTarget.StandaloneWindows64,
54+
options = BuildOptions.Development,
55+
scenes = scenesToAdd.ToArray()
56+
};
57+
58+
BuildPipeline.BuildPlayer(buildPlayerOptions);
59+
}
60+
61+
[MenuItem("Tools/Builder/Build Development Windows Mono")]
62+
static void BuildWinMono()
63+
{
64+
PlayerSettings.SetScriptingBackend(NamedBuildTarget.Standalone, ScriptingImplementation.Mono2x);
65+
PlayerSettings.SetUseDefaultGraphicsAPIs(BuildTarget.StandaloneWindows64, false); // disable auto graphic to use our own custom list
66+
PlayerSettings.SetGraphicsAPIs(BuildTarget.StandaloneWindows64, new []{GraphicsDeviceType.Vulkan, GraphicsDeviceType.Direct3D11, GraphicsDeviceType.Direct3D12}); // We need to specify the graphics API for Android builds to ensure proper shader compilation. Vulkan is recommended for modern devices.
67+
68+
AssetDatabase.SaveAssets();
69+
70+
List<string> scenesToAdd = new List<string>();
71+
foreach (var scene in EditorBuildSettings.scenes)
72+
{
73+
if (scene.enabled)
74+
{
75+
Debug.Log("Adding scene to build: " + scene.path);
76+
scenesToAdd.Add(scene.path);
77+
}
78+
}
79+
80+
var buildPlayerOptions = new BuildPlayerOptions
81+
{
82+
locationPathName = "./build/Windows_Mono/PlaytestBuild.exe",
83+
target = BuildTarget.StandaloneWindows64,
84+
options = BuildOptions.Development,
85+
scenes = scenesToAdd.ToArray()
86+
};
87+
88+
BuildPipeline.BuildPlayer(buildPlayerOptions);
89+
}
90+
91+
[MenuItem("Tools/Builder/Build Development Mac Mono")]
92+
static void BuildMacMono()
93+
{
94+
PlayerSettings.SetScriptingBackend(NamedBuildTarget.Standalone, ScriptingImplementation.Mono2x);
95+
PlayerSettings.SetUseDefaultGraphicsAPIs(BuildTarget.StandaloneOSX, false); // disable auto graphic to use our own custom list
96+
PlayerSettings.SetGraphicsAPIs(BuildTarget.StandaloneOSX, new []{GraphicsDeviceType.Metal}); // enforcing Metal Graphics API. Without this there will be shader errors in the final build.
97+
98+
AssetDatabase.SaveAssets();
99+
100+
List<string> scenesToAdd = new List<string>();
101+
foreach (var scene in EditorBuildSettings.scenes)
102+
{
103+
if (scene.enabled)
104+
{
105+
Debug.Log("Adding scene to build: " + scene.path);
106+
scenesToAdd.Add(scene.path);
107+
}
108+
}
109+
110+
var buildPlayerOptions = new BuildPlayerOptions
111+
{
112+
locationPathName = "./build/macOS_Mono/PlaytestBuild.app",
113+
target = BuildTarget.StandaloneOSX,
114+
options = BuildOptions.Development,
115+
scenes = scenesToAdd.ToArray()
116+
};
117+
118+
BuildPipeline.BuildPlayer(buildPlayerOptions);
119+
}
120+
121+
[MenuItem("Tools/Builder/Build Development Mac Il2cpp")]
122+
static void BuildMacIl2cpp()
123+
{
124+
PlayerSettings.SetScriptingBackend(NamedBuildTarget.Standalone, ScriptingImplementation.IL2CPP);
125+
PlayerSettings.SetGraphicsAPIs(BuildTarget.StandaloneOSX, new []{GraphicsDeviceType.Metal}); // enforcing Metal Graphics API. Without this there will be shader errors in the final build.
126+
127+
AssetDatabase.SaveAssets();
128+
129+
List<string> scenesToAdd = new List<string>();
130+
foreach (var scene in EditorBuildSettings.scenes)
131+
{
132+
if (scene.enabled)
133+
{
134+
Debug.Log("Adding scene to build: " + scene.path);
135+
scenesToAdd.Add(scene.path);
136+
}
137+
}
138+
139+
var buildPlayerOptions = new BuildPlayerOptions
140+
{
141+
locationPathName = "./build/macOS_Il2cpp/PlaytestBuild.app",
142+
target = BuildTarget.StandaloneOSX,
143+
options = BuildOptions.Development,
144+
scenes = scenesToAdd.ToArray()
145+
};
146+
147+
BuildPipeline.BuildPlayer(buildPlayerOptions);
148+
}
149+
150+
[MenuItem("Tools/Builder/Build Development Android Il2cpp")]
151+
static void BuildAndroidIl2cpp()
152+
{
153+
PlayerSettings.SetScriptingBackend(NamedBuildTarget.Android, ScriptingImplementation.IL2CPP);
154+
PlayerSettings.SetApplicationIdentifier(NamedBuildTarget.Android, "com.UnityTestRunner.UnityTestRunner"); // This is needed only for mobiles since by default the application identifier quite often contains invalid characters like spaces so we wan't to make sure that this has a valid value. It's needed only for mobile since that's an app store requirement
155+
PlayerSettings.SetUseDefaultGraphicsAPIs(BuildTarget.Android, false); // disable auto graphic to use our own custom list
156+
PlayerSettings.SetGraphicsAPIs(BuildTarget.Android, new []{GraphicsDeviceType.Vulkan}); // We need to specify the graphics API for Android builds to ensure proper shader compilation. Vulkan is recommended for modern devices.
157+
PlayerSettings.SetArchitecture(BuildTargetGroup.Android,1); // An integer value associated with the architecture of the build target. 0 - None, 1 - ARM64, 2 - Universal. most modern Android devices use the ARM64 architecture
158+
159+
AssetDatabase.SaveAssets();
160+
161+
List<string> scenesToAdd = new List<string>();
162+
foreach (var scene in EditorBuildSettings.scenes)
163+
{
164+
if (scene.enabled)
165+
{
166+
Debug.Log("Adding scene to build: " + scene.path);
167+
scenesToAdd.Add(scene.path);
168+
}
169+
}
170+
171+
var buildPlayerOptions = new BuildPlayerOptions
172+
{
173+
locationPathName = "./build/Android_Il2cpp_Vulkan/PlaytestBuild.apk",
174+
target = BuildTarget.Android,
175+
options = BuildOptions.Development,
176+
scenes = scenesToAdd.ToArray()
177+
};
178+
179+
BuildPipeline.BuildPlayer(buildPlayerOptions);
180+
}
181+
}

0 commit comments

Comments
 (0)