chore: Small yamato tweaks to try speed up our PR trigger runs #3711

EmandM · 2025-10-10T22:54:47Z

Purpose of this PR

I've spent too much time staring at yamato over #3647 . Decided to see if we can tweak our PR trigger to be more efficient.

Jira ticket

n/a

Changelog

Removed ::GPU from the agent.type for desktop_standalone_build, desktop_standalone_test and cmb_service_standalone_test. All of these jobs disable the GPU so they don't need a machine with a GPU (gpu machines are much rarer)
Swapped the PR trigger to run desktop_standalone_test_testproject_ubuntu_il2cpp_trunk rather than desktop_standalone_test_testproject_win_il2cpp_6000.0 that way the desktop_standalone_test and the cmb_service_standalone_test can use the same desktop_standalone_build step rather than needing two different build steps
- This does mean we no longer run PR tests on ubuntu_6000.0.
- I think this is fine as mac and windows are by far the most used platforms.
- We will still test ubuntu_6000.0 on the nightly and weekly builds.
Defined a smaller_flavor field in test_platforms.desktop to use with lighter weight jobs (package_pack, package_tests, project_pack). I chose which jobs to use a smaller VM from looking at the machine usage stats on bokken (like I say, I stared at yamato TOO MUCH this week).
Updated upm-ci to UnifiedTestRunner for the project_pack and project_test jobs.

Documentation

No documentation changes or additions were necessary.

Testing & QA (How your changes can be verified during release Playtest)

No testing needed.

Functional Testing

Manual testing :

Manual testing done

Automated tests:

Covered by existing automated tests
Covered by new automated tests

Does the change require QA team to:

Review automated tests?
Execute manual tests?
Provide feedback about the PR?

If any boxes above are checked the QA team will be automatically added as a PR reviewer.

Backports

Will backport to 1.x because faster tests makes for happier devs.

…service

… into chore/yamato-tweaks Add update of NetworkSceneManagerEventNotifications for cmb to test the whole PR trigger

…low us to only rerun failed tests

…aviour

…ier unity standards check

… it last so the other two are earlier in the VM queue

… standalone and cmb retry functionality

michalChrobot · 2025-10-12T13:25:19Z

.yamato/vetting-test.yml

 vetting_test:
  name: NGO - Vetting Test (Win, {{editor}} LTS)
-  agent: { type: Unity::VM, flavor: b1.large, image: package-ci/win11:v4 }
+  agent: { type: Unity::VM, flavor: b1.medium, image: package-ci/win11:v4 }


Fair, to be hones we could even try with small since vetting test is analyzing the PR for the release (API check etc) so maybe nothing really memory expensive is being run

I was looking at the CPU usage and the vetting test does end up using a fair amount of CPU so I left it at medium
(Here's the machine usage for this job). We could try it on small and see what it looks like?

michalChrobot · 2025-10-12T13:28:30Z

.yamato/project-updated-dependencies-test.yml

  dependencies:
-    - .yamato/project-pack.yml#project_pack_-_{{ project.name }}_{{ platform.name }}
+    # - .yamato/project-pack.yml#project_pack_-_{{ project.name }}_{{ platform.name }}
+    - .yamato/package-pack.yml#package_pack_-_ngo_{{ platform.name }}


Will this work? (packing package instead of project and all its packages)

Let's make sure to validate that this test passes (this is the same question per each "project related" test)

If you look at the logs of the project_pack job, it only packed our package anyway 🙃

michalChrobot · 2025-10-12T13:31:34Z

.yamato/project-tests.yml

-    - npm install upm-ci-utils@stable -g --registry https://artifactory.prd.cds.internal.unity3d.com/artifactory/api/npm/upm-npm # upm-ci is not preinstalled on the image so we need to download it
    - unity-downloader-cli --fast --wait -u {{ editor }} -c Editor {% if platform.name == "mac" %} --arch arm64 {% endif %} # For macOS we use ARM64 models. Installing basic editor for tests execution
-    - upm-ci project test -u {{ editor }} --project-path {{ project.path }} --type project-tests --extra-utr-arg="--reruncount=1 --clean-library-on-rerun" # project tests execution via upm-ci
+    - UnifiedTestRunner --testproject={{ project.path }} --suite=editor --artifacts-path=test-results --editor-location=.Editor --rerun-strategy=Test --retry=1  --timeout=1800


You can use --suite=editor --suite=playmode so both kind of tests are being run.

Don't we need to specify the editor in the command? (I don't have perfect memory for those configurations so some questions may seem basic)

I think rerun-strategy is something new since I don't remember seeing it before

--rerun-strategy will let us only rerun the tests that failed rather than the whole suite!. But only running the test that failed only works when you run the two suites separately for some reason.

In these logs for package_tests you can see that ValidateUrlsAreValid was run twice and the other tests are only run once.

michalChrobot · 2025-10-12T13:34:46Z

.yamato/performance-tests.yml

 {% endif %}
  commands:
      - unity-downloader-cli -u {{ editor }} -c Editor --wait {% if platform.name == "mac" %} --arch arm64 {% endif %} # For macOS we use ARM64 models. Installing basic editor
-      - UnifiedTestRunner --suite=editor --suite=playmode --testproject={{ project.path }} --editor-location=.Editor --timeout=3600 --artifacts-path=artifacts --extra-editor-arg=-assemblyNames --extra-editor-arg=Unity.NetCode.* --extra-editor-arg=-testCategory --extra-editor-arg=Performance --extra-editor-arg=-enablePackageManagerTraces --reruncount=1 --clean-library-on-rerun --dontreportperformancedata


typo? I know there is some issue with regeneration sometimes when it changes this param. Same question to other scripts where it changed

I'm putting it back! I thought it was this argument that made it rerun all the tests when one failed but it's not this

michalChrobot · 2025-10-12T13:35:50Z

.yamato/mobile-standalone-test.yml

    # Run tests for Android devices
    - |
      set ANDROID_DEVICE_CONNECTION=%BOKKEN_DEVICE_IP%
-      UnifiedTestRunner --suite=playmode --platform={{ platform.standalone }} --artifacts-path=test-results --player-load-path=build/players --testproject={{ project.path }} --editor-location=.Editor --player-connection-ip=%BOKKEN_HOST_IP% --fail-on-assert --reruncount=1 --clean-library-on-rerun --timeout=3600


Why to remove it? Same question for other instances

michalChrobot · 2025-10-12T13:38:16Z

.yamato/_triggers.yml

    # Run standalone test. We run it only on Ubuntu since it's the fastest machine, and it was noted that for example distribution on macOS is taking 40m since we switched to Apple Silicon
    # Coverage on other standalone machines is present in Nightly job so it's enough to not run all of them for PRs
-    - .yamato/desktop-standalone-tests.yml#desktop_standalone_test_testproject_win_il2cpp_6000.0
+    - .yamato/desktop-standalone-tests.yml#desktop_standalone_test_testproject_ubuntu_il2cpp_trunk


Any reason why to run this on trunk instead of the "minimal supported editor"?

It re-uses the same standalone-build step as the CMB tests so we don't need to distribute and run two separate standalone-build steps

michalChrobot · 2025-10-12T13:41:15Z

.yamato/_run-all.yml

+    # ensure the code is running to our current standards
    - .yamato/project-standards.yml#standards_ubuntu_testproject_trunk
-    # Run API validation to early-detect all new APIs that would force us to release new minor version of the package. Note that for this to work the package version in package.json must correspond to "actual package state" which means that it should be higher than last released version
-    - .yamato/vetting-test.yml#vetting_test


Any reason why to remove it?

The configuration is a bit tricky in the flow but those jobs run before any other job gets executed so we take resources only if we don't have any basic mistakes. Vetting test will fail when for example something gets flagged with new API

Note that if this will pass and we declare it as dependency later (in the trigger) then Yamato will just reuse the result of this job so we are not wasting any resources/time here

The Vetting test was generally taking longer to run than the standards test (Standards check is normally 5m, Vetting test is normally 9m). So far, it's also been some thing that fails very rarely.

I also moved the vetting test to a smaller machine than before so it'll get distributed faster. That means the job was running a bit slower again. I was finding it a bit slow to be a quick initial check, and seeing as it shouldn't fail very often, I was thinking it made more sense to run not as a "quick" check.

EmandM added 2 commits October 10, 2025 13:54

chore: Update NetworkSceneManagerEventNotifications to work with CMB …

146e9cc

…service

chore: Small yamato tweaks to try speed up our PR trigger runs

816fea9

EmandM requested review from a team and NoelStephensUnity as code owners October 10, 2025 22:54

EmandM added the port:1.x-needed This issue needs to be ported to 1.X branch label Oct 10, 2025

Merge branch 'chore/fix-NetworkSceneManagerEventNotifications-for-da'…

469f7cc

… into chore/yamato-tweaks Add update of NetworkSceneManagerEventNotifications for cmb to test the whole PR trigger

EmandM requested a review from a team as a code owner October 10, 2025 23:08

EmandM added 21 commits October 10, 2025 19:12

re-add the dependency on in

790bc62

Ensure CMB Service tests are running on nightly and weekly jobs

2aa215e

Re-add update to project-tests

5a240a9

Swap --clean-library-on-rerun to --clean-library to see if that'll al…

96c9611

…low us to only rerun failed tests

Force NetworkSceneManagerEventNotifications to fail to test rerun beh…

59463bf

…aviour

Run the lighter dotnet standards check first, before running the heav…

ecb9bce

…ier unity standards check

Remove empty folder and use correct pvp argument

46e042f

See if standards check requires the editor

7dc9799

package_pack runs faster than vetting_test or standards_check. Define…

49b35d2

… it last so the other two are earlier in the VM queue

Put project_pack back to upm-ci

4731ff9

Try running the vetting test on a smaller vm

255a05c

move vetting_test out of quick_initial_checks

14ee396

Add run_quick_checks to desktop_standalone dependencies

2390cbc

I think project_pack does nothing??

40c759d

Remove --clean-library from any test runs

7b2d2c8

try swapping reruncount to retry

37aff39

Remove --clean-library from project_tests

454cf58

Separate editor and playmode tests

c0ff854

try something else

acbebff

Use the working retry syntax

753aa65

Remove runtime tests from project_tests and package_tests to test the…

56098df

… standalone and cmb retry functionality

michalChrobot requested changes Oct 12, 2025

View reviewed changes

Try --clean-library-on-rerun

6b06a12

svc-netcode-sdk requested a review from a team October 13, 2025 13:56

EmandM added 3 commits October 13, 2025 10:31

Put back --clean-library-on-rerun

8703ac6

completely delete project-pack

fccced3

Revert failure in HelpUrlTests

11aa246

chore: Small yamato tweaks to try speed up our PR trigger runs #3711

Are you sure you want to change the base?

chore: Small yamato tweaks to try speed up our PR trigger runs #3711

Uh oh!

Conversation

EmandM commented Oct 10, 2025

Purpose of this PR

Jira ticket

Changelog

Documentation

Testing & QA (How your changes can be verified during release Playtest)

Functional Testing

Backports

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EmandM Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EmandM Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

EmandM Oct 13, 2025 •

edited

Loading

EmandM Oct 13, 2025 •

edited

Loading