ci: Adding minimal required job set and validation preventing PRs from merging when tests are running (#3682)

This PR aims to address 3 problems that we currently have

1. Quite often if no checks are required it's easy to undertest a PR
which leads to simple whitespace errors and such. Because of that I
created `pr_minimal_required_checks` which consists of PVP checks and
standard check which I will require on all PRs as minimum
2. Emma noticed that PRs are allowed to be merged when jobs are running
but none yet failed. This is an issue created by
#3580
and the fact that we want to run CI conditionally (for example we don't
want to run it when we only have DOCS changes) so we had to remove PR
trigger as required job in branch protection rules. This creates a
problem that without required check PR is allowed to be merged until a
test fails
3. Another problem that I spotted is that Documentation~ folder is
actually located inside package folder so this check implemented there
won't work correctly either way

The solution that this PR proposes is implementation of 2 independent
checks (triggered by expression trigger) where

1. `pr_minimal_required_checks` will trigger on all PRs targetting
develop or release branches. It consist of simplest PVP and standards
checks
2. `pr_code_changes_checks` will trigger on PRs that are modifying files
under relevant package or project related paths (see expression trigger)

Those jobs WON'T be required by the branch protection rules but I
introduced new GitHub action of Yamato PR Supervisor which WILL be
required and how it works is that it monitors checks running on PRs and
will fail if any of them fails or pass if all checks are green. In this
way we can have a workflow that both runs only necessary tests but also
gates PRs from merging too early.

Note that you can still use `/ci ngo` comment trigger to run the check
on draft branch or branches not targeting develop nor release branches

After this is approved and merged I will add this check as required on
PRs in branch protection rules

N/A

Comments were added to the jobs (soon I also plan to create doc
describing our CI practices/learnings)

Playtest)
I tested scenarios with code changes and without code changes and the
workflow works as expected

Will do

Author: michalChrobot

.github/workflows/pr-supervisor.yaml

@@ -0,0 +1,66 @@
+# GitHub Actions workflow to monitor Yamato CI job state on pull requests
+# We are using https://cli.github.com/manual/gh_pr_checks
+# The aim is to ensure that conditionally triggered Yamato jobs are completed successfully before allowing merges
+
+# This job will be required in branch protection rules for develop, develop-2.0.0, and release/* branches. It's only goal will be to ensure that Yamato jobs are completed successfully before allowing Pr to merge.
+# Note that conditional jobs will have 30s to show which is always the cas since they are showing up as soon as in distribution stage.
+
+name: Yamato PR Supervisor
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches:
+      - develop
+      - develop-2.0.0
+      - release/*
+        
+concurrency:
+  group: pr-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  yamato-supervisor:
+    runs-on: ubuntu-latest
+    timeout-minutes: 720
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        
+      - name: Wait and Verify Yamato Job Status
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          set -e
+          
+          
+          MAX_ATTEMPTS=$((12*60))
+          INTERVAL=60
+          
+          sleep $INTERVAL
+          for ((i=1;i<=MAX_ATTEMPTS;i++)); do
+            echo "Polling PR checks (attempt $i/$MAX_ATTEMPTS)..."
+          
+            # We want to watch for pending checks beside this check
+            checks=$(gh pr checks $PR_NUMBER --json name,state --jq '[ .[] | select(.name != "yamato-supervisor") ]')
+            
+            pending=$(echo "$checks" | jq '[.[] | select(.state == "PENDING")] | length')
+            skipping=$(echo "$checks" | jq '[.[] | select(.state == "SKIPPED")] | length')
+            passed=$(echo "$checks" | jq '[.[] | select(.state == "SUCCESS")] | length')
+            failed=$(echo "$checks" | jq '[.[] | select(.state == "FAILURE")] | length')
+          
+            echo "Pending checks: $pending, Skipping checks: $skipping", Passed checks: $passed, Failed checks: $failed
+          
+            if [[ "$failed" -gt 0 ]]; then
+              echo "A check has failed! Failing fast."
+              exit 1
+            fi
+
+            if [[ "$pending" -eq 0 ]] && [[ "$passed" -gt 0 ]]; then
+              echo "All non-supervisor checks are completed!"
+              exit 0
+            fi
+          
+            sleep $INTERVAL
+          done

.yamato/_triggers.yml

@@ -4,20 +4,22 @@
 # DESCRIPTION--------------------------------------------------------------------------
   # This configuration defines three main CI trigger patterns:
     # 1. Pull Request Validation: Validation performed on PR basis
+      # - we have mandatory minimal checks that run on all PRs (even if only docs are changed)
+      # - we have more extensive pr_code_changes_checks that run if code is changed
     # 2. Nightly Development: Test set run nightly (validates most important test cases)
     # 3. Weekly Full Validation: Test set run weekly (validates all test cases to prevent any surprises)
   # Each pattern represents different balance between validation depth, execution time and CI resource usage
 
 # TRIGGER PATTERNS-------------------------------------------------------------------
   # Pull Request:
-    # This test validates Standards, Package tests, Project tests and Desktop standalone tests to ensure that main platforms are covered
-    # Focuses on critical validation paths that we should validate before merging PRs. It also cancels previous runs on new commits
-    # By default it's triggered if
+    # We have two PR triggers:
+      # 1) Minimal PR checks that run on all PRs (even if only docs are changed)
+      # 2) More extensive pr_code_changes_checks that run if code is changed. This test validates Standards, Package tests, Project tests and Desktop standalone tests to ensure that main platforms are covered
+    # By default pr_minimal_required_checks it's triggered if
       # 1) PR targets develop, develop-2.0.0 or release branches
       # 2) PR is not a draft
-      # 3) PR changes files in package or testproject folders (doesn't run on for example DOCS only changes)
-  
-    # Note that in other cases you can trigger it by writing a comment "/ci ngo" in the PR thread
+    # Then pr_code_changes_checks it's triggered if the same conditions apply plus:
+      # 1) PR changes code in com.unity.netcode.gameobjects package (Editor, Runtime or Tests folders) or in testproject folder or package.json file
 
   # Nightly:
     # This test validates same subset as pull_request_trigger with addition of mobile/console tests and webgl builds
@@ -39,14 +41,32 @@
     # It's important to ensure that all dependencies exist (this can be verified in Yamato) since a modification in parameters may result in a given job not being generated, and thus we will not be able to run such erroneous job.
 
 
-#-----------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------------  
+
+# After some experimenting with CI setups we discovered that even though sometimes we don't need CI to run (no reason to run package tests if only Documentation is changed) there are some checks that devs may not realize but changes in seemingly unrelated files will cause their failures
+# This trigger was created to ensure that ALL PRs run this minimal check even when we don't need to run full tests
+pr_minimal_required_checks:
+  name: Minimal PR checks
+  dependencies:
+    - .yamato/package-pack.yml#package_pack_-_ngo_win
+    - .yamato/project-standards.yml#standards_ubuntu_testproject_trunk
+  triggers:
+    expression: |-
+      (pull_request.comment eq "ngo" OR
+      (pull_request.target eq "develop" OR
+      pull_request.target eq "develop-2.0.0" OR
+      pull_request.target match "release/*")) AND
+      NOT pull_request.draft
+    cancel_old_ci: true
 
 
 # Run all relevant tasks when a pull request targeting the develop or release branch is created or updated.
 # In order to have better coverage we run desktop standalone tests with different configurations which allows to mostly cover for different platforms, scripting backends and editor versions.
-# Since standards job is a part of initial checks it's not present as direct dependency here
-pull_request_trigger:
-  name: Pull Request Trigger (develop, develop-2.0.0, & release branches)
+# This job will FIRST run "run_quick_checks" jobs (defined in _run-all.yml) since it's the dependency of project pack jobs which is on the lowest dependency tier. This runs the fastest checks (like PVP or code standards) and ONLY IF those pass it will run the rest of the tests.
+# This optimization allows to speed up feedback look for any "obvious" errors and save resources.
+# Since standards job is a part of initial checks it's not present as direct dependency here!!!!!!!!!!!!!!!!!!!!
+pr_code_changes_checks:
+  name: Code changes PR checks
   # Run the following tests on a selection of different desktop platforms
   dependencies:
     # Run package EditMode and Playmode package tests on 6000.2 (latest supported editor) and an older supported editor (2022.3) (2022.3 will soon be a minimum supported editor)
@@ -65,18 +85,31 @@ pull_request_trigger:
     # Coverage on other standalone machines is present in Nightly job so it's enough to not run all of them for PRs
     - .yamato/desktop-standalone-tests.yml#desktop_standalone_test_testproject_ubuntu_il2cpp_2022.3
   triggers:
-    # Note that PR tests will run ONLY if we are changing package/sample code. If changes are let's say docs only no tests will be triggered
-    # TODO: consider setting up and running tests from Examples/
-    # TODO: or docs only changes are spelling/link check
     expression: |-
-      pull_request.comment eq "ngo" OR
-      ((pull_request.target eq "develop" OR
+      (pull_request.comment eq "ngo" OR
+      (pull_request.target eq "develop" OR
       pull_request.target eq "develop-2.0.0" OR
-      pull_request.target match "release/*") AND
+      pull_request.target match "release/*")) AND
       NOT pull_request.draft AND
-      (pull_request.changes.any match "com.unity.netcode.gameobjects/**" OR
-      pull_request.changes.any match "testproject/**"))
+      pull_request.changes.any match [
+        "com.unity.netcode.gameobjects/Editor/**",
+        "com.unity.netcode.gameobjects/Runtime/**",
+        "com.unity.netcode.gameobjects/Tests/**",
+        "testproject/**",
+        "com.unity.netcode.gameobjects/package.json"
+      ] AND
+      NOT pull_request.changes.all match [
+        "**/Documentation~/**",
+        "**/InternalDocs~/**",
+        "**/*.md"
+      ]
     cancel_old_ci: true
+      
+      
+      
+
+
+
 
 # Run all tests on 6000.2 (latest supported editor) on nightly basis.
 # Same subset as pull_request_trigger with addition of mobile/desktop/console tests and webgl builds