ci: [NGOv1.X] Corrections of CI setup issues

.yamato/README.md

@@ -0,0 +1,80 @@
+# Netcode for GameObjects CI Documentation
+
+## Overview
+This document provides an overview of the Continuous Integration (CI) implementation for Netcode for GameObjects.
+Specifics of each test are described within related files (for example .yamato/package-tests.yml) and this file present high level overview related to overall implementation.
+
+## Test Configurations
+CI related files are present inside .yamato/ folder and we can distinguish specific tests
+
+### Helper jobs
+- `.yamato/package-pack.yml` responsible for generating package artifacts (.tgz) required for testing and publishing.
+- `.yamato/project-pack.yml` responsible for generating package artifacts (.tgz) required for testing and publishing. This packs all packages of a given project.
+- `.yamato/_run-all.yml` responsible for grouping tests into groups for easier management (for example "all console tests").
+- `.yamato/_triggers.yml` responsible for defining triggers (PR, nightly, weekly etc.) and defining which tests to run.
+- `disable-burst-if-requested.py` responsible for helping to disable burst if needed.
+- `.yamato/project.metafile` being the central configuration file defining test environments, platforms, projects, editors and other parameters used.
+
+### In-Editor Tests
+- `.yamato/package-tests.yml` responsible for running package tests.
+- `.yamato/project-tests.yml` responsible for running project tests.
+- `.yamato/project-updated-dependencies-test.yml` responsible for running tests after updating dependencies to latest ones.
+- `.yamato/project-standards.yml` responsible for running standards check on a package.
+- `.yamato/code-coverage.yml` responsible for running code coverage analysis.
+- `.yamato/performance-tests.yml` responsible for running performance tests.
+- `.yamato/webgl-build.yml` responsible for making a build for webgl platform.
+
+### Standalone Tests
+- `.yamato/desktop-standalone-tests.yml` responsible for running tests on standalone desktops.
+- `.yamato/console-standalone-test.yml` responsible for running tests on standalone consoles.
+- `.yamato/mobile-standalone-test.yml` responsible for running tests on standalone mobiles.
+
+### Wrench jobs
+This file is auto generated by wrench and used for automating release process.
+Those tests can also be used by our CI as dependencies.
+- `.yamato/wrench/api-validation-jobs.yml` responsible for validating api for a given package version (for example if there are no breaking changes when updating patch version)
+- `.yamato/wrench/package-pack-jobs.yml` responsible for generating package artifacts (.tgz) required for testing and publishing.
+- `.yamato/wrench/preview-a-p-v.yml` responsible for validating PVP (Package Verification Profiles) standards for the package.
+- `.yamato/wrench/promotion-jobs.yml` responsible for publishing.
+- `.yamato/wrench/publish-trigger.yml` responsible for defining jobs that need to pass in order to publish.
+- `.yamato/wrench/recipe-regeneration.yml` responsible for verifying regeneration of wrench recipes (no manual changes).
+- `.yamato/wrench/validation-jobs.yml` responsible for running package tests.
+- `.yamato/wrench/wrench_config.json` this is the configuration file for wrench.
+- `.yamato/generated-scripts/` is a collection of infrastructure instability detection tools.
+
+## Jobs Configuration Structure
+This section explains some configuration details that may be confusing while looking at .yml files.
+
+### Job Generation
+Our configuration uses a dynamic job generation approach based on the `.yamato/project.metafile` configuration. While it may appear that only one job is defined per file, the system (Yamato) actually generates multiple jobs through nested loops.
+Usually only 1 job is defined and then generated with multiple configurations/parameters but in case of standalone tests 2 jobs are defined which are responsible for building and then testing.
+
+### Job Naming Convention
+Because the jobs are being generated their names will be filled by given parameters from project.metfaile. For example jobs from .yamato/console-standalone-test may have names like:
+- Display Name (in Yamato): `Build testproject - [win, 6000.0, il2cpp]`
+- Internal Job Name (used when adding dependency for example): `console_standalone_build_testproject_win_6000.0`
+
+## Platform Support
+Currently, the CI implementation supports the following platforms:
+1. Console Platforms:
+    - Switch
+    - PS4
+    - PS5
+    - Xbox360
+    - XboxOne
+2. Mobile Platforms:
+    - Android
+    - iOS
+3. Desktop Platforms:
+    - Windows
+    - Ubuntu
+    - macOS
+4. Unity Editor Versions
+    - Supports NGOv1.X (2021.3+ editors)
+5. Architectures
+    - x64
+    - ARM64 (This is present for consoles/mobiles but will be extended. More information is present in specific standalone test files)
+
+## Design Considerations
+In theory, we could manually write jobs for every configuration. However, this approach would be more error-prone, especially when modifications or fixes are needed, as it would require keeping track of all configurations.
+The downside of our current approach is that it can sometimes impact readability due to the use of nested if and for statements.

.yamato/_run-all.yml

@@ -1,6 +1,16 @@
-{% metadata_file .yamato/project.metafile %}
+{% metadata_file .yamato/project.metafile %} # All configuration that is used to create different configurations (used in for loops) is taken from this file.
 ---
 
+# DESCRIPTION--------------------------------------------------------------------------
+  # Those jobs group together related test as dependencies to allow to easily manage running a given set of tests.
+  # This enables efficient test execution for different validation scenarios
+
+# QUALITY CONSIDERATIONS---------------------------------------------------------------
+  # It's important to ensure that all dependencies exist (this can be verified in Yamato) since a modification in parameters may result in a given job not being generated, and thus we will not be able to run such erroneous job.
+
+#-----------------------------------------------------------------------------------
+
+
 # Runs all package tests
 run_all_package_tests:
   name: Run All Package Tests
@@ -130,13 +140,7 @@ run_all_project_tests_desktop_standalone:
 {% for platform in test_platforms.desktop -%}
 {% for editor in validation_editors.all -%}
 {% for backend in scripting_backends -%}
-# There is an error 'No usable version of libssl' when it comes to building with combination of ubuntu and 2021 editor. It's not really worth to address it since 
-# 1) The 2021 editor will soon be out of support
-# 2) We still have the same coverage by using mac and ubuntu is used in a whole bunch of different tests, so I'm quite sure it won't lead to any bug leaks
-# 3) The solution could be to manually install libss1.1 (since image has libssl3 and this causes conflicts on 2021 editor related to dotnet versions and support). There were several threads on this topic which can be looked on
-{% if editor != "2021.3" or platform.name != "ubuntu" %}
     - .yamato/desktop-standalone-tests.yml#desktop_standalone_test_{{ project.name }}_{{ platform.name }}_{{ backend }}_{{ editor }}
-{% endif -%}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -164,13 +168,7 @@ run_all_project_tests_desktop_standalone_2021:
 {% for project in projects.default -%}
 {% for platform in test_platforms.desktop -%}
 {% for backend in scripting_backends -%}
-# There is an error 'No usable version of libssl' when it comes to building with combination of ubuntu and 2021 editor. It's not really worth to address it since 
-# 1) The 2021 editor will soon be out of support
-# 2) We still have the same coverage by using mac and ubuntu is used in a whole bunch of different tests, so I'm quite sure it won't lead to any bug leaks
-# 3) The solution could be to manually install libss1.1 (since image has libssl3 and this causes conflicts on 2021 editor related to dotnet versions and support). There were several threads on this topic which can be looked on
-{% if platform.name != "ubuntu" %}
     - .yamato/desktop-standalone-tests.yml#desktop_standalone_test_{{ project.name }}_{{ platform.name }}_{{ backend }}_2021.3
-{% endif -%}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -182,7 +180,7 @@ run_all_project_tests_mobile_standalone:
 {% for project in projects.default -%}
 {% for platform in test_platforms.mobile_test -%}
 {% for editor in validation_editors.all -%}
-    - .yamato/mobile-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_{{ editor }}
+    - .yamato/mobile-standalone-test.yml#mobile_standalone_test_{{ project.name }}_{{ platform.name }}_{{ editor }}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -195,7 +193,7 @@ run_all_project_tests_mobile_standalone_trunk:
 {% for project in projects.default -%}
 {% for platform in test_platforms.mobile_test -%}
 {% for editor in validation_editors.default -%}
-    - .yamato/mobile-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_{{ editor }}
+    - .yamato/mobile-standalone-test.yml#mobile_standalone_test_{{ project.name }}_{{ platform.name }}_{{ editor }}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -206,7 +204,7 @@ run_all_project_tests_mobile_standalone_2021:
   dependencies:
 {% for project in projects.default -%}
 {% for platform in test_platforms.mobile_test -%}
-    - .yamato/mobile-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_2021.3
+    - .yamato/mobile-standalone-test.yml#mobile_standalone_test_{{ project.name }}_{{ platform.name }}_2021.3
 {% endfor -%}
 {% endfor -%}
 
@@ -218,10 +216,7 @@ run_all_project_tests_console_standalone:
 {% for project in projects.default -%}
 {% for platform in test_platforms.console_test -%}
 {% for editor in validation_editors.all -%}
-# When building consoles with 2023 editor there are errors related to SDK version or uninstalled components. I will disable those for now and will add to my investigation list 
-{% if editor != "2023.3" %}
-    - .yamato/console-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_{{ editor }}
-{% endif -%}
+    - .yamato/console-standalone-test.yml#console_standalone_test_{{ project.name }}_{{ platform.name }}_{{ editor }}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -234,7 +229,7 @@ run_all_project_tests_console_standalone_trunk:
 {% for project in projects.default -%}
 {% for platform in test_platforms.console_test -%}
 {% for editor in validation_editors.default -%}
-    - .yamato/console-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_{{ editor }}
+    - .yamato/console-standalone-test.yml#console_standalone_test_{{ project.name }}_{{ platform.name }}_{{ editor }}
 {% endfor -%}
 {% endfor -%}
 {% endfor -%}
@@ -245,6 +240,6 @@ run_all_project_tests_console_standalone_2021:
   dependencies:
 {% for project in projects.default -%}
 {% for platform in test_platforms.console_test -%}
-    - .yamato/console-standalone-test.yml#run_{{ project.name }}_tests_{{ platform.name }}_2021.3
+    - .yamato/console-standalone-test.yml#console_standalone_test_{{ project.name }}_{{ platform.name }}_2021.3
 {% endfor -%}
 {% endfor -%}

.yamato/_triggers.yml

@@ -1,6 +1,43 @@
-{% metadata_file .yamato/project.metafile %}
+{% metadata_file .yamato/project.metafile %} # All configuration that is used to create different configurations (used in for loops) is taken from this file.
 ---
 
+# DESCRIPTION--------------------------------------------------------------------------
+  # This configuration defines three main CI trigger patterns:
+    # 1. Pull Request Validation: Validation performed on PR basis
+    # 2. Nightly Development: Test set run nightly (validates most important test cases)
+    # 3. Weekly Full Validation: Test set run weekly (validates all test cases to prevent any surprises)
+  # Each pattern represents different balance between validation depth, execution time and CI resource usage
+
+# TRIGGER PATTERNS-------------------------------------------------------------------
+  # Pull Request:
+    # This test validates Standards, Package tests, Project tests and Desktop standalone tests to ensure that main platforms are covered
+    # Triggers on PRs to develop, develop, and release branches
+    # Focuses on critical validation paths that we should validate before merging PRs
+    # Cancels previous runs on new commits
+    # Excludes draft PRs
+
+  # Nightly:
+    # This test validates same subset as pull_request_trigger with addition of mobile/console tests and webgl builds
+    # Runs daily on develop (local configuration)
+    # Includes all test types but only on trunk.
+    # Adds platform-specific and APV validation
+
+  # Weekly:
+    # This test validates same subset as develop_nightly but runs per all supported editors as well as executes code coverage test and runs project standards per project
+    # Runs across all supported editor versions
+    # Includes code coverage analysis
+    # Validates all projects and standards
+
+# CONFIGURATION STRUCTURE--------------------------------------------------------------
+    # Jobs configurations are generated by ensuring that all dependencies are successful.
+    # The dependencies are taken from _run-all.yml file where we can gather multiple tests into proper sets
+
+# QUALITY CONSIDERATIONS---------------------------------------------------------------
+    # It's important to ensure that all dependencies exist (this can be verified in Yamato) since a modification in parameters may result in a given job not being generated, and thus we will not be able to run such erroneous job.
+
+
+#-----------------------------------------------------------------------------------
+
 # Run all relevant tasks when a pull request targeting the develop or release branch is created or updated.
 pull_request_trigger:
   name: Pull Request Trigger (develop, develop-2.0.0, & release branches)
@@ -66,6 +103,9 @@ develop_nightly:
     # Build player for webgl platform on trunk and 2021 editors
     - .yamato/project-updated-dependencies-test.yml#updated-dependencies_testproject_NGO_ubuntu_trunk
     - .yamato/project-updated-dependencies-test.yml#updated-dependencies_testproject_NGO_win_2021.3
+    # Clean import test
+    - .yamato/clean-import-job.yml#clean_import_testproject_trunk
+    - .yamato/clean-import-job.yml#clean_import_testproject_2021.3
 
 
 # Run all tests on weekly bases

.yamato/code-coverage.yml

@@ -1,10 +1,30 @@
-{% metadata_file .yamato/project.metafile %}
+{% metadata_file .yamato/project.metafile %} # All configuration that is used to create different configurations (used in for loops) is taken from this file.
 ---
 
-# Runs package tests in order to determine code coverage of the NGO package.
-# In essence it's performing the same task as .yamato/package-test jobs with the overhead being the measured code coverage
-# It's ok for code coverage to be performed only on one platform (default) since code coverage won't change much between those.
-# Default platform was chosen (ubuntu) since it's the fastest and most resource friendly with default editor.
+# DESCRIPTION--------------------------------------------------------------------------
+  # This job is responsible for executing package tests with code coverage analysis enabled.
+  # Coverage analysis provides insights into:
+  # Test coverage metrics for NGO assemblies
+  # Line and branch coverage statistics
+  # Generated HTML reports for coverage visualization
+  # Additional metrics for coverage analysis
+
+# CONFIGURATION STRUCTURE--------------------------------------------------------------
+  # Jobs are generated using nested loops through:
+  # 1. For default platform only (Ubuntu) since coverage would not vary between platforms (no need for checks on more platforms)
+  # 2. For default editor version (trunk) since coverage would not vary between editors (no need for checks on more editors)
+
+#TECHNICAL CONSIDERATIONS---------------------------------------------------------------
+  # In theory this job also runs package tests, but we don't want to use it as default since is heavier (because of added coverage analysis) and coverage is not changing that often
+  # Requires Unity Editor installation
+  # Burst compilation is disabled to ensure accurate coverage measurement
+  # In order to properly use -coverage-results-path parameter we need to start it with $PWD (which means the absolute path). Otherwise, coverage results will not be visible
+
+# QUALITY CONSIDERATIONS--------------------------------------------------------------------
+  # TODO: somewhere in 2025 we will be able to upload resuls to CodeCov from public repos
+  # To see where this job is included (in trigger job definitions) look into _triggers.yml file
+
+
 {% for platform in test_platforms.default -%}
 {% for editor in validation_editors.default -%}
 code_coverage_{{ platform.name }}_{{ editor }}:
@@ -13,16 +33,18 @@ code_coverage_{{ platform.name }}_{{ editor }}:
     type: {{ platform.type }}
     image: {{ platform.image }}
     flavor: {{ platform.flavor }}
+{% if platform.model %}
+    model: {{ platform.model }} # This is set only in platforms where we want non-default model to use (more information in project.metafile)
+{% endif %}
   commands:
-    - npm install -g upm-ci-utils@stable --registry https://artifactory.prd.cds.internal.unity3d.com/artifactory/api/npm/upm-npm
-    - unity-downloader-cli -u {{ editor }} -c Editor --fast --wait
-    - DISPLAY=:0 upm-ci package test -u {{ editor }} --package-path com.unity.netcode.gameobjects --enable-code-coverage --code-coverage-options 'generateAdditionalMetrics;generateHtmlReport;assemblyFilters:+Unity.Netcode.Editor,+Unity.Netcode.Runtime' --extra-utr-arg="--extra-editor-arg=--burst-disable-compilation --extra-editor-arg=testCategory --extra-editor-arg=!Performance --timeout=1800 --reruncount=1 --clean-library-on-rerun"
+    - unity-downloader-cli --fast --wait -u {{ editor }} -c Editor {% if platform.name == "mac" %} --arch arm64 {% endif %} # For macOS we use ARM64 models
+    - upm-pvp create-test-project test-project --packages "upm-ci~/packages/*.tgz" --unity .Editor
+    - UnifiedTestRunner --suite=editor --suite=playmode --editor-location=.Editor --testproject=test-project --enable-code-coverage -coverage-results-path=$PWD/test-results/CodeCoverage --coverage-options="generateHtmlReport;generateAdditionalMetrics;assemblyFilters:+Unity.Netcode.Editor,+Unity.Netcode.Runtime" --extra-editor-arg=--burst-disable-compilation --timeout=1800 --reruncount=1 --clean-library-on-rerun --artifacts-path=test-results
   artifacts:
     logs:
       paths:
-        - "upm-ci~/test-results/**/*"
-        - "upm-ci~/test-results/CoverageResults/**/*"
+        - "test-results/**/*"
   dependencies:
-    - .yamato/package-pack.yml#package_pack_-_ngo_{{ platform.name }}_upmCI
+    - .yamato/package-pack.yml#package_pack_-_ngo_{{ platform.name }}
 {% endfor -%}
 {% endfor -%}