diff --git a/docs/03-github-orchestrator/01-introduction.mdx b/docs/03-github-orchestrator/01-introduction.mdx index 3ae13efb..0f222435 100644 --- a/docs/03-github-orchestrator/01-introduction.mdx +++ b/docs/03-github-orchestrator/01-introduction.mdx @@ -1,104 +1,71 @@ # Introduction -## Concept - What Does Orchestrator Do +## What Does Orchestrator Do? -**Orchestrator enables you to run, build and test (Unity) projects in the cloud. You can start jobs -from the command line, the "Workbench" GUI in the Unity Editor or from GitHub Actions.** +**Orchestrator runs Unity builds on cloud infrastructure.** Start jobs from GitHub Actions, the +command line, or any CI system. Orchestrator provisions a cloud environment, sends your project to +be built, and streams results back. -**Orchestrator will automatically provision an environment at a Cloud Provider such as GCP and AWS. -It will then send the project to be built and/or tested depending on your workflow configuration.** - -**Orchestrator is especially useful for game development because it supports large projects. -Orchestrator provides first class support for the Unity game engine.** - -Orchestrator uses git to track and syncronize your projects and uses native cloud services such as -AWS Fargate and Kubernetes to run your jobs. Other version control systems are not actively -supported. - -## Why Orchestrator? - -1. **Orchestrator is flexible and elastic** - 1. _You can balance your use of high-performance and cost saving modes._ Configurable cost/speed - effeciency - 2. _Works great for projects of almost any size, from AAA projects and assets to micro projects_ - 3. _Extended configuration options._ More control over disk size, memory and CPU - 4. _Easily scale all job resources as your project grows_ e.g storage, CPU and memory -2. **Scale fully on demand from zero (not in use) to many concurrent jobs.** Benefits from - "pay-for-what-you-use" cloud billing models. We have made an effort to make sure that it costs - you nothing (aside from artifact and cache storage) while there are no builds running (no - guarantees) -3. **Easy setup and configuration** -4. **Run custom jobs** and extend the system for any workload - -## Why not orchestrator? - -1. Your project is small in size. Below 5GB Orchestrator should not be needed. -2. You already have dedicated servers running you can use. - -Although the speed of a CI pipelines is an important metric to consider, there are real challenges -for game development pipelines. - -This solution prefers convenience, ease of use, scalability, throughput and flexibility. - -Faster solutions exist, but would all involve self-hosted hardware with an immediate local cache of -the large project files and working directory and a dedicated server. - -# Orchestrator Release Status - -Orchestrator is in "active development" โš ๏ธ๐Ÿ”จ - -Orchestrator overall release status: `preview` This means some APIs may change, features are still -being added but the minimum feature set works and is stable. - -Release Stages: `experimental` โžก๏ธ `preview` โžก๏ธ `full release` - -You must use a provider with Orchestrator, each provider's release status is described below. This -indicates the stability and support for orchestrator features and workflows. - -### Supported Orchestrator Platforms - -```md -| Cloud Provider Platform | Release Status | -| ----------------------- | ------------------ | -| Kubernetes | โœ”๏ธ preview release | -| AWS | โœ”๏ธ full release | -| GCP | โš  Considered | -| Azure | โš  Considered | +``` + Your Machine / CI Cloud Provider + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” git push โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ GitHub โ”‚ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ AWS Fargate โ”‚ + โ”‚ Actions, โ”‚ โ”‚ Kubernetes โ”‚ + โ”‚ GitLab CI, โ”‚ โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚ Local Docker โ”‚ + โ”‚ CLI, etc. โ”‚ build artifacts โ”‚ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ + โ”‚ Orchestrator handles: โ”‚ + โ”‚ * Provisioning โ”‚ + โ”‚ * Git sync + LFS โ”‚ + โ”‚ * Caching (S3 / rclone) โ”‚ + โ”‚ * Automatic cleanup โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` -_Note for Kubernetes support:_ _Usually the cluster needs to be up and running at all times, as -starting up a cluster is slow._ _Use Google Cloud's Kubernetes Autopilot you can scale down to the -free tier automatically while not in use._ _Kubernetes support currently requires cloud storage, -only S3 support is built-in._ +Orchestrator supports large projects with first-class Unity support and native cloud services like +AWS Fargate and Kubernetes. -```md -| Git Platform | Release Status | -| --------------------- | ------------------ | -| GitHub | โœ”๏ธ full release | -| GitLab | โœ”๏ธ preview release | -| Command Line | โœ”๏ธ preview release | -| Any Git repository | โœ”๏ธ preview release | -| Any Git automation/Ci | โœ”๏ธ preview release | -``` +## โœ… Why Orchestrator? -## External Links +1. **Flexible and elastic** โ€” balance speed and cost, configure CPU, memory, and disk per build +2. **Scale from zero** โ€” no idle servers, pay only while builds run +3. **Easy setup** โ€” minimal configuration to [get started](getting-started) +4. **Extensible** โ€” run [custom hooks](advanced-topics/hooks/container-hooks), or bring your own + [provider plugin](providers/custom-providers) -### Orchestrator Releases +## โŒ When You Don't Need It -[Game CI Releases - GitHub](https://github.com/game-ci/unity-builder/releases) _Packaged and -released with game-ci's unity-builder module._ +- Your project is under 5 GB โ€” standard GitHub runners should work fine +- You have dedicated build servers already running -### Open Incoming Pull Requests +## ๐Ÿ“ฆ Supported Providers -[Orchestrator PRs - GitHub](https://github.com/game-ci/unity-builder/pulls?q=is%3Apr+orchestrator) +| Cloud Provider | Description | +| -------------------------------------- | -------------------------------------------------------- | +| [AWS Fargate](providers/aws) | Fully managed containers on AWS. No servers to maintain. | +| [Kubernetes](providers/kubernetes) | Run on any Kubernetes cluster. | +| [Local Docker](providers/local-docker) | Docker containers on the local machine. | +| [Local](providers/local) | Direct execution on the host machine. | -### ๐Ÿ’ฌSuggestions and ๐Ÿ›Bugs (GitHub Issues): +See [Providers](providers/overview) for the full list including [custom](providers/custom-providers) +and [community](providers/community-providers) providers. -[Game CI Issues - GitHub](https://github.com/game-ci/unity-builder/labels/orchestrator) +## ๐Ÿ–ฅ๏ธ Supported Platforms -### Community +| Platform | Description | +| ---------------------------------------------- | ------------------------------------- | +| [GitHub Actions](providers/github-integration) | First-class support with Checks. | +| [GitLab CI](providers/gitlab-integration) | Via the Command Line mode. | +| [Command Line](examples/command-line) | Run from any terminal or script. | +| Any CI system | Anything that can run shell commands. | -**Share your feedback with us!** +## ๐Ÿ”— External Links -- [**Discord Channel**](https://discord.com/channels/710946343828455455/789631903157583923) -- [**Feedback Form**](https://forms.gle/3Wg1gGf9FnZ72RiJ9) +- [Releases](https://github.com/game-ci/unity-builder/releases) โ€” packaged with + game-ci/unity-builder +- [Pull Requests](https://github.com/game-ci/unity-builder/pulls?q=is%3Apr+orchestrator) โ€” open + orchestrator PRs +- [Issues](https://github.com/game-ci/unity-builder/labels/orchestrator) โ€” bugs and feature requests +- [Discord](https://discord.com/channels/710946343828455455/789631903157583923) โ€” community chat +- [Feedback Form](https://forms.gle/3Wg1gGf9FnZ72RiJ9) โ€” share your experience diff --git a/docs/03-github-orchestrator/02-game-ci-vs-orchestrator.mdx b/docs/03-github-orchestrator/02-game-ci-vs-orchestrator.mdx index 369bc10e..e4ce3c61 100644 --- a/docs/03-github-orchestrator/02-game-ci-vs-orchestrator.mdx +++ b/docs/03-github-orchestrator/02-game-ci-vs-orchestrator.mdx @@ -1,42 +1,46 @@ -# Game-CI vs Orchestrator +--- +sidebar_label: Game-CI vs Orchestrator +--- -# Standard Game-CI (Use Cases) +# Standard Game-CI vs Orchestrator Mode -The Game CI core is a maintained set of docker images that can be used to run workloads in many -scenarios. +## ๐ŸŽฎ Standard Game-CI -Game CI also provides specific GitHub actions for running workflows on GitHub. And a similar -workflow for running Game CI on GitLab and Circle CI. _All of these options use the build server -resources provided by those systems, this can be a constraint or very convenient depending on the -size of your project and the workloads you need to run._ +Game CI provides Docker images and GitHub Actions for running Unity workflows on the build server +resources provided by your CI platform (GitHub, GitLab, Circle CI). -# Orchestrator (Use Cases) +**Best for:** Small to medium projects that fit within GitHub's resource limits. -## Sending Builds to the cloud +## โ˜๏ธ Orchestrator Mode -You may want to take advantage of cloud resources for lots of reasons (scale, speed, cost, -flexibility) or may want to start remote builds from the command line without slowing down your -development machine. Orchestrator can help you do this. +Orchestrator sends builds to cloud infrastructure (AWS Fargate, Kubernetes) instead of running on +the CI runner itself. This is useful when: -This may be a preference, more effecient or you may want to use systems that struggle to handle -large game development projects (GitHub being a good example). +- Your project **exceeds disk space limits** on GitHub-hosted runners +- You need **more CPU or memory** than the CI platform provides +- You want to **scale to many concurrent builds** without managing servers -### Large GitHub Projects +``` + Standard Game-CI Orchestrator Mode -GitHub Actions by default run on -[build machines provided by GitHub](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners). -For Unity projects the available disk size is quite small. You may experience an error related to -running out of disk space. You may also want to run the build on a server with more memory or -processing resources. + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ GitHub โ”‚ โ”‚ GitHub Action โ”‚ โ”‚ โ”‚ + โ”‚ Runner โ”‚ โ”‚ CLI / Any CI โ”‚โ”€โ”€โ”€โ–บโ”‚ Cloud โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ Container โ”‚ + โ”‚ (builds โ”‚ โ”‚ (dispatches โ”‚โ—„โ”€โ”€โ”€โ”‚ (builds โ”‚ + โ”‚ locally) โ”‚ โ”‚ only) โ”‚ โ”‚ remotely) โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + ~14 GB disk Configurable CPU, memory, disk + Fixed resources Scales to zero when idle +``` -### GitHub Self-Hosted Runners vs Game CI Orchestrator +## Self-Hosted Runners vs Orchestrator -_GitHub users can consider: -[GitHub self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) -and Orchestrator. Both can enable you to build larger projects._ +Both options let you build larger projects. Here's when to pick which: -_Orchestrator is better if you don't have a server setup or don't want to manage or maintain your -own build server._ - -_Self-hosted runners are best used when you already have a server available, running 24/7 that you -can setup as a runner. And you're happy to maintain and keep that server available and running._ +| | Self-Hosted Runners | Orchestrator | +| --------------- | ---------------------------------- | ------------------------------------- | +| **Setup** | Requires a dedicated server | Cloud account + credentials | +| **Maintenance** | You manage the server 24/7 | Fully managed, no servers to maintain | +| **Cost model** | Fixed (server always running) | Pay per build (scales to zero) | +| **Best for** | Teams with existing infrastructure | Teams without dedicated servers | diff --git a/docs/03-github-orchestrator/02-getting-started.mdx b/docs/03-github-orchestrator/02-getting-started.mdx index f14c1a59..def9b8ce 100644 --- a/docs/03-github-orchestrator/02-getting-started.mdx +++ b/docs/03-github-orchestrator/02-getting-started.mdx @@ -7,7 +7,7 @@ runners. This is useful for large projects that exceed GitHub's disk or resource - A Unity project in a GitHub repository - A cloud provider account (AWS or a Kubernetes cluster) -- Provider credentials configured as GitHub secrets +- Provider credentials configured as GitHub [secrets](secrets) ## Quick Start @@ -15,10 +15,11 @@ runners. This is useful for large projects that exceed GitHub's disk or resource 2. **Configure credentials** for your chosen provider 3. **Add the orchestrator step** to your workflow -See the provider-specific examples for complete setup: +See the provider-specific setup guides: -- [AWS Example](examples/github-examples/aws) -- [Kubernetes Example](examples/github-examples/kubernetes) +- [AWS Fargate](providers/aws) +- [Kubernetes](providers/kubernetes) +- [Local Docker](providers/local-docker) - [Command Line](examples/command-line) ## Minimal Example diff --git a/docs/03-github-orchestrator/03-examples/01-command-line.mdx b/docs/03-github-orchestrator/03-examples/01-command-line.mdx index 636144e9..9159f7d3 100644 --- a/docs/03-github-orchestrator/03-examples/01-command-line.mdx +++ b/docs/03-github-orchestrator/03-examples/01-command-line.mdx @@ -1,49 +1,67 @@ # Command Line -_Preview Support Only_ - You can install Game CI locally and start orchestrator jobs from the command line or by integrating -your own tools. All parameters in [API Reference](../api-reference) can be specified as command line -input fields. - -# Install +your own tools. All parameters in the [API Reference](../api-reference) can be specified as CLI +flags. -Currently (development) +## Install ```bash git clone https://github.com/game-ci/unity-builder.git +cd unity-builder yarn install -yarn run cli -m {mode parameter} --projectPath {Your project path} {... other command line parameters} ``` -# Planned (does not work currently) +## Usage + +```bash +yarn run cli -m --projectPath [options] +``` + +### Examples + +Run a standard build: + +```bash +yarn run cli -m cli-build \ + --projectPath /path/to/your/unity/project \ + --providerStrategy aws \ + --targetPlatform StandaloneLinux64 \ + --gitPrivateToken $GIT_TOKEN +``` + +List active resources: + +```bash +yarn run cli -m list-resources --providerStrategy aws +``` -We plan to offer support for Game CI via Deno. This will enable fast, typescript native runtime and -you will be able to access this via the following: +Watch a running build: ```bash -dpx game-ci build +yarn run cli -m watch --providerStrategy aws ``` -# Help +Clean up old resources: -_You can run `yarn run cli -h` or `yarn run cli --help` to list all modes and paramters with -descriptions_ +```bash +yarn run cli -m garbage-collect --providerStrategy aws +``` -# Main Command Parameters +## Help -- Default: `cli` (runs a standard build workflow) -- See API Reference "Modes" +Run `yarn run cli --help` to list all modes and parameters with descriptions. -# Keeping command line parameters short +## Keeping Commands Short -You can avoid specifying long command line input for credentials by using environment variables or -[the input override feature](../advanced-topics/configuration-override#example) to shorten commands -signficantly. +You can avoid long CLI flags for credentials by using environment variables or the +[Pull Secrets](../secrets#-pulling-secrets-from-external-sources) feature. -This enables you to provide a command to pull input, e.g you can pull from a file or from a secret -manager. +This lets you pull input from a secret manager: ```bash -yarn run cli --populateOverride true --pullInputList UNITY_EMAIL,UNITY_SERIAL,UNITY_PASSWORD --inputPullCommand="gcloud secrets versions access 1 --secret=\"{0}\"" +yarn run cli \ + --populateOverride true \ + --pullInputList UNITY_EMAIL,UNITY_SERIAL,UNITY_PASSWORD \ + --inputPullCommand='gcloud secrets versions access 1 --secret="{0}"' ``` diff --git a/docs/03-github-orchestrator/03-examples/02-github-actions.mdx b/docs/03-github-orchestrator/03-examples/02-github-actions.mdx new file mode 100644 index 00000000..e069d311 --- /dev/null +++ b/docs/03-github-orchestrator/03-examples/02-github-actions.mdx @@ -0,0 +1,338 @@ +# GitHub Actions + +Orchestrator has first-class GitHub Actions support. This page shows complete, copy-paste workflow +files for every provider. + +## ๐Ÿ”‘ Prerequisites + +1. A Unity project in a GitHub repository +2. Provider credentials stored as + [GitHub Actions secrets](https://docs.github.com/en/actions/security/encrypted-secrets) +3. A `UNITY_LICENSE` or activation secret (see the + [Game CI activation docs](https://game.ci/docs/github/activation)) + +## Minimal Workflow + +The simplest possible Orchestrator workflow. Uses AWS Fargate with default CPU and memory. + +```yaml +name: Build with Orchestrator + +on: + push: + branches: [main] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: eu-west-2 + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +## โ˜๏ธ AWS Fargate + +Full workflow with custom CPU/memory, S3 artifact export, and GitHub Checks. + +```yaml +name: Orchestrator โ€” AWS Fargate + +on: + push: + branches: [main, develop] + pull_request: + branches: [main] + +jobs: + build: + name: Build (${{ matrix.targetPlatform }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + targetPlatform: + - StandaloneLinux64 + - StandaloneWindows64 + - StandaloneOSX + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: eu-west-2 + + - uses: game-ci/unity-builder@v4 + id: build + with: + providerStrategy: aws + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + unityVersion: 2022.3.0f1 + containerCpu: 2048 + containerMemory: 8192 + # Export build artifacts to S3: + containerHookFiles: aws-s3-upload-build + # Show build progress as GitHub Checks: + githubCheck: true +``` + +### Required Secrets + +| Secret | Description | +| ----------------------- | ---------------------------------------------------------------- | +| `AWS_ACCESS_KEY_ID` | IAM access key with ECS, CloudFormation, S3, Kinesis, CloudWatch | +| `AWS_SECRET_ACCESS_KEY` | IAM secret key | + +See the [AWS provider page](../providers/aws) for allowed CPU/memory combinations and full setup. + +## โ˜ธ๏ธ Kubernetes + +Full workflow targeting a Kubernetes cluster. + +```yaml +name: Orchestrator โ€” Kubernetes + +on: + push: + branches: [main] + +jobs: + build: + name: Build (${{ matrix.targetPlatform }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + targetPlatform: + - StandaloneLinux64 + - StandaloneWindows64 + env: + kubeConfig: ${{ secrets.KUBE_CONFIG }} + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - uses: game-ci/unity-builder@v4 + id: build + with: + providerStrategy: k8s + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + unityVersion: 2022.3.0f1 + kubeVolumeSize: 10Gi + containerCpu: 1024 + containerMemory: 4096 + containerHookFiles: aws-s3-upload-build + githubCheck: true +``` + +### Required Secrets + +| Secret | Description | +| ------------- | ------------------------------- | +| `KUBE_CONFIG` | Base64-encoded kubeconfig file. | + +Generate it with: + +```bash +cat ~/.kube/config | base64 -w 0 +``` + +See the [Kubernetes provider page](../providers/kubernetes) for cluster tips and full setup. + +## ๐Ÿณ Local Docker (Self-Hosted Runner) + +Run builds in Docker on your own machine. No cloud account needed. + +```yaml +name: Orchestrator โ€” Local Docker + +on: + push: + branches: [main] + +jobs: + build: + runs-on: self-hosted + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: local-docker + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +Requires Docker installed on the self-hosted runner. + +## โณ Async Mode + +For long builds, use async mode so the GitHub Action returns immediately. Monitor progress via +GitHub Checks. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + asyncOrchestrator: true + githubCheck: true +``` + +The build runs in the background. Check progress from the **Checks** tab on your pull request. + +See [GitHub Integration](../providers/github-integration) for more on async mode and GitHub Checks. + +## ๐Ÿ—‘๏ธ Scheduled Garbage Collection + +Add a scheduled workflow to clean up stale cloud resources. Useful as a safety net alongside the +automatic cleanup cron. + +```yaml +name: Orchestrator โ€” Garbage Collect + +on: + schedule: + - cron: '0 4 * * *' # Daily at 4 AM UTC + +jobs: + cleanup: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: eu-west-2 + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + mode: garbage-collect + garbageMaxAge: 24 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +See [Garbage Collection](../advanced-topics/garbage-collection) for details. + +## ๐Ÿ“ฆ Multi-Platform Matrix Build + +Build for multiple platforms in parallel. Each platform runs as a separate Orchestrator job. + +```yaml +name: Orchestrator โ€” Multi-Platform + +on: + push: + branches: [main] + +jobs: + build: + name: Build ${{ matrix.targetPlatform }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + targetPlatform: + - StandaloneLinux64 + - StandaloneWindows64 + - StandaloneOSX + - iOS + - Android + - WebGL + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: eu-west-2 + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + containerCpu: 2048 + containerMemory: 8192 + containerHookFiles: aws-s3-upload-build + githubCheck: true +``` + +## ๐Ÿ” Retained Workspaces for Faster Rebuilds + +For large projects, keep the entire project folder cached between builds. Dramatically speeds up +rebuilds at the cost of more storage. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + maxRetainedWorkspaces: 3 + containerCpu: 2048 + containerMemory: 8192 +``` + +See [Retained Workspaces](../advanced-topics/retained-workspace) and +[Caching](../advanced-topics/caching) for details on storage strategies. + +## ๐Ÿช Container Hooks โ€” S3 Upload + Steam Deploy + +Chain multiple container hooks to export builds to S3 and deploy to Steam in a single workflow. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + containerHookFiles: aws-s3-upload-build,steam-deploy-client + env: + STEAM_USERNAME: ${{ secrets.STEAM_USERNAME }} + STEAM_PASSWORD: ${{ secrets.STEAM_PASSWORD }} + STEAM_APPID: ${{ secrets.STEAM_APPID }} +``` + +See [Built-In Hooks](../advanced-topics/hooks/built-in-hooks) for all available hooks (S3, rclone, +Steam). + +## ๐Ÿ”— Reference + +- [API Reference](../api-reference) โ€” full list of all parameters +- [Providers](../providers/overview) โ€” setup guides for each provider +- [Secrets](../secrets) โ€” how credentials are transferred to build containers +- [Real-world pipeline](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/orchestrator-integrity.yml) + โ€” Game CI's own Orchestrator test pipeline diff --git a/docs/03-github-orchestrator/03-examples/02-github-examples/02-aws.mdx b/docs/03-github-orchestrator/03-examples/02-github-examples/02-aws.mdx deleted file mode 100644 index 3c18f3e3..00000000 --- a/docs/03-github-orchestrator/03-examples/02-github-examples/02-aws.mdx +++ /dev/null @@ -1,71 +0,0 @@ -# AWS - -## Requirements - -- You must have an AWS account setup and ready to create resources. -- Create a service account and generate an AWS access key and key id. - -## AWS Credentials - -Setup the following as `env` variables for orchestrator to use: - -- `AWS_ACCESS_KEY_ID` -- `AWS_SECRET_ACCESS_KEY` -- `AWS_DEFAULT_REGION` (should be the same AWS region as the base stack e.g `eu-west-2`) - -If you're using GitHub you can use a GitHub Action: - -```yaml -- name: Configure AWS Credentials - uses: aws-actions/configure-aws-credentials@v4 - with: - aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} - aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - aws-region: eu-west-2 -``` - -_Note:_ _This enables Orchestrator access AWS._ - -## Configuration For AWS Orchestrator Jobs - -Refer to [Configuration page](../../api-reference) or the [example below](#example). - -### Allowed CPU/Memory Combinations - -There are some limitations to the CPU and Memory parameters. AWS will only accept the following -combinations: -[AWS Fargate Documentation, Allowed CPU and memory values (Task Definitions)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_size) - -#### Summary Of Format - -- Values are represented as 1024:1 GB or CPU. -- Do not include the vCPU or GB suffix. -- 1 CPU can go to a max of 6 GB of memory. 2 CPU's are required to go higher. - -#### Valid CPU and Memory Values - -```yaml -- orchestratorMemory: 4096 -- orchestratorCpu: 1024 -``` - -## Example - -```yaml -- uses: game-ci/unity-builder@orchestrator-develop - id: aws-fargate-unity-build - with: - providerStrategy: aws - versioning: None - projectPath: `your path here` - unityVersion: `unity version here` - targetPlatform: ${{ matrix.targetPlatform }} - gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} - # You may want to export your builds somewhere external so you can access it: - containerHookFiles: aws-s3-upload-build -``` - -_[Custom Steps](../../advanced-topics/custom-hooks/container-hooks)_ - -A full workflow example can be seen in builder's -[Orchestrator GitHub sourcecode for GitHub Pipeline](https://github.com/game-ci/unity-builder/blob/309d668d637ae3e7ffe90d61612968db92e1e376/.github/workflows/orchestrator-pipeline.yml#L109). diff --git a/docs/03-github-orchestrator/03-examples/02-github-examples/03-kubernetes.mdx b/docs/03-github-orchestrator/03-examples/02-github-examples/03-kubernetes.mdx deleted file mode 100644 index a71f41a8..00000000 --- a/docs/03-github-orchestrator/03-examples/02-github-examples/03-kubernetes.mdx +++ /dev/null @@ -1,51 +0,0 @@ -# Kubernetes - -## Requirements - -- You must have a Kubernetes cluster setup and ready that supports persistent volumes. -- Create a kubeconfig and encode it as base64. - -## K8s Credentials - -Setup the following as `env` variables for the GitHub build step: - -- `kubeConfig` (should be encoded as base64) - -## Configuration For Kubernetes Orchestrator Jobs - -Refer to [Configuration page](../../api-reference) or the [example below](#example). - -### Allowed CPU/Memory Combinations - -- `0.25 vCPU` - 0.5 GB, 1 GB, 2 GB -- `0.5 vCPU` - 1 GB, 2 GB, 3 GB, 4 GB -- `1 vCPU` - 2 GB, 3 GB, 4 GB, 5 GB, 6 GB, 7 GB, 8 GB -- `2 vCPU` - Between 4 GB and 16 GB in 1-GB increments -- `4 vCPU` - Between 8 GB and 30 GB in 1-GB increments - -#### Summary Of Format - -- Values are represented as 1024:1 GB or CPU. - -Do not include the vCPU or GB suffix. - -### Example - -```yaml -- uses: game-ci/unity-builder@orchestrator-develop - id: k8s-unity-build - with: - providerStrategy: k8s - versioning: None - projectPath: `your path here` - unityVersion: `unity version here` - targetPlatform: ${{ matrix.targetPlatform }} - gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} - # You may want to export your builds somewhere external so you can access it: - containerHookFiles: aws-s3-upload-build -``` - -_[Custom Steps](../../advanced-topics/custom-hooks/container-hooks)_ - -A full workflow example can be seen in builder's -[Orchestrator GitHub sourcecode for AWS Pipeline](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/orchestrator-k8s-pipeline.yml). diff --git a/docs/03-github-orchestrator/04-api-reference.mdx b/docs/03-github-orchestrator/04-api-reference.mdx deleted file mode 100644 index 6a46b5c3..00000000 --- a/docs/03-github-orchestrator/04-api-reference.mdx +++ /dev/null @@ -1,259 +0,0 @@ -# API Reference - -## Configuration - -_You can specify input parameters via any of the following methods._ - -- **GitHub Action `with`** _See "Getting Started" examples._ -- **Command Line** _You can specify input parameters via command line._ -- **Environment Variables** _You can specify input parameters via environment variables._ -- **Configuration Override** _[Advanced Topics / Overrides](advanced-topics/configuration-override)_ - -## Modes - -Orchestrator can accept a parameter to run a specific mode, by default cli-build is run. - -```bash -cli-build -``` - -_runs an orchestrator build_ - -```bash -list-resources -``` - -_lists active resources_ - -```bash -list-workflow -``` - -_lists running workflows_ - -```bash -watch -``` - -_follows logs of a running workflow_ - -```bash -garbage-collect -``` - -_runs garbage collection_ - -```bash -- cache-push -- cache-pull -``` - -Cache commands to push and pull from the local caching directory. Used in orchestrator workflows. -Uses `cachePullFrom` and `cachePushTo` parameters. - -```bash -- hash (hash folder contents recursively) -- print-input (prints all input parameters) -``` - -Utility commands - -```bash -- remote-cli-pre-build (sets up a repository, usually before a game-ci build) -- remote-cli-post-build (pushes to LFS and Library cache) -``` - -Commands called during orchestrator workflows before/after a build. - -## Common Parameters - -### Git synchronization parameters - -```bash -gitPrivateToken (should be a GitHub access token with permission to get repositories) -``` - -Used to authenticate remote job's access to repository. Also used for LFS file pulling if -`GIT_PRIVATE_TOKEN` is not set separately. - -```bash -- GITHUB_REPOSITORY -- GITHUB_REF || branch || GitSHA -``` - -Used to synchronize the repository to the Orchestrator job. If parameters are not provided, will -attempt to read them from current directory's git repo (e.g branch, commit SHA, remote URL). - -### Orchestrator parameters - -```bash -providerStrategy -``` - -Specifies the Cloud Provider to use for Orchestrator jobs. Accepted values: `aws`, `k8s`, -`local-docker`, `local`. - -```bash -- containerCpu -- containerMemory -``` - -Specifies the CPU and Memory resources to be used for cloud containers created by Orchestrator. -(See: getting started section for more configuration options per provider.) - -```bash -orchestratorBranch -``` - -Specifies the release branch of Orchestrator to use for remote containers. Accepted values: `main` -(default), `orchestrator-develop` (latest/development). - -```bash -cloneDepth -``` - -Specifies the depth of the git clone for the repository. Defaults to `50`. Use `0` for a full clone. - -### Custom commands from files parameters - -```bash -- containerHookFiles -- commandHookFiles -- commandHooks -- postBuildContainerHooks -- preBuildContainerHooks -``` - -Specifies the name of custom hook or step files to include in workflow. (Accepted Format: see -"[container hooks](advanced-topics/custom-hooks/container-hooks) -[command hooks](advanced-topics/custom-hooks/command-hooks)") - -### Custom commands from yaml parameters - -```bash -customJob -``` - -Specifies a custom job to override default build workflow. (Accepted Format: see -"[advanced topics / custom job](advanced-topics/custom-hooks/custom-job)") - -### Configuration Override - -```bash -readInputOverrideCommand -``` - -Read parameter from command line output, such as a secret manager. Must include a `{0}` to inject -the name of the parameter to pull. Built-in presets: `gcp-secret-manager`, `aws-secret-manager`. -(See: [Configuration Override](advanced-topics/configuration-override)) - -```bash -readInputFromOverrideList -``` - -Comma separated list of parameters to apply with `readInputOverrideCommand`. (See: -[Configuration Override](advanced-topics/configuration-override)) - -### Storage - -```bash -storageProvider -``` - -Specifies the storage backend for caching and artifacts. Accepted values: `s3` (default), `rclone`. - -```bash -rcloneRemote -``` - -Configures the rclone remote storage endpoint. Required when using `storageProvider: rclone`. - -### AWS - -```bash -awsStackName -``` - -Name of the persistent shared base stack, used to store artifacts and caching. Defaults to -`game-ci`. - -```bash -- awsEndpoint (base endpoint override for all AWS services) -- awsCloudFormationEndpoint -- awsEcsEndpoint -- awsKinesisEndpoint -- awsCloudWatchLogsEndpoint -- awsS3Endpoint -``` - -Optional AWS service endpoint overrides. Useful for testing with LocalStack or other AWS-compatible -services. - -### K8s - -```bash -- kubeConfig (base64 encoded kubernetes config) -- kubeVolume -- kubeVolumeSize (default: 5Gi) -- kubeStorageClass -``` - -Override name of persistent volume used, size of volume and storage class used. - -### Caching - -```bash -cacheKey -``` - -Defaults to branch name. Defines the scope for sharing cache entries. - -### Utility - -```bash -- orchestratorDebug (Debug logging for Orchestrator) -- resourceTracking (Enable resource tracking logs for disk usage summaries) -- useLargePackages (Any packages in manifest.json containing phrase "LargePackage" will be - redirected to a shared folder for all builds sharing a cache key) -- useSharedBuilder (Use a shared clone of Game-CI, saves some storage space and can be used if - you're using one release branch of Orchestrator) -- useCompressionStrategy (Use Lz4 compression for cache and build artifacts. Enabled by default) -- watchToEnd (Whether to watch the build to the end, default: true) -- asyncOrchestrator (Run in async mode, returns immediately without waiting for build completion) -``` - -### Retained Workspace - -```bash -- maxRetainedWorkspaces -``` - -See: [Advanced Topics / Retained Workspaces](advanced-topics/retained-workspace), enables caching -entire project folder. - -### Garbage Collection - -```bash -- garbageMaxAge (Maximum age in hours before resources are cleaned up, default: 24) -``` - -## Command Line Only Parameters - -```bash -- populateOverride -- cachePushFrom -- cachePushTo -- artifactName -- select -``` - -## Other Environment Variables - -```bash -- USE_IL2CPP (Set to `false`) -``` - -# External Links - -All accepted parameters given here with a description: -[https://github.com/game-ci/unity-builder/blob/main/action.yml](https://github.com/game-ci/unity-builder/blob/main/action.yml) diff --git a/docs/03-github-orchestrator/04-jobs.mdx b/docs/03-github-orchestrator/04-jobs.mdx new file mode 100644 index 00000000..5318cbd0 --- /dev/null +++ b/docs/03-github-orchestrator/04-jobs.mdx @@ -0,0 +1,189 @@ +# Orchestrator Jobs + +Orchestrator executes work as **jobs** โ€” containerized or local tasks that run on your chosen +provider. Understanding job types and their flow is key to customizing your build pipeline. + +## Job Flow + +Every Orchestrator run follows the same lifecycle, regardless of provider: + +``` + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Setup โ”‚โ”€โ”€โ”€โ”€โ–บโ”‚ Pre-Build โ”‚โ”€โ”€โ”€โ”€โ–บโ”‚ Build โ”‚โ”€โ”€โ”€โ”€โ–บโ”‚ Post-Build โ”‚ + โ”‚ Provider โ”‚ โ”‚ Jobs โ”‚ โ”‚ Job โ”‚ โ”‚ Jobs โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ Cleanup โ”‚โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +1. **Setup** โ€” Provision cloud resources (stacks, volumes, secrets). Skipped for local providers. +2. **Pre-build jobs** โ€” Clone the repository, pull LFS, restore caches, run pre-build hooks. +3. **Build job** โ€” Execute the Unity build (or custom editor method). +4. **Post-build jobs** โ€” Push caches, upload artifacts, run post-build hooks. +5. **Cleanup** โ€” Release locks, tear down cloud resources, update GitHub Checks. + +## Job Types + +### Build Job + +The standard job โ€” runs the Unity Editor to produce a build artifact. This is what most users +care about. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + targetPlatform: StandaloneLinux64 + providerStrategy: aws + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +The build job: +- Installs the toolchain (Node.js, git-lfs) inside the container +- Clones `game-ci/unity-builder` into the container +- Runs `remote-cli-pre-build` to set up the workspace +- Executes the Unity build via the Game CI entrypoint +- Runs `remote-cli-post-build` to push caches and artifacts + +### Test Jobs + +Run Unity tests without producing a build. Use a custom `buildMethod` that runs tests and exits: + +```yaml +- uses: game-ci/unity-builder@v4 + with: + targetPlatform: StandaloneLinux64 + buildMethod: MyNamespace.TestRunner.RunEditModeTests + manualExit: true + providerStrategy: aws + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +With `manualExit: true`, Unity doesn't quit automatically โ€” your build method should call +`EditorApplication.Exit(0)` after tests complete. This gives you full control over the test +lifecycle. + +### Custom Editor Method Jobs + +Run any static C# method in the Unity Editor. Useful for: +- Asset processing or validation +- Addressables builds +- Custom pipeline steps +- Code generation + +```yaml +- uses: game-ci/unity-builder@v4 + with: + targetPlatform: StandaloneLinux64 + buildMethod: MyNamespace.Pipeline.ProcessAssets + manualExit: true + providerStrategy: aws + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +The `buildMethod` must be a fully qualified static method: `Namespace.Class.Method`. + +### Custom Jobs + +Replace the entire build workflow with your own container steps. Useful for non-Unity workloads +or fully custom pipelines that still benefit from Orchestrator's cloud infrastructure. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + customJob: | + - name: my-custom-step + image: ubuntu:22.04 + commands: | + echo "Running custom workload" + ./my-script.sh + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +See [Custom Job](advanced-topics/custom-job) for the full reference. + +### Async Jobs + +For long-running builds, Orchestrator can dispatch the job and return immediately. The build +continues in the cloud. Progress is reported via GitHub Checks. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + targetPlatform: StandaloneLinux64 + providerStrategy: aws + asyncOrchestrator: true + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +This is useful when builds exceed GitHub Actions' job time limits, or when you want to +free up your CI runner immediately. + +## Pre-Build and Post-Build + +Orchestrator runs additional steps before and after the main build job. + +### Pre-Build Steps + +The `remote-cli-pre-build` phase handles: +- Git clone of the target repository +- Git LFS pull +- Cache restoration (Library folder, LFS objects) +- Retained workspace setup +- Submodule initialization (if using [submodule profiles](advanced-topics/submodule-profiles)) +- Custom LFS agent configuration (e.g., [elastic-git-storage](advanced-topics/lfs-agents)) + +You can inject additional pre-build steps: + +```yaml +- uses: game-ci/unity-builder@v4 + with: + preBuildSteps: | + - name: install-dependencies + image: node:18 + commands: npm install +``` + +### Post-Build Steps + +The `remote-cli-post-build` phase handles: +- Library folder cache push +- Build artifact upload +- LFS cache push + +You can inject additional post-build steps: + +```yaml +- uses: game-ci/unity-builder@v4 + with: + postBuildSteps: | + - name: upload-to-steam + image: steamcmd + commands: ./upload.sh +``` + +### Hooks + +For more granular control, use [container hooks](advanced-topics/hooks/container-hooks) or +[command hooks](advanced-topics/hooks/command-hooks) to inject steps at specific points in the +build lifecycle. + +## Job Execution by Provider + +Each provider runs jobs differently: + +| Provider | How jobs execute | +|----------|----------------| +| **AWS Fargate** | ECS Fargate task with CloudFormation stack. Logs streamed via Kinesis. | +| **Kubernetes** | Kubernetes Job with PVC for workspace. Logs streamed from pod. | +| **Local Docker** | Docker container with volume mounts. Logs piped to stdout. | +| **Local** | Direct shell execution on the host. No container isolation. | +| **CLI Provider** | Delegated to external executable via JSON protocol. | + +## Next Steps + +- [Custom Job](advanced-topics/custom-job) โ€” Full reference for custom job definitions +- [Hooks](advanced-topics/hooks/container-hooks) โ€” Inject steps at specific lifecycle points +- [Architecture](advanced-topics/architecture) โ€” Deep dive into internal components diff --git a/docs/03-github-orchestrator/05-api-reference.mdx b/docs/03-github-orchestrator/05-api-reference.mdx new file mode 100644 index 00000000..4048dd84 --- /dev/null +++ b/docs/03-github-orchestrator/05-api-reference.mdx @@ -0,0 +1,158 @@ +# API Reference + +## โš™๏ธ Configuration Methods + +| Method | Description | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| **GitHub Action `with`** | Pass parameters directly in your workflow file. See [Getting Started](getting-started). | +| **Command Line** | Pass parameters as CLI flags. See [Command Line](examples/command-line). | +| **Environment Variables** | Set parameters as environment variables in your shell or CI environment. | +| **Pull Secrets** | Pull parameters dynamically from secret managers or files. See [Secrets](secrets#-pulling-secrets-from-external-sources). | + +## ๐Ÿ”ง Modes + +Set the mode to control what Orchestrator does. Default: `cli-build`. + +| Mode | Description | +| ----------------------- | ------------------------------------------------------------------------------------- | +| `cli-build` | Run a standard build workflow. | +| `list-resources` | List active cloud resources. | +| `list-workflow` | List running workflows. | +| `watch` | Follow logs of a running workflow. | +| `garbage-collect` | Clean up old resources. See [Garbage Collection](advanced-topics/garbage-collection). | +| `cache-push` | Push to the caching directory. Uses `cachePushTo`. | +| `cache-pull` | Pull from the caching directory. Uses `cachePullFrom`. | +| `hash` | Hash folder contents recursively. | +| `print-input` | Print all resolved input parameters. | +| `remote-cli-pre-build` | Set up a repository before a build (used internally by workflows). | +| `remote-cli-post-build` | Push LFS files and Library cache after a build (used internally). | + +## ๐Ÿ“‹ Parameters + +### Provider + +| Parameter | Default | Description | +| ---------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `providerStrategy` | `local` | Cloud provider to use. Built-in: `aws`, `k8s`, `local-docker`, `local`. Also accepts a GitHub URL, NPM package, or local path for [custom providers](providers/custom-providers). | +| `containerCpu` | `1024` | CPU units for cloud containers (`1024` = 1 vCPU). See provider setup guides for allowed values. | +| `containerMemory` | `3072` | Memory in MB for cloud containers (`4096` = 4 GB). See provider setup guides for allowed values. | +| `orchestratorBranch` | `main` | Release branch of Orchestrator for remote containers. Use `orchestrator-develop` for latest development builds. | +| `orchestratorRepoName` | `game-ci/unity-builder` | Repository for Orchestrator source. Override to use a fork for testing or custom builds. | + +### Git Synchronization + +| Parameter | Default | Description | +| ------------------- | -------- | ------------------------------------------------------------------- | +| `gitPrivateToken` | โ€” | GitHub access token with repo access. Used for git clone and LFS. | +| `githubOwner` | โ€” | GitHub owner or organization name. | +| `GITHUB_REPOSITORY` | _(auto)_ | Repository in `owner/repo` format. Auto-detected in GitHub Actions. | +| `GITHUB_REF` | _(auto)_ | Git ref to build. Falls back to `branch` or `GitSHA` parameters. | +| `cloneDepth` | `50` | Depth of the git clone. Use `0` for a full clone. | +| `allowDirtyBuild` | `false` | Allow building from a branch with uncommitted changes. | + +### Custom Hooks + +| Parameter | Default | Description | +| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------- | +| `containerHookFiles` | โ€” | Names of [container hook](advanced-topics/hooks/container-hooks) files from `.game-ci/container-hooks/`. | +| `customHookFiles` | โ€” | Names of custom hook files from `.game-ci/hooks/`. | +| `customCommandHooks` | โ€” | Inline [command hooks](advanced-topics/hooks/command-hooks) as YAML. | +| `postBuildSteps` | โ€” | Post-build job in YAML format with keys: `image`, `secrets`, `command`. | +| `preBuildSteps` | โ€” | Pre-build job (after repo setup, before build) in YAML format. | +| `postBuildContainerHooks` | โ€” | Container hook files to run after the build step. | +| `preBuildContainerHooks` | โ€” | Container hook files to run before the build step. | +| `customJob` | โ€” | Custom job YAML to override the default build workflow. See [Custom Job](advanced-topics/custom-job). | + +### Pull Secrets + +| Parameter | Default | Description | +| --------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `readInputOverrideCommand` | โ€” | Command to read a parameter value from an external source. Use `{0}` as the parameter name placeholder. Built-in presets: `gcp-secret-manager`, `aws-secret-manager`. See [Secrets](secrets). | +| `readInputFromOverrideList` | โ€” | Comma-separated list of parameter names to pull via `readInputOverrideCommand`. | + +### Storage + +| Parameter | Default | Description | +| ----------------- | ------- | ------------------------------------------------------------------------------------------------------ | +| `storageProvider` | `s3` | Storage backend for [caching](advanced-topics/caching) and artifacts. Accepted values: `s3`, `rclone`. | +| `rcloneRemote` | โ€” | Rclone remote endpoint. Required when `storageProvider` is `rclone`. | + +### AWS + +| Parameter | Default | Description | +| --------------------------- | --------- | -------------------------------------------------------------- | +| `awsStackName` | `game-ci` | Name of the persistent shared CloudFormation base stack. | +| `awsEndpoint` | โ€” | Base endpoint override for all AWS services (e.g. LocalStack). | +| `awsCloudFormationEndpoint` | โ€” | CloudFormation service endpoint override. | +| `awsEcsEndpoint` | โ€” | ECS service endpoint override. | +| `awsKinesisEndpoint` | โ€” | Kinesis service endpoint override. | +| `awsCloudWatchLogsEndpoint` | โ€” | CloudWatch Logs service endpoint override. | +| `awsS3Endpoint` | โ€” | S3 service endpoint override. | + +### Kubernetes + +| Parameter | Default | Description | +| ------------------ | ------- | ----------------------------------------------------------------------- | +| `kubeConfig` | โ€” | Base64-encoded Kubernetes config file. | +| `kubeVolume` | โ€” | Name of the persistent volume claim to use. | +| `kubeVolumeSize` | `5Gi` | Size of the persistent volume. | +| `kubeStorageClass` | โ€” | Storage class for the persistent volume. Empty = auto-install via rook. | + +### Caching + +| Parameter | Default | Description | +| ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| `cacheKey` | _(branch name)_ | Scope for sharing cache entries. Builds with the same key share a cache. | +| `maxRetainedWorkspaces` | `0` | Maximum number of [retained workspaces](advanced-topics/retained-workspace). `0` = unlimited. Above the limit, jobs use standard caching. | + +### GitHub Integration + +| Parameter | Default | Description | +| ------------------- | ------- | --------------------------------------------------------------------------------------------------------- | +| `githubCheck` | `false` | Create a GitHub Check for each orchestrator step. See [GitHub Integration](providers/github-integration). | +| `asyncOrchestrator` | `false` | Run in async mode โ€” returns immediately without waiting for the build to complete. | +| `watchToEnd` | `true` | Whether to follow the build logs until completion. | + +### Build Options + +| Parameter | Default | Description | +| ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------ | +| `orchestratorDebug` | `false` | Enable verbose debug logging (resource tracking, directory tree, environment listing). | +| `resourceTracking` | `false` | Enable resource tracking logs with disk usage summaries. | +| `useLargePackages` | `false` | Redirect packages containing "LargePackage" in `manifest.json` to a shared folder across builds with the same cache key. | +| `useSharedBuilder` | `false` | Use a shared clone of Game CI. Saves storage when using a single Orchestrator release branch. | +| `useCompressionStrategy` | `true` | Use LZ4 compression for cache and build artifacts. | +| `useCleanupCron` | `true` | Create an AWS CloudFormation cron job to automatically clean up old resources. | + +### Garbage Collection + +| Parameter | Default | Description | +| --------------- | ------- | ----------------------------------------------------- | +| `garbageMaxAge` | `24` | Maximum age in hours before resources are cleaned up. | + +## ๐Ÿ–ฅ๏ธ CLI-Only Parameters + +These parameters are only available when using Orchestrator from the command line. + +| Parameter | Description | +| ------------------ | -------------------------------------------------------------------------------------------------- | +| `populateOverride` | Enable [pulling secrets](secrets#-pulling-secrets-from-external-sources) from an external command. | +| `cachePushFrom` | Local directory to push cache from. | +| `cachePushTo` | Remote path to push cache to. | +| `artifactName` | Name for the build artifact. | +| `select` | Select a specific workflow or resource by name. | + +## ๐ŸŒ Environment Variables + +| Variable | Description | +| ---------------------------------- | --------------------------------------------------------------------------------- | +| `USE_IL2CPP` | Set to `false` to disable IL2CPP builds. | +| `AWS_FORCE_PROVIDER` | Force provider when LocalStack is detected. Values: `aws`, `aws-local`, or empty. | +| `ORCHESTRATOR_AWS_STACK_WAIT_TIME` | CloudFormation stack timeout in seconds. Default: `600`. | +| `PURGE_REMOTE_BUILDER_CACHE` | Set to clear the remote builder cache before a build. | +| `GIT_PRIVATE_TOKEN` | Separate token for LFS pulls (falls back to `gitPrivateToken`). | + +## ๐Ÿ”— External Links + +All parameters with descriptions: +[game-ci/unity-builder action.yml](https://github.com/game-ci/unity-builder/blob/main/action.yml) diff --git a/docs/03-github-orchestrator/05-providers/01-overview.mdx b/docs/03-github-orchestrator/05-providers/01-overview.mdx new file mode 100644 index 00000000..be952d2d --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/01-overview.mdx @@ -0,0 +1,48 @@ +# Providers + +A **provider** is the backend that Orchestrator uses to run your builds. You choose a provider by +setting the `providerStrategy` parameter. + +``` + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ aws โ”‚ โ”‚ k8s โ”‚ โ”‚ local-docker โ”‚ โ”‚ local โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ Fargate โ”‚ โ”‚ Cluster โ”‚ โ”‚ Container โ”‚ โ”‚ Direct โ”‚ + โ”‚ Fully โ”‚ โ”‚ Bring your โ”‚ โ”‚ No cloud โ”‚ โ”‚ No container โ”‚ + โ”‚ managed โ”‚ โ”‚ own cluster โ”‚ โ”‚ needed โ”‚ โ”‚ needed โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + Cloud scaling Cloud scaling Local builds Local builds + No servers Flexible Docker required Simplest setup +``` + +## Built-in Providers + +These providers ship with Orchestrator and are maintained by the Game CI team. + +| Provider | `providerStrategy` | Description | +| -------------- | ------------------ | ----------------------------------------------------------------------------- | +| AWS Fargate | `aws` | Runs jobs on AWS Fargate (ECS). Fully managed, no servers to maintain. | +| Kubernetes | `k8s` | Runs jobs on any Kubernetes cluster. Flexible but requires a running cluster. | +| Local Docker | `local-docker` | Runs jobs in Docker containers on the local machine. | +| Local (direct) | `local` | Runs jobs directly on the local machine without containers. | + +Each provider has its own page with setup instructions: + +- [AWS Fargate](aws) +- [Kubernetes](kubernetes) +- [Local Docker](local-docker) +- [Local](local) + +## Custom Providers + +Extend Orchestrator with your own provider by pointing `providerStrategy` at a GitHub repository, +NPM package, or local file path. + +See [Custom Providers](custom-providers) for the full guide. + +## Community Providers + +Third-party providers shared by the Game CI community. + +See the [Community Providers](community-providers) page for the current list and how to submit your +own. diff --git a/docs/03-github-orchestrator/05-providers/02-aws.mdx b/docs/03-github-orchestrator/05-providers/02-aws.mdx new file mode 100644 index 00000000..3beabbe1 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/02-aws.mdx @@ -0,0 +1,96 @@ +# AWS + +## Architecture + +Orchestrator creates and manages these AWS resources automatically: + +``` + CloudFormation (base stack) + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ ECS โ”‚ โ”‚ S3 โ”‚ โ”‚ + โ”‚ โ”‚ Fargate โ”‚ โ”‚ Bucket โ”‚ โ”‚ + โ”‚ โ”‚ (build โ”‚ โ”‚ (cache + โ”‚ โ”‚ + โ”‚ โ”‚ tasks) โ”‚ โ”‚ artifacts) โ”‚ โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ + โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ”‚ โ”‚ CloudWatch โ”‚โ”€โ”€โ”€โ–บโ”‚ Kinesis โ”‚โ”€โ”€โ–บ Log stream โ”‚ + โ”‚ โ”‚ Logs โ”‚ โ”‚ Stream โ”‚ to CI โ”‚ + โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + โ”‚ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Requirements + +- An AWS account with permission to create resources (ECS, CloudFormation, S3, Kinesis, CloudWatch). +- An IAM user or role with an access key and secret key. + +## AWS Credentials + +Set the following as `env` variables in your workflow: + +| Variable | Description | +| ----------------------- | ------------------------------------------------------- | +| `AWS_ACCESS_KEY_ID` | IAM access key ID. | +| `AWS_SECRET_ACCESS_KEY` | IAM secret access key. | +| `AWS_DEFAULT_REGION` | AWS region matching your base stack (e.g. `eu-west-2`). | + +If you're using GitHub Actions, configure credentials with: + +```yaml +- name: Configure AWS Credentials + uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: eu-west-2 +``` + +## CPU and Memory + +AWS Fargate only accepts specific CPU/memory combinations. Values use the format `1024 = 1 vCPU` or +`1 GB`. Do not include the vCPU or GB suffix. + +See the full list: +[AWS Fargate Task Definitions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_size) + +Common combinations: + +| CPU (`containerCpu`) | Memory (`containerMemory`) | +| -------------------- | -------------------------- | +| `256` (0.25 vCPU) | `512`, `1024`, `2048` | +| `512` (0.5 vCPU) | `1024` โ€“ `4096` | +| `1024` (1 vCPU) | `2048` โ€“ `8192` | +| `2048` (2 vCPU) | `4096` โ€“ `16384` | +| `4096` (4 vCPU) | `8192` โ€“ `30720` | + +## Example Workflow + +```yaml +- uses: game-ci/unity-builder@v4 + id: aws-fargate-unity-build + with: + providerStrategy: aws + versioning: None + projectPath: path/to/your/project + unityVersion: 2022.3.0f1 + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + containerCpu: 1024 + containerMemory: 4096 + # Export builds to S3: + containerHookFiles: aws-s3-upload-build +``` + +See [Container Hooks](../advanced-topics/hooks/container-hooks) for more on `containerHookFiles`. + +A full workflow example is available in the builder source: +[orchestrator-pipeline.yml](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/orchestrator-pipeline.yml). + +## AWS Parameters + +For the full list of AWS-specific parameters (`awsStackName`, endpoint overrides, etc.), see the +[API Reference โ€” AWS section](../api-reference#aws). diff --git a/docs/03-github-orchestrator/05-providers/03-kubernetes.mdx b/docs/03-github-orchestrator/05-providers/03-kubernetes.mdx new file mode 100644 index 00000000..5453a8e2 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/03-kubernetes.mdx @@ -0,0 +1,64 @@ +# Kubernetes + +## Requirements + +- A running Kubernetes cluster that supports persistent volumes. +- A kubeconfig file encoded as base64. + +## K8s Credentials + +Pass the base64-encoded kubeconfig via the `kubeConfig` parameter or as an environment variable: + +```yaml +env: + kubeConfig: ${{ secrets.KUBE_CONFIG }} +``` + +## CPU and Memory + +Kubernetes accepts the same unit format as AWS โ€” `1024 = 1 vCPU`, memory in MB. Do not include the +vCPU or GB suffix. + +| CPU (`containerCpu`) | Memory (`containerMemory`) | +| -------------------- | -------------------------- | +| `256` (0.25 vCPU) | `512`, `1024`, `2048` | +| `512` (0.5 vCPU) | `1024` โ€“ `4096` | +| `1024` (1 vCPU) | `2048` โ€“ `8192` | +| `2048` (2 vCPU) | `4096` โ€“ `16384` | +| `4096` (4 vCPU) | `8192` โ€“ `30720` | + +## Example Workflow + +```yaml +- uses: game-ci/unity-builder@v4 + id: k8s-unity-build + with: + providerStrategy: k8s + versioning: None + projectPath: path/to/your/project + unityVersion: 2022.3.0f1 + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + kubeVolumeSize: 10Gi + containerCpu: 1024 + containerMemory: 4096 + # Export builds to S3: + containerHookFiles: aws-s3-upload-build +``` + +See [Container Hooks](../advanced-topics/hooks/container-hooks) for more on `containerHookFiles`. + +A full workflow example is available in the builder source: +[orchestrator-k8s-pipeline.yml](https://github.com/game-ci/unity-builder/blob/main/.github/workflows/orchestrator-k8s-pipeline.yml). + +### Cluster Tips + +- **Keep the cluster running.** Cold-starting a Kubernetes cluster is slow. If you need auto-scaling + to zero, consider Google Cloud Kubernetes Autopilot. +- **Cloud storage required.** Kubernetes requires cloud storage for + [caching](../advanced-topics/caching). S3 is built-in, or use rclone for other backends. + +## K8s Parameters + +For the full list of Kubernetes parameters (`kubeConfig`, `kubeVolume`, `kubeStorageClass`, etc.), +see the [API Reference โ€” Kubernetes section](../api-reference#kubernetes). diff --git a/docs/03-github-orchestrator/05-providers/04-local-docker.mdx b/docs/03-github-orchestrator/05-providers/04-local-docker.mdx new file mode 100644 index 00000000..0d20acf9 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/04-local-docker.mdx @@ -0,0 +1,34 @@ +# Local Docker + +Runs the build workflow inside a Docker container on the local machine. No cloud account required. + +## Requirements + +- Docker installed and running on the build machine. + +## Example Workflow + +### GitHub Actions (self-hosted runner) + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: local-docker + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +### Command Line + +```bash +yarn run cli -m cli-build \ + --providerStrategy local-docker \ + --projectPath /path/to/your/project \ + --targetPlatform StandaloneLinux64 +``` + +## When to Use + +- You have a self-hosted runner with Docker installed +- You want container isolation without cloud infrastructure +- Testing builds locally before deploying to AWS or Kubernetes diff --git a/docs/03-github-orchestrator/05-providers/05-local.mdx b/docs/03-github-orchestrator/05-providers/05-local.mdx new file mode 100644 index 00000000..14287da2 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/05-local.mdx @@ -0,0 +1,43 @@ +# Local + +Runs builds directly on the host machine with no container isolation. The simplest provider โ€” useful +for development and testing. + +## Requirements + +- Unity installed on the build machine. +- No Docker or cloud account needed. + +## Example Workflow + +### GitHub Actions (self-hosted runner) + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: local + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +### Command Line + +```bash +yarn run cli -m cli-build \ + --providerStrategy local \ + --projectPath /path/to/your/project \ + --targetPlatform StandaloneLinux64 +``` + +## When to Use + +- Local development and testing of Orchestrator workflows +- Debugging build issues before deploying to cloud providers +- Self-hosted runners where you want direct execution without containers + +## โš ๏ธ Notes + +- Builds run directly on the host with no isolation. Ensure the machine has the required Unity + version and dependencies installed. +- This is the fallback provider โ€” if a custom provider fails to load, Orchestrator falls back to + `local`. diff --git a/docs/03-github-orchestrator/05-providers/06-custom-providers.mdx b/docs/03-github-orchestrator/05-providers/06-custom-providers.mdx new file mode 100644 index 00000000..93688b1b --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/06-custom-providers.mdx @@ -0,0 +1,195 @@ +# Custom Providers + +Orchestrator uses a plugin system that lets you extend it with custom providers. A **provider** is a +pluggable backend that controls where and how your builds run. Built-in providers include `aws`, +`k8s`, `local-docker`, and `local`. + +With custom providers, you can point `providerStrategy` at a GitHub repository, NPM package, or +local path and Orchestrator will dynamically load it at runtime. + +``` + providerStrategy Build + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” fetch โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ "user/repo" โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ Clone repo / โ”‚ โ”‚ โ”‚ + โ”‚ "npm-package" โ”‚ โ”‚ Install pkg / โ”‚โ”€โ”€โ–บโ”‚ Provider โ”‚ + โ”‚ "./local/path" โ”‚ โ”‚ Resolve path โ”‚ โ”‚ Interface โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ + cached in โ”‚ setupWorkflowโ”‚ + .provider-cache/ โ”‚ runTask โ”‚ + โ”‚ cleanup โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Using a Custom Provider + +Set `providerStrategy` to a provider source instead of a built-in name: + +```yaml +# GitHub repository +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: 'https://github.com/your-org/my-provider' + targetPlatform: StandaloneLinux64 + +# GitHub shorthand +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: 'your-org/my-provider' + targetPlatform: StandaloneLinux64 + +# Specific branch +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: 'your-org/my-provider@develop' + targetPlatform: StandaloneLinux64 +``` + +### Supported Source Formats + +| Format | Example | +| ------------------------------------- | -------------------------------------------------------- | +| GitHub HTTPS URL | `https://github.com/user/repo` | +| GitHub URL with branch | `https://github.com/user/repo/tree/main` | +| GitHub URL with branch and path | `https://github.com/user/repo/tree/main/src/my-provider` | +| GitHub shorthand | `user/repo` | +| GitHub shorthand with branch | `user/repo@develop` | +| GitHub shorthand with branch and path | `user/repo@develop/src/my-provider` | +| GitHub SSH | `git@github.com:user/repo.git` | +| NPM package | `my-provider-package` | +| Scoped NPM package | `@scope/my-provider` | +| Local relative path | `./my-local-provider` | +| Local absolute path | `/path/to/provider` | + +## Creating a Custom Provider + +A provider is a module that exports a class implementing the `ProviderInterface`. The module must +have an entry point at one of: `index.js`, `index.ts`, `src/index.js`, `src/index.ts`, +`lib/index.js`, `lib/index.ts`, or `dist/index.js`. + +### Required Methods + +Every provider must implement these 7 methods: + +```typescript +interface ProviderInterface { + setupWorkflow( + buildGuid: string, + buildParameters: BuildParameters, + branchName: string, + defaultSecretsArray: { + ParameterKey: string; + EnvironmentVariable: string; + ParameterValue: string; + }[], + ): Promise; + + runTaskInWorkflow( + buildGuid: string, + image: string, + commands: string, + mountdir: string, + workingdir: string, + environment: OrchestratorEnvironmentVariable[], + secrets: OrchestratorSecret[], + ): Promise; + + cleanupWorkflow( + buildParameters: BuildParameters, + branchName: string, + defaultSecretsArray: { + ParameterKey: string; + EnvironmentVariable: string; + ParameterValue: string; + }[], + ): Promise; + + garbageCollect( + filter: string, + previewOnly: boolean, + olderThan: Number, + fullCache: boolean, + baseDependencies: boolean, + ): Promise; + + listResources(): Promise; + listWorkflow(): Promise; + watchWorkflow(): Promise; +} +``` + +### Example Implementation + +```typescript +// index.ts +export default class MyProvider { + constructor(private buildParameters: any) {} + + async setupWorkflow(buildGuid, buildParameters, branchName, defaultSecretsArray) { + // Initialize your build environment + } + + async runTaskInWorkflow(buildGuid, image, commands, mountdir, workingdir, environment, secrets) { + // Execute the build task in your environment + return 'Build output'; + } + + async cleanupWorkflow(buildParameters, branchName, defaultSecretsArray) { + // Tear down resources after the build + } + + async garbageCollect(filter, previewOnly, olderThan, fullCache, baseDependencies) { + // Clean up old resources + return 'Garbage collection complete'; + } + + async listResources() { + // Return active resources + return []; + } + + async listWorkflow() { + // Return running workflows + return []; + } + + async watchWorkflow() { + // Stream logs from a running workflow + return ''; + } +} +``` + +## How It Works + +When `providerStrategy` is set to a value that doesn't match a built-in provider name, Orchestrator +will: + +1. **Detect the source type** โ€” GitHub URL, NPM package, or local path. +2. **Fetch the provider** โ€” For GitHub repos, the repository is cloned (shallow, depth 1) into a + `.provider-cache/` directory. Cached repos are automatically updated on subsequent runs. +3. **Load the module** โ€” The entry point is imported and the default export is used. +4. **Validate the interface** โ€” All 7 required methods are checked. If any are missing, loading + fails. +5. **Fallback** โ€” If loading fails for any reason, Orchestrator logs the error and falls back to the + local provider so your pipeline doesn't break. + +## Caching + +GitHub repositories are cached in the `.provider-cache/` directory, keyed by owner, repo, and +branch. On subsequent runs the loader checks for updates and pulls them automatically. + +### Environment Variables + +| Variable | Default | Description | +| -------------------- | ----------------- | --------------------------------------- | +| `PROVIDER_CACHE_DIR` | `.provider-cache` | Custom cache directory for cloned repos | +| `GIT_TIMEOUT` | `30000` | Git operation timeout in milliseconds | + +## Best Practices + +- **Pin a branch or tag** โ€” Use `user/repo@v1.0` or a specific branch to avoid unexpected changes. +- **Test locally first** โ€” Use a local path during development before publishing. +- **Handle errors gracefully** โ€” Your provider methods should throw clear errors so Orchestrator can + log them and fall back if needed. +- **Keep it lightweight** โ€” The provider module is loaded at runtime. Minimize dependencies to keep + startup fast. diff --git a/docs/03-github-orchestrator/05-providers/07-community-providers.mdx b/docs/03-github-orchestrator/05-providers/07-community-providers.mdx new file mode 100644 index 00000000..7e5b7ed4 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/07-community-providers.mdx @@ -0,0 +1,47 @@ +# Community Providers + +Community providers are third-party Orchestrator providers built and shared by the community. They +are **not maintained by the Game CI team** but are listed here to help you discover and evaluate +options. + +:::caution + +Community providers are provided as-is. Review the source code and documentation of any community +provider before using it in your pipelines. + +::: + +## Provider List + +_No community providers have been submitted yet. Yours could be the first!_ + +## Submit Your Provider + +Built a custom provider? Share it with the community by adding it to this page. + +1. [Edit this file directly on GitHub](https://github.com/game-ci/documentation/edit/main/docs/03-github-orchestrator/05-providers/07-community-providers.mdx) +2. Add your provider using the template below +3. Submit a pull request for review + +### Template + +```markdown +### Provider Name + +| | | +| -------------------- | -------------------------------------------- | +| **Repository** | [user/repo](https://github.com/user/repo) | +| **providerStrategy** | `user/repo` | +| **Description** | Brief description of what the provider does. | +| **Maintainer** | [@username](https://github.com/username) | +``` + +Your provider should: + +- Have a public GitHub repository or published NPM package +- Implement the full [ProviderInterface](custom-providers#required-methods) +- Include a README with setup instructions +- Be actively maintained + +The Game CI team will review submissions for completeness before merging. Inclusion in this list +does not imply endorsement or a security guarantee. diff --git a/docs/03-github-orchestrator/05-providers/08-github-integration.mdx b/docs/03-github-orchestrator/05-providers/08-github-integration.mdx new file mode 100644 index 00000000..9225960f --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/08-github-integration.mdx @@ -0,0 +1,53 @@ +# GitHub Integration + +Orchestrator has first-class support for GitHub Actions. When running from a GitHub Actions +workflow, Orchestrator automatically detects the repository, branch, and commit from the +environment. + +## GitHub Checks + +By enabling the [`githubCheck`](../api-reference#github-integration) parameter, the orchestrator job +will create a GitHub check for each step. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + githubCheck: true + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +The step will show useful details about the job. This is especially useful in combination with async +mode, as you can run very long jobs and monitor their progress directly from the GitHub pull request +UI. + +## Async Mode + +Set [`asyncOrchestrator: true`](../api-reference#github-integration) to start a build without +waiting for it to complete. The GitHub Action will return immediately and you can check progress via +GitHub Checks or by running the [`watch` mode](../api-reference#modes). + +``` + GitHub Action Cloud GitHub PR + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ 1. Dispatch โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ 2. Building โ”‚ โ”‚ โ”‚ + โ”‚ 3. Return โ”‚ โ”‚ ... โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ 4. Check โ”‚ + โ”‚ (done) โ”‚ โ”‚ ... โ”‚ update โ”‚ updated โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ 5. Complete โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ 6. Check โ”‚ + Action finishes โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + in seconds Build runs for Monitor from + minutes/hours the PR page +``` + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + asyncOrchestrator: true + githubCheck: true + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +See the [AWS](aws) and [Kubernetes](kubernetes) provider pages for full workflow files. diff --git a/docs/03-github-orchestrator/05-providers/09-gitlab-integration.mdx b/docs/03-github-orchestrator/05-providers/09-gitlab-integration.mdx new file mode 100644 index 00000000..a6d5d944 --- /dev/null +++ b/docs/03-github-orchestrator/05-providers/09-gitlab-integration.mdx @@ -0,0 +1,30 @@ +# GitLab Integration + +You can use GitLab with Orchestrator via the Command Line mode. Orchestrator is not limited to +GitHub Actions โ€” any CI system that can run shell commands can trigger orchestrator builds. + +## Setup + +1. Install the Orchestrator CLI (see [Command Line](../examples/command-line)) +2. Set your git credentials and provider configuration as environment variables +3. Run the CLI from your `.gitlab-ci.yml` pipeline + +## Example `.gitlab-ci.yml` + +```yaml +build-unity: + stage: build + script: + - git clone https://github.com/game-ci/unity-builder.git /tmp/game-ci + - cd /tmp/game-ci && yarn install + - > + yarn run cli -m cli-build --projectPath $CI_PROJECT_DIR --providerStrategy aws + --gitPrivateToken $GIT_TOKEN + variables: + AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY + AWS_DEFAULT_REGION: eu-west-2 +``` + +See the [Command Line](../examples/command-line) page for more details on CLI usage and the +[Secrets](../secrets) page for managing credentials. diff --git a/docs/03-github-orchestrator/03-examples/02-github-examples/_category_.yaml b/docs/03-github-orchestrator/05-providers/_category_.yaml similarity index 55% rename from docs/03-github-orchestrator/03-examples/02-github-examples/_category_.yaml rename to docs/03-github-orchestrator/05-providers/_category_.yaml index b9807118..94aafd4d 100644 --- a/docs/03-github-orchestrator/03-examples/02-github-examples/_category_.yaml +++ b/docs/03-github-orchestrator/05-providers/_category_.yaml @@ -1,5 +1,5 @@ --- -position: 2.0 -label: GitHub +position: 5.0 +label: Providers collapsible: true collapsed: true diff --git a/docs/03-github-orchestrator/06-advanced-topics/01-caching.mdx b/docs/03-github-orchestrator/06-advanced-topics/01-caching.mdx deleted file mode 100644 index 7e04489b..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/01-caching.mdx +++ /dev/null @@ -1,42 +0,0 @@ -# Caching - -Orchestrator supports two main caching modes: - -- Standard Caching -- Retained Workspace - -_You can even mix the two by specifying a "MaxRetainedWorkspaces" parameter. Above the max_ -_concurrent jobs a new job will use standard caching._ - -## Storage Providers - -Orchestrator supports two storage backends for caching: - -- **S3** (default) - Uses AWS S3 for cache storage. Works with both AWS and LocalStack. -- **Rclone** - Uses rclone for flexible cloud storage. Supports many storage backends. - -Configure via the `storageProvider` parameter. When using rclone, also set `rcloneRemote` to your -configured remote endpoint. - -## Standard Caching - -#### Good For - -- Minimum storage use -- Smaller projects - -#### What is cached between builds - -- LFS files -- Unity Library folder - -## Retained Workspace - -#### Good For - -- Maximum build speed -- Larger projects with long import times - -#### What is cached between builds - -- Entire Project Folder diff --git a/docs/03-github-orchestrator/06-advanced-topics/02-retained-workspace.mdx b/docs/03-github-orchestrator/06-advanced-topics/02-retained-workspace.mdx deleted file mode 100644 index 0e1bfbcd..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/02-retained-workspace.mdx +++ /dev/null @@ -1,8 +0,0 @@ -# Retained Workspaces - -Caches the entire project folder. This option provides the best responsiveness, but also can consume -lots of storage. - -Using API: `maxRetainedWorkspaces`. You can specify a maximum number of retained workspaces, only -one job can use a retained workspace at one time. Each retained workspace consumes more cloud -storage. diff --git a/docs/03-github-orchestrator/06-advanced-topics/03-garbage-collection.mdx b/docs/03-github-orchestrator/06-advanced-topics/03-garbage-collection.mdx deleted file mode 100644 index b49d1a52..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/03-garbage-collection.mdx +++ /dev/null @@ -1,37 +0,0 @@ -# Garbage Collection - -Orchestrator creates, manages and destroys cloud workloads you request. Resources have to be -created. - -It is possible a resource doesn't get deleted by orchestrator after a failed or interrupted build. - -You can use garbage collection to verify everything has been cleaned up. - -Use the **Mode**: `garbage-collect`. - -## Parameters - -```bash -garbageMaxAge -``` - -Maximum age in hours before resources are considered stale and eligible for cleanup. Defaults to -`24`. - -## Usage - -### GitHub Actions - -```yaml -- uses: game-ci/unity-builder@main - with: - providerStrategy: aws - mode: garbage-collect - gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} -``` - -### Command Line - -```bash -yarn run cli -m garbage-collect --providerStrategy aws -``` diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-configuration-override.mdx b/docs/03-github-orchestrator/06-advanced-topics/04-configuration-override.mdx deleted file mode 100644 index 94134fc1..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/04-configuration-override.mdx +++ /dev/null @@ -1,19 +0,0 @@ -# Configuration Override - -When running any unity workload you must provide valid unity credentials. In addition to any other -credentials this is already quite a lot of input. For this reason, it is common to use the command -line mode with input override (link here). This enables you to provide a command to pull input, with -this approach you can create a file to store credentials or pull from a secret manager. - -## Example - -```bash -game-ci -m cli --populateOverride true --readInputFromOverrideList UNITY_EMAIL,UNITY_SERIAL,UNITY_PASSWORD --readInputOverrideCommand="gcloud secrets versions access 1 --secret=\"{0}\"" -``` - -## Required Parameters - -- `populateOverride` - Must be true to run the commands. -- `readInputFromOverrideList` - Comma separated list of parameters to read from override command. -- `readInputOverrideCommand` - A command line command to run (The command is formatted to replace - "{0}" with the parameter parameter name). diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/01-custom-job.mdx b/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/01-custom-job.mdx deleted file mode 100644 index ade52d59..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/01-custom-job.mdx +++ /dev/null @@ -1,4 +0,0 @@ -# Custom Jobs - -You can run a custom job instead of the default build workflow simplfy by specifying the `customJob` -parameter. diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/03-command-hooks.mdx b/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/03-command-hooks.mdx deleted file mode 100644 index ee1f2378..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/03-command-hooks.mdx +++ /dev/null @@ -1,12 +0,0 @@ -# Command Hooks - -Custom commands can be injected into the standard build workflow via yaml or files. - -```yaml -- name: example hook - hook: | - step: - - before - commands: | - echo "hello world!" -``` diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/04-container-hooks.mdx b/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/04-container-hooks.mdx deleted file mode 100644 index 6e0a9adc..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/04-container-hooks.mdx +++ /dev/null @@ -1,13 +0,0 @@ -# Container Hooks - -Custom docker container steps can be run as part of the standard build workflow. Custom steps can -also be run standalone. - -Custom steps can be specified via the CustomSteps parameter or via Custom Step files. - -```yaml -- name: upload - image: amazon/aws-cli - commands: | - echo "hello world!" -``` diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/05-premade-container-jobs.mdx b/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/05-premade-container-jobs.mdx deleted file mode 100644 index a87c9586..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/05-premade-container-jobs.mdx +++ /dev/null @@ -1,38 +0,0 @@ -# Premade Container Jobs - -## Cache syncronization - -### Upload Cache entry to AWS S3 - -Upload cache results from build to AWS S3. - -`aws-s3-upload-cache` - -### Download Latest Cache entry from AWS S3 - -Downloads library and git LFS cache from AWS S3. - -`aws-s3-pull-cache` - -## Artifacts - -## Upload Build Artifact To AWS S3 - -`aws-s3-upload-build` - -Upload build artifact to AWS S3. (Currently only supports lz4 enabled which is default.) - -## Upload Project Artifact To AWS S3 (To Do) - -Upload project artifact to AWS S3. (Currently only supports lz4 enabled which is default.) - -## Artifact entire project folder (To Do) - -archive to tar format, compress with lz4 if enabled and store in persistent cache folder. (Can then -upload.) - -## Deploy - -## Upload to Steam (To Do) - -upload build folder to given steam branch diff --git a/docs/03-github-orchestrator/06-advanced-topics/05-logging.mdx b/docs/03-github-orchestrator/06-advanced-topics/05-logging.mdx deleted file mode 100644 index 99660d31..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/05-logging.mdx +++ /dev/null @@ -1,13 +0,0 @@ -# Logging - -Logs are streamed from the workload to the GameCI origin unless you use the - -## Kubernetes - -- Native Kubernetes logging api - -## AWS - -- Fargate log to Cloud Watch -- Subscription from Cloud Watch to Kinesis -- GameCI streams from logs from Kinesis diff --git a/docs/03-github-orchestrator/06-advanced-topics/06-secrets.mdx b/docs/03-github-orchestrator/06-advanced-topics/06-secrets.mdx deleted file mode 100644 index 17bae040..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/06-secrets.mdx +++ /dev/null @@ -1,14 +0,0 @@ -# Secrets - -Secrets are transferred to workload containers as secrets via the built-in secrets system the -provider being used supports. - -## Kubernetes - -Secrets are created as native Kubernetes secret objects and mounted to job containers as env -variables. - -## AWS - -Secrets are created as aws secret manager secrets and mounted to fargate tasks from secrets to env -variables. diff --git a/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx b/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx deleted file mode 100644 index 70e5079e..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx +++ /dev/null @@ -1,3 +0,0 @@ -# GitLab Introduction - -You can use GitLab with Orchestrator via the Command Line mode. diff --git a/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml b/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml deleted file mode 100644 index 0407bcd5..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml +++ /dev/null @@ -1,5 +0,0 @@ ---- -position: 9.0 -label: GitLab Integration -collapsible: true -collapsed: true diff --git a/docs/03-github-orchestrator/06-advanced-topics/10-github/01-github-checks.mdx b/docs/03-github-orchestrator/06-advanced-topics/10-github/01-github-checks.mdx deleted file mode 100644 index c0625fe3..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/10-github/01-github-checks.mdx +++ /dev/null @@ -1,7 +0,0 @@ -# GitHub Checks - -By enabling the `githubCheck` parameter, the orchestrator job will create a GitHub check for each -step. - -The step will show useful details about the job. This is especially useful in combination with async -mode, as you can run very long jobs. diff --git a/docs/03-github-orchestrator/06-advanced-topics/10-github/_category_.yaml b/docs/03-github-orchestrator/06-advanced-topics/10-github/_category_.yaml deleted file mode 100644 index ecbee03f..00000000 --- a/docs/03-github-orchestrator/06-advanced-topics/10-github/_category_.yaml +++ /dev/null @@ -1,5 +0,0 @@ ---- -position: 10.0 -label: GitHub Integration -collapsible: true -collapsed: true diff --git a/docs/03-github-orchestrator/06-secrets.mdx b/docs/03-github-orchestrator/06-secrets.mdx new file mode 100644 index 00000000..9354f08d --- /dev/null +++ b/docs/03-github-orchestrator/06-secrets.mdx @@ -0,0 +1,57 @@ +# Secrets + +Orchestrator securely transfers secrets to remote build containers using each provider's native +secrets system. + +## Kubernetes + +Secrets are created as native **Kubernetes Secret** objects and mounted as environment variables in +job containers. + +## AWS + +Secrets are stored in **AWS Secrets Manager** and injected into Fargate tasks as environment +variables. + +## ๐Ÿ” Pulling Secrets from External Sources + +You can pull parameter values from external secret managers or files at runtime instead of +hardcoding credentials. This keeps CLI commands short and secrets out of your repository. + +``` + Orchestrator External Source + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” command โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Reads input โ”‚ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ Secret Manager โ”‚ + โ”‚ override โ”‚ โ”‚ (GCP, AWS, file) โ”‚ + โ”‚ list โ”‚ โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚ โ”‚ + โ”‚ โ”‚ value โ”‚ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### Parameters + +| Parameter | Default | Description | +| --------------------------- | ------- | ----------------------------------------------------------------------------------------------------- | +| `readInputOverrideCommand` | โ€” | Command to run for each secret. Use `{0}` as a placeholder for the parameter name. | +| `readInputFromOverrideList` | โ€” | Comma-separated list of parameter names to pull via `readInputOverrideCommand`. | +| `populateOverride` | โ€” | Must be `true` to enable pulling secrets (CLI only). Auto-enabled in GitHub Actions when command set. | + +### Built-in Presets + +Instead of writing a full command, use these presets as the `readInputOverrideCommand`: + +| Preset | Expands to | +| -------------------- | ----------------------------------------------------- | +| `gcp-secret-manager` | `gcloud secrets versions access 1 --secret="{0}"` | +| `aws-secret-manager` | `aws secretsmanager get-secret-value --secret-id {0}` | + +### Example + +```bash +yarn run cli -m cli-build \ + --populateOverride true \ + --readInputFromOverrideList UNITY_EMAIL,UNITY_SERIAL,UNITY_PASSWORD \ + --readInputOverrideCommand='gcloud secrets versions access 1 --secret="{0}"' +``` + +This runs the GCP command for each parameter name in the list and uses the output as the value. diff --git a/docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx b/docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx new file mode 100644 index 00000000..cc458ebb --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx @@ -0,0 +1,77 @@ +# Caching + +Orchestrator supports two caching strategies. You can mix both by setting +[`maxRetainedWorkspaces`](../api-reference#caching) โ€” once the limit is reached, additional jobs +fall back to standard caching. + +``` + Standard Caching Retained Workspace + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ LFS files โ”‚ โ”‚ Entire project โ”‚ + โ”‚ Library/ โ”‚ โ”‚ folder โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ Smaller storage โ”‚ โ”‚ Faster builds โ”‚ + โ”‚ Good for small โ”‚ โ”‚ Good for large โ”‚ + โ”‚ projects โ”‚ โ”‚ projects โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Standard Caching + +Caches only the Unity **Library** folder and **LFS** files between builds. Uses less storage but +requires re-importing unchanged assets. + +- โœ… Minimum storage cost +- โœ… Best for smaller projects +- โš ๏ธ Slower rebuilds for large asset libraries + +## Build Caching + +Orchestrator automatically caches build output alongside the Library cache. After each build, the +compiled output folder is archived and stored using the same cache key (branch name by default). On +the next build with the same cache key, the previous build output is available at +`/data/cache/{cacheKey}/build/`. + +This happens automatically โ€” no configuration required. The cache key controls which builds share +output: + +```yaml +# Builds on the same branch share cached output (default behavior) +cacheKey: ${{ github.ref_name }} + +# Or share across branches by using a fixed key +cacheKey: shared-cache +``` + +Build caching uses the same compression and storage provider as Library caching. Archives are stored +as `build-{buildGuid}.tar.lz4` (or `.tar` if compression is disabled). See [Storage](storage) for +details on compression and storage backends. + +## Retained Workspace + +Caches the **entire project folder** between builds. Provides the fastest rebuilds but consumes more +storage. + +- โœ… Maximum build speed +- โœ… Best for large projects with long import times +- โš ๏ธ Higher storage cost + +See [Retained Workspaces](retained-workspace) for configuration details. + +## ๐Ÿ—„๏ธ Storage Providers + +| Provider | `storageProvider` | Description | +| -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| S3 | `s3` (default) | AWS S3 storage. Works with both AWS and LocalStack. | +| Rclone | `rclone` | Flexible cloud storage via [rclone](https://rclone.org). Supports 70+ backends (Google Cloud, Azure Blob, Backblaze, SFTP, etc). | + +Configure with the [`storageProvider`](../api-reference#storage) parameter. When using rclone, also +set `rcloneRemote` to your configured remote endpoint. + +## ๐Ÿ”’ Workspace Locking + +When using retained workspaces, Orchestrator uses distributed locking (via S3 or rclone) to ensure +only one build uses a workspace at a time. This enables safe concurrent builds that share and reuse +workspaces without conflicts. + +Locking is managed automatically โ€” no configuration required beyond setting `maxRetainedWorkspaces`. diff --git a/docs/03-github-orchestrator/07-advanced-topics/02-retained-workspace.mdx b/docs/03-github-orchestrator/07-advanced-topics/02-retained-workspace.mdx new file mode 100644 index 00000000..f4076e34 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/02-retained-workspace.mdx @@ -0,0 +1,47 @@ +# Retained Workspaces + +Retained workspaces cache the **entire project folder** between builds. This provides the fastest +possible rebuilds at the cost of more cloud storage. + +## Configuration + +Set `maxRetainedWorkspaces` to control how many workspaces are kept: + +| Value | Behavior | +| ----- | ------------------------------------------------------------------------- | +| `0` | Unlimited retained workspaces (default). | +| `> 0` | Keep at most N workspaces. Additional jobs fall back to standard caching. | + +Each retained workspace is locked during use โ€” only one build can use a workspace at a time. +Orchestrator handles locking automatically via S3 or rclone. See [Caching](caching) for storage +provider details. + +``` + maxRetainedWorkspaces: 3 + + Workspace 1 Workspace 2 Workspace 3 + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ [locked] โ”‚ โ”‚ [locked] โ”‚ โ”‚ (idle) โ”‚ + โ”‚ Build A โ”‚ โ”‚ Build B โ”‚ โ”‚ โ”‚ + โ”‚ Full project โ”‚ โ”‚ Full project โ”‚ โ”‚ Full project โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + + Build C arrives --> claims Workspace 3 + Build D arrives --> all locked --> falls back to standard caching +``` + +## Example + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + maxRetainedWorkspaces: 3 + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +## โš ๏ธ Storage Considerations + +Each retained workspace stores a full copy of your project. For a 20 GB project with 3 retained +workspaces, expect ~60 GB of cloud storage usage. diff --git a/docs/03-github-orchestrator/07-advanced-topics/03-custom-job.mdx b/docs/03-github-orchestrator/07-advanced-topics/03-custom-job.mdx new file mode 100644 index 00000000..4bcaddc8 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/03-custom-job.mdx @@ -0,0 +1,19 @@ +# Custom Job + +Override the default build workflow entirely by specifying the `customJob` parameter with a YAML job +definition. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + customJob: | + - name: my-custom-step + image: ubuntu + commands: | + echo "Running custom job" + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +This replaces the standard build steps with your own. Useful for running non-Unity workloads or +fully custom pipelines through Orchestrator's cloud infrastructure. diff --git a/docs/03-github-orchestrator/07-advanced-topics/04-garbage-collection.mdx b/docs/03-github-orchestrator/07-advanced-topics/04-garbage-collection.mdx new file mode 100644 index 00000000..30830fba --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/04-garbage-collection.mdx @@ -0,0 +1,56 @@ +# Garbage Collection + +Orchestrator creates cloud resources (containers, stacks, volumes) for each build and cleans them up +automatically. If a build fails or is interrupted, resources may be left behind. + +``` + Normal Build Failed / Interrupted Build + + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” Auto โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Create โ”‚โ”€โ–บโ”‚ Build โ”‚โ”€โ–บClean โ”‚ Create โ”‚โ”€โ–บโ”‚ Build โ”‚โ”€โ”€โ–บ crash + โ”‚ resourcesโ”‚ โ”‚ โ”‚ up โ”‚ resourcesโ”‚ โ”‚ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + | + v + Resources left behind + | + v + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ garbage-collect โ”‚ + โ”‚ removes after โ”‚ + โ”‚ garbageMaxAge โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +Use garbage collection to clean up stale resources. See the +[API Reference](../api-reference#garbage-collection) for all parameters. + +## Usage + +### GitHub Actions + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + mode: garbage-collect + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +### Command Line + +```bash +yarn run cli -m garbage-collect --providerStrategy aws +``` + +## Parameters + +| Parameter | Default | Description | +| --------------- | ------- | ----------------------------------------------------- | +| `garbageMaxAge` | `24` | Maximum age in hours before resources are cleaned up. | + +## ๐Ÿ”„ Automatic Cleanup + +When using the AWS provider, Orchestrator can create a CloudFormation-based cleanup cron job that +automatically removes old ECS task definitions and resources. This is controlled by the +`useCleanupCron` parameter (enabled by default). diff --git a/docs/03-github-orchestrator/07-advanced-topics/05-hooks/03-command-hooks.mdx b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/03-command-hooks.mdx new file mode 100644 index 00000000..5e0a71cc --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/03-command-hooks.mdx @@ -0,0 +1,40 @@ +# Command Hooks + +Inject custom shell commands into the standard build workflow at specific points. + +## Format + +```yaml +- name: example hook + hook: | + step: + - before + commands: | + echo "hello world!" +``` + +## Hook Points + +| Step | When it runs | +| -------- | ------------------------------- | +| `before` | Before the build step starts. | +| `after` | After the build step completes. | + +## Usage + +Pass hooks inline via the `commandHooks` parameter or reference files from the `.game-ci/hooks/` +directory via `customHookFiles`. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + customHookFiles: my-hook + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +Place hook files at `.game-ci/hooks/my-hook.yaml` in your repository. + +For running Docker containers as build steps, see [Container Hooks](container-hooks). For +ready-to-use hooks (S3, rclone, Steam), see [Built-In Hooks](built-in-hooks). diff --git a/docs/03-github-orchestrator/07-advanced-topics/05-hooks/04-container-hooks.mdx b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/04-container-hooks.mdx new file mode 100644 index 00000000..f88a41ae --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/04-container-hooks.mdx @@ -0,0 +1,45 @@ +# Container Hooks + +Run custom Docker containers as steps in the build workflow. Useful for uploading artifacts, +deploying builds, or running additional tools. For inline shell commands instead, see +[Command Hooks](command-hooks). + +``` + Build Pipeline + + preBuildContainerHooks Unity Build postBuildContainerHooks + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Pull cache โ”‚ โ”‚ โ”‚ โ”‚ Upload build โ”‚ + โ”‚ Setup deps โ”‚โ”€โ”€โ–บโ”‚ Build โ”‚โ”€โ”€โ–บโ”‚ Deploy to Steam โ”‚ + โ”‚ ... โ”‚ โ”‚ โ”‚ โ”‚ ... โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + Runs before build Core build step Runs after build +``` + +## Format + +```yaml +- name: upload + image: amazon/aws-cli + commands: | + echo "hello world!" +``` + +## Usage + +Define container hooks inline via `preBuildContainerHooks` / `postBuildContainerHooks`, or reference +files from `.game-ci/container-hooks/` via `containerHookFiles`. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + containerHookFiles: aws-s3-upload-build + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +## Built-In Hooks + +Orchestrator ships with ready-to-use hooks for S3, rclone, and Steam. See +[Built-In Hooks](built-in-hooks) for the full list. diff --git a/docs/03-github-orchestrator/07-advanced-topics/05-hooks/05-built-in-hooks.mdx b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/05-built-in-hooks.mdx new file mode 100644 index 00000000..40b53c80 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/05-built-in-hooks.mdx @@ -0,0 +1,46 @@ +# Built-In Hooks + +Orchestrator ships with pre-built container hooks for common tasks. Use them by name with the +`containerHookFiles` parameter. + +```yaml +containerHookFiles: aws-s3-upload-build +``` + +## ๐Ÿ“ฆ AWS S3 + +| Hook Name | Description | +| --------------------- | ----------------------------------------- | +| `aws-s3-upload-build` | Upload build artifacts to S3. | +| `aws-s3-pull-build` | Pull cached build artifacts from S3. | +| `aws-s3-upload-cache` | Upload Unity Library and LFS cache to S3. | +| `aws-s3-pull-cache` | Pull Unity Library and LFS cache from S3. | + +Requires AWS credentials configured. Respects `useCompressionStrategy` for LZ4 compression. + +## ๐Ÿ“‚ Rclone + +| Hook Name | Description | +| --------------------- | ---------------------------------------------- | +| `rclone-upload-build` | Upload build artifacts via rclone. | +| `rclone-pull-build` | Pull cached build artifacts via rclone. | +| `rclone-upload-cache` | Upload Unity Library and LFS cache via rclone. | +| `rclone-pull-cache` | Pull Unity Library and LFS cache via rclone. | + +Requires `storageProvider: rclone` and `rcloneRemote` to be configured. Uses the `rclone/rclone` +Docker image. Respects `useCompressionStrategy` for LZ4 compression. + +## ๐ŸŽฎ Steam + +| Hook Name | Description | +| ---------------------- | --------------------------------------------- | +| `steam-deploy-client` | Deploy a client build to Steam via SteamCMD. | +| `steam-deploy-project` | Deploy a project build to Steam via SteamCMD. | + +Uses the `steamcmd/steamcmd` Docker image. Requires the following secrets to be configured: + +- `STEAM_USERNAME`, `STEAM_PASSWORD` +- `STEAM_APPID` +- `STEAM_SSFN_FILE_NAME`, `STEAM_SSFN_FILE_CONTENTS` +- `STEAM_CONFIG_VDF_1` through `STEAM_CONFIG_VDF_4` +- `BUILD_GUID_TARGET`, `RELEASE_BRANCH` diff --git a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/_category_.yaml b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/_category_.yaml similarity index 72% rename from docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/_category_.yaml rename to docs/03-github-orchestrator/07-advanced-topics/05-hooks/_category_.yaml index 445f7ab4..6983ed19 100644 --- a/docs/03-github-orchestrator/06-advanced-topics/04-custom-hooks/_category_.yaml +++ b/docs/03-github-orchestrator/07-advanced-topics/05-hooks/_category_.yaml @@ -1,5 +1,5 @@ --- position: 4.0 -label: Custom Hooks +label: Hooks collapsible: true collapsed: true diff --git a/docs/03-github-orchestrator/07-advanced-topics/06-logging.mdx b/docs/03-github-orchestrator/07-advanced-topics/06-logging.mdx new file mode 100644 index 00000000..e7c8c9af --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/06-logging.mdx @@ -0,0 +1,34 @@ +# Logging + +Orchestrator streams logs from the remote build back to your CI runner in real time. + +``` + Cloud Container Orchestrator Your CI + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Build output โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ Log stream โ”‚โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ Console โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ output โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Provider-Specific Log Transport + +### Kubernetes + +Uses the native Kubernetes logging API to stream pod logs directly. + +### AWS + +Logs flow through a CloudWatch โ†’ Kinesis pipeline: + +1. Orchestrator job (Fargate task) writes logs to **CloudWatch Logs** +2. A **Kinesis** subscription forwards logs in real time +3. Orchestrator consumes from the Kinesis stream + +## ๐Ÿ› Debug Logging + +Enable [`orchestratorDebug: true`](../api-reference#build-options) to get verbose output including: + +- Resource allocation summaries (CPU, memory, disk) +- Directory structure via `tree` +- Environment variable listing +- Disk usage snapshots (`df -h`, `du -sh`) diff --git a/docs/03-github-orchestrator/07-advanced-topics/07-load-balancing.mdx b/docs/03-github-orchestrator/07-advanced-topics/07-load-balancing.mdx new file mode 100644 index 00000000..dcd45b12 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/07-load-balancing.mdx @@ -0,0 +1,257 @@ +# Load Balancing + +Orchestrator supports multiple providers, and you can use standard GitHub Actions scripting to route +builds across them. This lets you distribute builds based on platform, project size, runner +availability, or any custom logic. + +``` + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Incoming Build โ”‚ + โ”‚ Request โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Router Logic โ”‚ + โ”‚ (GitHub Actions / โ”‚ + โ”‚ script / matrix) โ”‚ + โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”˜ + โ”‚ โ”‚ โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ” โ”Œโ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ aws โ”‚ โ”‚ k8s โ”‚ โ”‚ local-docker โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ Fargate โ”‚ โ”‚ Cluster โ”‚ โ”‚ Self-hosted โ”‚ + โ”‚ (scalable) โ”‚ โ”‚ (flexible) โ”‚ โ”‚ (no cost) โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Why Load Balance? + +- **Cost control** โ€” Route small or test builds to free self-hosted runners and reserve cloud + providers for production builds. +- **Availability** โ€” Fall back to a cloud provider when your self-hosted runner is busy or offline. +- **Platform routing** โ€” Send Linux builds to AWS and Windows builds to a local runner. +- **Parallel scaling** โ€” Split a multi-platform matrix across providers to finish faster. + +## Basic Provider Routing + +Use a GitHub Actions matrix or conditional logic to route builds to different providers. + +### Route by Platform + +```yaml +name: Platform-Based Routing + +on: + push: + branches: [main] + +jobs: + build: + name: Build ${{ matrix.targetPlatform }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + # Linux builds go to AWS (fast, scalable) + - targetPlatform: StandaloneLinux64 + provider: aws + # Windows builds go to self-hosted runner + - targetPlatform: StandaloneWindows64 + provider: local-docker + # WebGL builds go to Kubernetes + - targetPlatform: WebGL + provider: k8s + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: ${{ matrix.provider }} + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +### Route by Branch + +Send production builds to a high-resource cloud provider and development builds to a cheaper option. + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - name: Select provider + id: provider + run: | + if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then + echo "strategy=aws" >> "$GITHUB_OUTPUT" + echo "cpu=4096" >> "$GITHUB_OUTPUT" + echo "memory=16384" >> "$GITHUB_OUTPUT" + else + echo "strategy=local-docker" >> "$GITHUB_OUTPUT" + echo "cpu=1024" >> "$GITHUB_OUTPUT" + echo "memory=3072" >> "$GITHUB_OUTPUT" + fi + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: ${{ steps.provider.outputs.strategy }} + targetPlatform: StandaloneLinux64 + containerCpu: ${{ steps.provider.outputs.cpu }} + containerMemory: ${{ steps.provider.outputs.memory }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +## Fallback on Runner Availability + +If your self-hosted runner is offline or busy, fall back to a cloud provider. Use a short initial +job to check runner health, then pick the provider accordingly. + +```yaml +name: Fallback Routing + +on: + push: + branches: [main] + +jobs: + check-runner: + name: Check self-hosted availability + runs-on: ubuntu-latest + outputs: + provider: ${{ steps.pick.outputs.provider }} + steps: + - name: Check if self-hosted runner is available + id: pick + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + # Query runner status via the GitHub API + RUNNERS=$(gh api repos/${{ github.repository }}/actions/runners --jq '[.runners[] | select(.status == "online")] | length') + if [[ "$RUNNERS" -gt 0 ]]; then + echo "provider=local-docker" >> "$GITHUB_OUTPUT" + echo "Self-hosted runner available โ€” using local-docker" + else + echo "provider=aws" >> "$GITHUB_OUTPUT" + echo "No runners online โ€” falling back to AWS" + fi + + build: + name: Build + needs: check-runner + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: ${{ needs.check-runner.outputs.provider }} + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +## Async Mode for Load Balancing + +The [`asyncOrchestrator`](../api-reference#github-integration) parameter works well with load +balancing. When enabled, the GitHub Action dispatches the build and returns immediately without +waiting for it to finish. This means your routing job completes in seconds regardless of which +provider handles the actual build. + +``` + Router Job (seconds) Provider A Provider B + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ 1. Check runners โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ 2. Pick provider โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ 3. Dispatch build โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ–บโ”‚ 3a. Building โ”‚ OR โ”€โ–บโ”‚ 3b. Building โ”‚ + โ”‚ 4. Return (done) โ”‚ โ”‚ ... โ”‚ โ”‚ ... โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ ... โ”‚ โ”‚ ... โ”‚ + Completes instantly โ”‚ 5. Complete โ”‚ โ”‚ 5. Complete โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ GitHub Check updated โ”‚ + โ”‚ (monitor from PR page) โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +Benefits of combining async mode with load balancing: + +- **No wasted runner minutes** โ€” The router job finishes in seconds. You don't pay for a + GitHub-hosted runner to sit idle while the build runs in the cloud. +- **Parallel dispatch** โ€” Launch multiple builds to different providers simultaneously. Each + dispatches and returns immediately. +- **Monitor everything from one place** โ€” Enable `githubCheck: true` and all builds report status + back to the pull request Checks tab, regardless of which provider ran them. + +```yaml +jobs: + build: + name: Build ${{ matrix.targetPlatform }} + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + include: + - targetPlatform: StandaloneLinux64 + provider: aws + - targetPlatform: StandaloneWindows64 + provider: k8s + steps: + - uses: actions/checkout@v4 + with: + lfs: true + + - uses: game-ci/unity-builder@v4 + with: + providerStrategy: ${{ matrix.provider }} + targetPlatform: ${{ matrix.targetPlatform }} + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} + asyncOrchestrator: true + githubCheck: true +``` + +Each matrix job dispatches its build and exits immediately. The builds continue running on their +respective providers and report progress via GitHub Checks. + +## Weighted Distribution + +For teams with both cloud and self-hosted capacity, distribute builds based on a ratio. This example +sends 70% of builds to AWS and 30% to a local runner using a simple hash. + +```yaml +- name: Weighted provider selection + id: provider + run: | + # Hash the run ID to get a stable pseudo-random number + HASH=$(echo "${{ github.run_id }}" | md5sum | cut -c1-8) + DECIMAL=$((16#$HASH % 100)) + if [[ $DECIMAL -lt 70 ]]; then + echo "strategy=aws" >> "$GITHUB_OUTPUT" + else + echo "strategy=local-docker" >> "$GITHUB_OUTPUT" + fi +``` + +## Tips + +- **Start simple** โ€” A platform-based matrix with different providers per entry is the easiest + starting point. Add dynamic routing only when you need it. +- **Use async for long builds** โ€” Combining `asyncOrchestrator: true` with `githubCheck: true` keeps + your routing job fast and gives you build status on the PR page. +- **Cache keys are provider-independent** โ€” The [`cacheKey`](../api-reference#caching) parameter + works the same across all providers, so builds routed to different providers can still share + caches if they use the same storage backend. +- **Test fallback logic** โ€” Temporarily disable your self-hosted runner to verify that your fallback + routing works before you need it in production. +- **Custom providers** โ€” The same routing patterns work with + [custom providers](../providers/custom-providers). Set `providerStrategy` to a GitHub repo or NPM + package and Orchestrator loads it dynamically. diff --git a/docs/03-github-orchestrator/07-advanced-topics/08-storage.mdx b/docs/03-github-orchestrator/07-advanced-topics/08-storage.mdx new file mode 100644 index 00000000..900caa39 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/08-storage.mdx @@ -0,0 +1,215 @@ +# Storage + +Orchestrator manages three categories of files during a build: **project files** (your Unity project +and Git LFS assets), **build output** (the compiled game), and **caches** (Unity Library folder and +LFS files). This page explains how each category flows through the system and how to configure the +storage backend. + +``` + CI / Local Machine Build Container + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Git repository โ”‚โ”€โ”€โ”€โ”€ clone โ”€โ”€โ”€โ”€โ–บโ”‚ /data/{buildGuid}/repo/ โ”‚ + โ”‚ + LFS assets โ”‚ โ”‚ โ”œโ”€โ”€ Assets/ โ”‚ + โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Library/ (cached) โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ””โ”€โ”€ .git/lfs/ (cached) โ”‚ + โ”‚ โ”‚ + โ”‚ /data/cache/{cacheKey}/ โ”‚ + โ”‚ โ”œโ”€โ”€ Library/ (tar.lz4) โ”‚ + โ”‚ โ””โ”€โ”€ build/ (tar.lz4) โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Cloud Storage (S3 / rclone) โ”‚ + โ”‚ โ”œโ”€โ”€ Library cache archives โ”‚ + โ”‚ โ”œโ”€โ”€ Build artifacts โ”‚ + โ”‚ โ””โ”€โ”€ Workspace locks โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## File Categories + +### Project Files + +Your Unity project is cloned into the build container at `/data/{buildGuid}/repo/`. Orchestrator +handles Git and LFS automatically: + +1. **Shallow clone** โ€” The repository is cloned with `--depth` controlled by the + [`cloneDepth`](../api-reference#git-synchronization) parameter (default: 50). +2. **LFS pull** โ€” Git LFS is installed and configured inside the container. LFS files are pulled + after the clone completes. +3. **LFS hashing** โ€” Orchestrator generates `.lfs-assets-guid` and `.lfs-assets-guid-sum` files to + track LFS content for cache invalidation. + +For retained workspaces, the project folder persists between builds at +`/data/{lockedWorkspace}/repo/` instead of being cloned fresh each time. See +[Retained Workspaces](retained-workspace) for details. + +### Build Output + +After Unity finishes building, the compiled output lives at `/data/{buildGuid}/repo/{buildPath}/`. +Orchestrator archives this folder as `build-{buildGuid}.tar` (or `.tar.lz4` with compression +enabled). + +To export build artifacts out of the container, use [container hooks](hooks/container-hooks). The +most common approach is the built-in S3 or rclone upload hooks: + +```yaml +# Upload build artifacts to S3 +containerHookFiles: aws-s3-upload-build + +# Or upload via rclone to any backend +containerHookFiles: rclone-upload-build +``` + +See [Built-In Hooks](hooks/built-in-hooks) for all available hooks. + +### Caches + +Orchestrator caches two things between builds to speed up subsequent runs: + +| Cache | Contents | Path in container | +| ------------- | ------------------------------------------- | --------------------------------- | +| Library cache | Unity's `Library/` folder (imported assets) | `/data/cache/{cacheKey}/Library/` | +| Build cache | Previous build output | `/data/cache/{cacheKey}/build/` | + +Caches are scoped by the [`cacheKey`](../api-reference#caching) parameter, which defaults to the +branch name. Builds on the same branch share a cache. + +After a build completes, Orchestrator runs `remote-cli-post-build` which: + +1. Archives the Library folder as `lib-{buildGuid}.tar` (or `.tar.lz4`). +2. Archives the build output as `build-{buildGuid}.tar` (or `.tar.lz4`). +3. Pushes both archives to cloud storage via the configured storage provider. + +Before the next build, `remote-cli-pre-build` pulls these archives and extracts them, so Unity can +skip re-importing unchanged assets. + +## Storage Providers + +Orchestrator supports two storage backends for caches, artifacts, and workspace locks. + +``` + storageProvider: "s3" storageProvider: "rclone" + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ AWS S3 โ”‚ โ”‚ Rclone โ”‚ + โ”‚ โ”‚ โ”‚ โ”‚ + โ”‚ - Default backend โ”‚ โ”‚ - 70+ backends โ”‚ + โ”‚ - Works with โ”‚ โ”‚ - Google Cloud โ”‚ + โ”‚ LocalStack โ”‚ โ”‚ - Azure Blob โ”‚ + โ”‚ - Built-in lock โ”‚ โ”‚ - Backblaze B2 โ”‚ + โ”‚ support โ”‚ โ”‚ - SFTP, FTP โ”‚ + โ”‚ โ”‚ โ”‚ - Any rclone โ”‚ + โ”‚ โ”‚ โ”‚ remote โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### S3 (Default) + +S3 is the default storage backend. It works with AWS S3 and LocalStack (for local testing). + +No extra configuration is needed when using the `aws` provider โ€” the S3 bucket is created +automatically as part of the CloudFormation base stack. For other providers, ensure AWS credentials +and region are set in the environment. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: aws + # storageProvider defaults to "s3" + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +### Rclone + +[Rclone](https://rclone.org) is a command-line tool that supports 70+ cloud storage backends. Use it +when you want to store caches and artifacts somewhere other than S3. + +```yaml +- uses: game-ci/unity-builder@v4 + with: + providerStrategy: k8s + storageProvider: rclone + rcloneRemote: 'myremote:bucket/path' + targetPlatform: StandaloneLinux64 + gitPrivateToken: ${{ secrets.GITHUB_TOKEN }} +``` + +Rclone hooks run in the `rclone/rclone` Docker image. Configure your rclone remote beforehand (see +[rclone docs](https://rclone.org/docs/)). + +**Supported backends include:** Google Cloud Storage, Azure Blob, Backblaze B2, DigitalOcean Spaces, +Wasabi, MinIO, SFTP, FTP, and [many more](https://rclone.org/overview/). + +## Compression + +Orchestrator uses **LZ4 compression** by default for all cache and build archives. LZ4 is optimized +for speed over compression ratio, which is ideal for large Unity Library folders where fast +decompression matters more than file size. + +| `useCompressionStrategy` | Archive format | Description | +| ------------------------ | -------------- | ------------------------------------------------ | +| `true` (default) | `.tar.lz4` | LZ4 compressed. Faster extract, ~30-50% smaller. | +| `false` | `.tar` | Uncompressed. Slightly faster to create. | + +```yaml +# Disable compression (not recommended for most projects) +useCompressionStrategy: false +``` + +## Workspace Locking + +When using [retained workspaces](retained-workspace), Orchestrator uses distributed locking to +ensure only one build occupies a workspace at a time. Locks are stored in the configured storage +provider: + +- **S3**: Lock files at `s3://{awsStackName}/locks/{workspaceName}/{buildGuid}` +- **Rclone**: Lock files at `{rcloneRemote}/locks/{workspaceName}/{buildGuid}` + +Locking is fully automatic. When a build starts, it acquires a lock. When it finishes (or fails), +the lock is released. If all workspaces are locked, the build falls back to standard caching. + +## Large Packages + +The [`useLargePackages`](../api-reference#build-options) parameter optimizes storage for Unity +packages containing "LargePackage" in `manifest.json`. When enabled, these packages are redirected +to a shared folder so multiple builds with the same cache key share one copy instead of duplicating +data. + +```yaml +useLargePackages: true +``` + +## Container File System Layout + +Inside the build container, Orchestrator uses the `/data/` volume mount as the root for all project +and cache data. + +``` +/data/ +โ”œโ”€โ”€ {buildGuid}/ # Unique job folder (or {lockedWorkspace}/) +โ”‚ โ”œโ”€โ”€ repo/ # Cloned repository +โ”‚ โ”‚ โ”œโ”€โ”€ Assets/ +โ”‚ โ”‚ โ”œโ”€โ”€ Library/ # Unity Library (restored from cache) +โ”‚ โ”‚ โ”œโ”€โ”€ .git/ +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ lfs/ # Git LFS objects +โ”‚ โ”‚ โ””โ”€โ”€ {buildPath}/ # Build output +โ”‚ โ””โ”€โ”€ builder/ # Cloned game-ci/unity-builder +โ”‚ โ””โ”€โ”€ dist/ +โ”‚ โ””โ”€โ”€ index.js +โ””โ”€โ”€ cache/ + โ””โ”€โ”€ {cacheKey}/ # Scoped by branch name + โ”œโ”€โ”€ Library/ + โ”‚ โ””โ”€โ”€ lib-{guid}.tar.lz4 # Library cache archive + โ””โ”€โ”€ build/ + โ””โ”€โ”€ build-{guid}.tar.lz4 # Build output archive +``` + +## Storage Parameters + +For the full list of storage-related parameters, see the API Reference: + +- [Storage](../api-reference#storage) โ€” `storageProvider`, `rcloneRemote` +- [Caching](../api-reference#caching) โ€” `cacheKey`, `maxRetainedWorkspaces` +- [Build Options](../api-reference#build-options) โ€” `useCompressionStrategy`, `useLargePackages` +- [AWS](../api-reference#aws) โ€” `awsStackName`, `awsS3Endpoint` diff --git a/docs/03-github-orchestrator/07-advanced-topics/09-architecture.mdx b/docs/03-github-orchestrator/07-advanced-topics/09-architecture.mdx new file mode 100644 index 00000000..fc838dc1 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/09-architecture.mdx @@ -0,0 +1,308 @@ +# Architecture + +This page describes the internal architecture of Orchestrator โ€” how the components fit together, how +a build flows through the system, and where to look in the source code. + +``` + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Entry Point โ”‚ + โ”‚ GitHub Action / CLI โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Orchestrator โ”‚ + โ”‚ (orchestrator.ts) โ”‚ + โ”‚ โ”‚ + โ”‚ - setup() โ”‚ + โ”‚ - run() โ”‚ + โ”‚ - Provider selection โ”‚ + โ””โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”˜ + โ”‚ โ”‚ โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Provider โ”‚ โ”‚ Workflow โ”‚ โ”‚ Services โ”‚ + โ”‚ (pluggable) โ”‚ โ”‚ Composition โ”‚ โ”‚ โ”‚ + โ”‚ โ”‚ โ”‚ Root โ”‚ โ”‚ - Logger โ”‚ + โ”‚ - aws โ”‚ โ”‚ โ”‚ โ”‚ - Hooks โ”‚ + โ”‚ - k8s โ”‚ โ”‚ - Standard โ”‚ โ”‚ - Caching โ”‚ + โ”‚ - local-docker โ”‚ โ”‚ - Async โ”‚ โ”‚ - Locking โ”‚ + โ”‚ - local โ”‚ โ”‚ - Custom Job โ”‚ โ”‚ - LFS โ”‚ + โ”‚ - custom plugin โ”‚ โ”‚ โ”‚ โ”‚ - GitHub Checks โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Build Lifecycle + +A standard Orchestrator build follows these steps: + +``` + 1. Initialize 2. Setup Provider 3. Acquire Workspace + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Parse inputs โ”‚ โ”‚ Create cloud โ”‚ โ”‚ Lock retained โ”‚ + โ”‚ Select provider โ”‚โ”€โ”€โ–บโ”‚ resources โ”‚โ”€โ”€โ–บโ”‚ workspace โ”‚ + โ”‚ Generate GUID โ”‚ โ”‚ (stacks, PVCs) โ”‚ โ”‚ (if enabled) โ”‚ + โ”‚ Create GH check โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + 6. Cleanup 5. Post-Build 4. Build + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ Release workspaceโ”‚ โ”‚ Push Library โ”‚ โ”‚ Clone repo + LFS โ”‚ + โ”‚ Delete cloud โ”‚โ—„โ”€โ”€โ”‚ and build cache โ”‚โ—„โ”€โ”€โ”‚ Restore cache โ”‚ + โ”‚ resources โ”‚ โ”‚ Run post hooks โ”‚ โ”‚ Run Unity build โ”‚ + โ”‚ Update GH check โ”‚ โ”‚ โ”‚ โ”‚ Run pre/post โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ container hooks โ”‚ + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +### Step-by-Step + +1. **Initialize** โ€” `Orchestrator.setup()` parses inputs from GitHub Action `with`, CLI flags, or + environment variables. It selects a provider, generates a unique build GUID, and optionally + creates a GitHub Check. + +2. **Setup Provider** โ€” `Provider.setupWorkflow()` provisions cloud resources. For AWS this means + creating a CloudFormation base stack (ECS cluster, S3 bucket, Kinesis stream). For Kubernetes it + creates a service account. Local providers skip this step. + +3. **Acquire Workspace** โ€” If `maxRetainedWorkspaces` is set, `SharedWorkspaceLocking` acquires a + distributed lock on a workspace via S3 or rclone. If all workspaces are locked, the build falls + back to standard caching. + +4. **Build** โ€” The workflow composition root selects a workflow (standard, async, or custom job). + The standard workflow clones your repo, restores caches, runs the Unity build, and executes + pre/post container hooks. + +5. **Post-Build** โ€” `remote-cli-post-build` archives the Unity Library folder and build output, + pushes them to cloud storage, and runs any post-build hooks. + +6. **Cleanup** โ€” The workspace lock is released, cloud resources are torn down + (`Provider.cleanupWorkflow()`), and the GitHub Check is updated with the final result. + +## Core Components + +### Orchestrator (`orchestrator.ts`) + +The static `Orchestrator` class is the central coordinator. It holds: + +- `Provider` โ€” the selected provider implementation +- `buildParameters` โ€” all resolved configuration +- `buildGuid` โ€” unique identifier for this build +- `lockedWorkspace` โ€” retained workspace name (if any) + +`Orchestrator.run()` is the main entry point that drives the full lifecycle. Provider selection +happens in `setupSelectedBuildPlatform()`, which handles LocalStack detection, `AWS_FORCE_PROVIDER` +overrides, and fallback to the plugin loader for custom providers. + +### Provider System + +Providers implement the `ProviderInterface`: + +```typescript +interface ProviderInterface { + setupWorkflow(buildGuid, buildParameters, branchName, secrets): Promise; + runTaskInWorkflow( + buildGuid, + image, + commands, + mountdir, + workingdir, + env, + secrets, + ): Promise; + cleanupWorkflow(buildParameters, branchName, secrets): Promise; + garbageCollect(filter, previewOnly, olderThan, fullCache, baseDependencies): Promise; + listResources(): Promise; + listWorkflow(): Promise; + watchWorkflow(): Promise; +} +``` + +Each provider handles `runTaskInWorkflow()` differently: + +| Provider | How it runs the build | +| -------------- | -------------------------------------------------------------- | +| `aws` | Creates a CloudFormation job stack, starts an ECS Fargate task | +| `k8s` | Creates a PVC, Kubernetes Secret, and Job; streams pod logs | +| `local-docker` | Runs a Docker container with volume mounts | +| `local` | Executes shell commands directly on the host | + +#### Custom Provider Loading + +When `providerStrategy` doesn't match a built-in name, the provider loader: + +1. Parses the source (GitHub URL, NPM package, or local path) via `ProviderUrlParser` +2. Clones or installs the module via `ProviderGitManager` +3. Validates that all 7 interface methods exist +4. Falls back to the local provider if loading fails + +See [Custom Providers](../providers/custom-providers) for the user-facing guide. + +### Workflow System + +The `WorkflowCompositionRoot` selects which workflow to run: + +``` + asyncOrchestrator: true? + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ” + โ”‚ Yes โ”‚โ”€โ”€โ–บ AsyncWorkflow + โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ Dispatches build to cloud container + โ”‚ and returns immediately + โ”Œโ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ” + โ”‚ No โ”‚ + โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ + โ”‚ + customJob set? + โ”‚ + โ”Œโ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ” + โ”‚ Yes โ”‚โ”€โ”€โ–บ CustomWorkflow + โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”˜ Parses YAML job definition + โ”‚ and runs container steps + โ”Œโ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ” + โ”‚ No โ”‚โ”€โ”€โ–บ BuildAutomationWorkflow + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ Standard build pipeline +``` + +**BuildAutomationWorkflow** generates a shell script that runs inside the container. The script: + +1. Installs toolchain (Node.js, npm, yarn, git-lfs) for remote providers +2. Clones game-ci/unity-builder into the container +3. Runs `remote-cli-pre-build` (restores caches, clones the target repo) +4. Executes the Unity build via the standard Game CI entrypoint +5. Runs `remote-cli-post-build` (pushes caches) +6. Writes log markers for collection + +**AsyncWorkflow** runs the entire build inside a cloud container. It installs the AWS CLI, clones +both the builder and target repos, and executes `index.js -m async-workflow`. The calling GitHub +Action returns immediately. Progress is reported via +[GitHub Checks](../providers/github-integration). + +### Hook System + +Orchestrator has two hook mechanisms: + +**Command Hooks** โ€” Shell commands injected before or after the setup and build steps. Defined via +the `customCommandHooks` YAML parameter or as files in `.game-ci/command-hooks/`. + +**Container Hooks** โ€” Separate Docker containers that run before or after the build. Defined via +`containerHookFiles` (built-in names like `aws-s3-upload-build`) or `preBuildSteps` / +`postBuildSteps` YAML. Each hook specifies an image, commands, and optional secrets. + +See [Hooks](hooks/container-hooks) for the full guide. + +### Configuration Resolution + +Orchestrator reads configuration from multiple sources with this priority: + +``` + Highest Priority + โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” + โ”‚ GitHub Action inputs โ”‚ with: providerStrategy: aws + โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค + โ”‚ CLI flags โ”‚ --providerStrategy aws + โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค + โ”‚ Query overrides โ”‚ Pull Secrets from external sources + โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค + โ”‚ Environment variables โ”‚ PROVIDER_STRATEGY=aws + โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + Lowest Priority +``` + +The `OrchestratorOptions` class handles this resolution. Environment variables accept both +`camelCase` and `UPPER_SNAKE_CASE` formats. + +### Remote Client + +The remote client runs **inside** the build container (not on the CI runner). It provides two CLI +modes: + +- **`remote-cli-pre-build`** โ€” Called before the Unity build. Handles git clone, LFS pull, cache + restoration, retained workspace setup, large package optimization, and custom hook execution. +- **`remote-cli-post-build`** โ€” Called after the Unity build. Pushes the Library and build caches to + cloud storage. + +### GitHub Integration + +The `GitHub` class manages GitHub Checks and async workflow dispatch: + +- **`createGitHubCheck()`** โ€” Creates a check run on the commit via the GitHub API. +- **`updateGitHubCheck()`** โ€” Updates check status. In async environments, updates are routed + through the `Async Checks API` workflow (since containers can't call the Checks API directly). +- **`runUpdateAsyncChecksWorkflow()`** โ€” Triggers a GitHub Actions workflow that updates the check + run on behalf of the container. + +### Caching and Storage + +Caching is split between the remote client (push/pull logic) and the storage provider (S3 or +rclone): + +- **Standard caching** โ€” Archives the `Library/` folder and LFS files as `.tar.lz4` archives. +- **Retained workspaces** โ€” Keeps the entire project folder. Uses distributed locking via S3 or + rclone to prevent concurrent access. + +See [Storage](storage) for the full breakdown of file categories, compression, and storage backends. + +## CLI Modes + +The CLI system uses a decorator-based registry (`@CliFunction`). Each mode maps to a static method: + +| Mode | Description | +| ----------------------- | ---------------------------------------------------- | +| `cli-build` | Full build workflow (default) | +| `async-workflow` | Async build execution (called from within container) | +| `remote-cli-pre-build` | Pre-build setup (runs inside container) | +| `remote-cli-post-build` | Post-build cache push (runs inside container) | +| `remote-cli-log-stream` | Pipe and capture build logs | +| `checks-update` | Update GitHub Checks from async container | +| `cache-push` | Push a directory to cache storage | +| `cache-pull` | Pull a directory from cache storage | +| `garbage-collect` | Clean up old cloud resources | +| `list-resources` | List active cloud resources | +| `list-workflow` | List running workflows | +| `watch` | Follow logs of a running workflow | +| `hash` | Hash folder contents recursively | +| `print-input` | Print all resolved parameters | + +## Source Code Map + +``` +src/model/orchestrator/ +โ”œโ”€โ”€ orchestrator.ts # Main coordinator +โ”œโ”€โ”€ options/ +โ”‚ โ”œโ”€โ”€ orchestrator-options.ts # Configuration resolution +โ”‚ โ””โ”€โ”€ orchestrator-folders.ts # Path management (/data/...) +โ”œโ”€โ”€ workflows/ +โ”‚ โ”œโ”€โ”€ workflow-composition-root.ts # Workflow selection +โ”‚ โ”œโ”€โ”€ build-automation-workflow.ts # Standard build pipeline +โ”‚ โ”œโ”€โ”€ async-workflow.ts # Async dispatch +โ”‚ โ””โ”€โ”€ custom-workflow.ts # Custom job execution +โ”œโ”€โ”€ providers/ +โ”‚ โ”œโ”€โ”€ provider-interface.ts # 7-method contract +โ”‚ โ”œโ”€โ”€ provider-loader.ts # Dynamic plugin loading +โ”‚ โ”œโ”€โ”€ provider-url-parser.ts # GitHub/NPM/local parsing +โ”‚ โ”œโ”€โ”€ provider-git-manager.ts # Clone and cache repos +โ”‚ โ”œโ”€โ”€ aws/ # AWS Fargate provider +โ”‚ โ”œโ”€โ”€ k8s/ # Kubernetes provider +โ”‚ โ”œโ”€โ”€ docker/ # Local Docker provider +โ”‚ โ””โ”€โ”€ local/ # Direct execution provider +โ”œโ”€โ”€ services/ +โ”‚ โ”œโ”€โ”€ core/ +โ”‚ โ”‚ โ”œโ”€โ”€ orchestrator-logger.ts +โ”‚ โ”‚ โ”œโ”€โ”€ orchestrator-system.ts # Shell command execution +โ”‚ โ”‚ โ”œโ”€โ”€ shared-workspace-locking.ts +โ”‚ โ”‚ โ””โ”€โ”€ follow-log-stream-service.ts +โ”‚ โ”œโ”€โ”€ hooks/ +โ”‚ โ”‚ โ”œโ”€โ”€ command-hook-service.ts +โ”‚ โ”‚ โ””โ”€โ”€ container-hook-service.ts +โ”‚ โ””โ”€โ”€ cache/ +โ”‚ โ””โ”€โ”€ local-cache-service.ts +โ”œโ”€โ”€ remote-client/ +โ”‚ โ”œโ”€โ”€ index.ts # Pre/post build logic +โ”‚ โ””โ”€โ”€ caching.ts # Cache push/pull with LZ4 +โ””โ”€โ”€ tests/ + +src/model/cli/ +โ”œโ”€โ”€ cli.ts # CLI entry point +โ””โ”€โ”€ cli-functions-repository.ts # @CliFunction registry + +src/model/github.ts # GitHub Checks + async dispatch +``` diff --git a/docs/03-github-orchestrator/07-advanced-topics/10-lfs-agents.mdx b/docs/03-github-orchestrator/07-advanced-topics/10-lfs-agents.mdx new file mode 100644 index 00000000..c28adaa9 --- /dev/null +++ b/docs/03-github-orchestrator/07-advanced-topics/10-lfs-agents.mdx @@ -0,0 +1,92 @@ +# Custom LFS Agents + +Orchestrator supports custom Git LFS transfer agents โ€” external executables that handle +LFS upload and download instead of the default HTTPS transport. + +## elastic-git-storage (Built-in) + +[elastic-git-storage](https://github.com/frostebite/elastic-git-storage) is a custom Git LFS +transfer agent with first-class support in Orchestrator. It supports multiple storage backends: +local filesystem, WebDAV, and rclone remotes. + +When you set `lfsTransferAgent: elastic-git-storage`, Orchestrator will: + +1. Search PATH and known locations for an existing installation +2. If not found, download the correct platform binary from GitHub releases +3. Configure it as the standalone LFS transfer agent via `git config` + +### Basic Usage + +```yaml +- uses: game-ci/unity-builder@v4 + with: + targetPlatform: StandaloneLinux64 + lfsTransferAgent: elastic-git-storage + lfsStoragePaths: '/mnt/lfs-cache' +``` + +### Version Pinning + +Append `@version` to pin a specific release: + +```yaml +lfsTransferAgent: elastic-git-storage@v1.0.0 +``` + +Without a version suffix, the latest release is downloaded. + +### Multiple Storage Backends + +`lfsStoragePaths` accepts semicolon-separated paths. The agent tries each in order: + +```yaml +lfsStoragePaths: '/mnt/fast-ssd;webdav://lfs.example.com/storage;rclone://remote:lfs-bucket' +``` + +### Agent Arguments + +Pass additional flags via `lfsTransferAgentArgs`: + +```yaml +lfsTransferAgent: elastic-git-storage +lfsTransferAgentArgs: '--verbose --concurrency 4' +lfsStoragePaths: '/mnt/lfs-cache' +``` + +## Custom Agents + +Any Git LFS custom transfer agent can be used. Provide the path to the executable: + +```yaml +lfsTransferAgent: /usr/local/bin/lfs-folderstore +lfsTransferAgentArgs: '-dir /mnt/lfs-store' +``` + +Orchestrator configures the agent via: + +``` +git config lfs.customtransfer..path +git config lfs.customtransfer..args +git config lfs.standalonetransferagent +``` + +The agent name is derived from the executable filename (without extension). + +## Storage Paths + +The `lfsStoragePaths` input sets the `LFS_STORAGE_PATHS` environment variable. How these paths +are interpreted depends on the agent: + +| Agent | Path format | +|-------|------------| +| elastic-git-storage | Local paths, `webdav://` URLs, `rclone://` remotes | +| lfs-folderstore | Local directory paths | +| Custom | Agent-specific | + +## Inputs Reference + +| Input | Description | +|-------|-------------| +| `lfsTransferAgent` | Agent name (e.g., `elastic-git-storage`) or path to executable. Append `@version` for release pinning. | +| `lfsTransferAgentArgs` | Additional arguments passed to the agent | +| `lfsStoragePaths` | Semicolon-separated storage paths (set as `LFS_STORAGE_PATHS` env var) | diff --git a/docs/03-github-orchestrator/06-advanced-topics/_category_.yaml b/docs/03-github-orchestrator/07-advanced-topics/_category_.yaml similarity index 81% rename from docs/03-github-orchestrator/06-advanced-topics/_category_.yaml rename to docs/03-github-orchestrator/07-advanced-topics/_category_.yaml index da411968..24f7ad99 100644 --- a/docs/03-github-orchestrator/06-advanced-topics/_category_.yaml +++ b/docs/03-github-orchestrator/07-advanced-topics/_category_.yaml @@ -1,5 +1,5 @@ --- -position: 5.0 +position: 7.0 label: Advanced topics collapsible: true collapsed: true diff --git a/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx b/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx deleted file mode 100644 index 70e5079e..00000000 --- a/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx +++ /dev/null @@ -1,3 +0,0 @@ -# GitLab Introduction - -You can use GitLab with Orchestrator via the Command Line mode. diff --git a/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml b/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml deleted file mode 100644 index 0407bcd5..00000000 --- a/versioned_docs/version-2/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml +++ /dev/null @@ -1,5 +0,0 @@ ---- -position: 9.0 -label: GitLab Integration -collapsible: true -collapsed: true diff --git a/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx b/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx deleted file mode 100644 index 70e5079e..00000000 --- a/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/01-introduction-gitlab.mdx +++ /dev/null @@ -1,3 +0,0 @@ -# GitLab Introduction - -You can use GitLab with Orchestrator via the Command Line mode. diff --git a/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml b/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml deleted file mode 100644 index 0407bcd5..00000000 --- a/versioned_docs/version-3/03-github-orchestrator/06-advanced-topics/09-gitlab/_category_.yaml +++ /dev/null @@ -1,5 +0,0 @@ ---- -position: 9.0 -label: GitLab Integration -collapsible: true -collapsed: true