Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions jenkins-scripts/pr_job_workflow.md → docs/pr_job_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ The process begins when a developer opens a pull request on a Gazebo repository,

The multitude of Jenkins jobs for every library, collection, and platform are not configured manually. They are generated programmatically using Jenkins Job DSL, which is a feature that allows you to define jobs in code.

- **`gazebo_libs.dsl`**: The heart of the job generation is the Groovy script [jenkins-scripts/dsl/gazebo_libs.dsl](dsl/gazebo_libs.dsl). This script runs as part of a "seed" job in Jenkins and is responsible for creating all the other CI jobs.
- **`gz-collections.yaml`**: The DSL script is powered by a single source of truth: [jenkins-scripts/dsl/gz-collections.yaml](dsl/gz-collections.yaml). This YAML file defines the entire Gazebo ecosystem for the CI system, including all collections, the libraries within them, their major versions, and their corresponding branches.
- **`gazebo_libs.dsl`**: The heart of the job generation is the Groovy script [jenkins-scripts/dsl/gazebo_libs.dsl](../jenkins-scripts/dsl/gazebo_libs.dsl). This script runs as part of a "seed" job in Jenkins and is responsible for creating all the other CI jobs.
- **`gz-collections.yaml`**: The DSL script is powered by a single source of truth: [jenkins-scripts/dsl/gz-collections.yaml](../jenkins-scripts/dsl/gz-collections.yaml). This YAML file defines the entire Gazebo ecosystem for the CI system, including all collections, the libraries within them, their major versions, and their corresponding branches.

For our `gz-math` example in the `jetty` collection, the YAML looks like this:
```yaml
Expand Down Expand Up @@ -45,9 +45,9 @@ The DSL script parses this file and generates a job for each library and CI conf

A critical part of the process is ensuring a pull request for a specific library branch only triggers the correct set of jobs. This matching logic is explicitly coded in DSL helper scripts.

- **`*GitHub.groovy` Helper Scripts:** The logic is contained in files within `jenkins-scripts/dsl/_configs_/`, particularly `GenericAnyJobGitHub.groovy`. These scripts configure the "GitHub Pull Request Builder" (ghprb) plugin for each job generated by the DSL.
- **`*GitHub.groovy` Helper Scripts:** The logic is contained in files within `jenkins-scripts/dsl/_configs_/`, particularly [GenericAnyJobGitHub.groovy](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy). These scripts configure the "GitHub Pull Request Builder" (ghprb) plugin for each job generated by the DSL.

- **The `whiteListTargetBranches` Logic:** The `GenericAnyJobGitHub.groovy` script contains the precise matching logic:
- **The `whiteListTargetBranches` Logic:** The [GenericAnyJobGitHub.groovy](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy) script contains the precise matching logic:
```groovy
// In GenericAnyJobGitHub.groovy
...
Expand All @@ -64,9 +64,9 @@ A critical part of the process is ensuring a pull request for a specific library

### The End-to-End Matching Flow

1. **Blueprint:** `gazebo_libs.dsl` reads `gz-collections.yaml` and sees that `gz-math` in the `jetty` collection uses the branch `gz-math9`.
1. **Blueprint:** [`gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl) reads [`gz-collections.yaml`](../jenkins-scripts/dsl/gz-collections.yaml) and sees that `gz-math` in the `jetty` collection uses the branch `gz-math9`.
2. **Job Generation:** The DSL script generates the CI jobs (e.g., `gz_math-ci-pr_any-noble-amd64`). When doing so, it calls the helper scripts and passes `gz-math9` as a `supported_branch`.
3. **Job Configuration:** The `GenericAnyJobGitHub.groovy` script configures the Jenkins job, adding `gz-math9` to its "White-listed target branches".
3. **Job Configuration:** The [`GenericAnyJobGitHub.groovy`](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy) script configures the Jenkins job, adding `gz-math9` to its "White-listed target branches".
4. **The Match:** When a developer opens a PR targeting the `gz-math9` branch, the ghprb plugin sees that the PR's target branch is in the job's whitelist, and **a match is made**, triggering the build. A PR targeting any other branch would be ignored by this job.

## 4. Dependency Resolution: A Platform-Specific Deep Dive
Expand All @@ -77,30 +77,30 @@ Once the correct job is triggered, it begins executing. This is where dependenci

The Linux strategy prioritizes speed by pre-installing all required `.deb` packages into a Docker image that is generated on-the-fly at the beginning of the build job.

1. **Script Chain:** The process follows a chain of scripts: `gz_math-compilation.bash` -> `generic-building-base.bash` -> `docker_generate_dockerfile.bash`.
1. **Script Chain:** The process follows a chain of scripts: [gz_math-compilation.bash](../jenkins-scripts/docker/lib/gz_math-compilation.bash) -> [generic-building-base.bash](../jenkins-scripts/docker/lib/generic-building-base.bash) -> [docker_generate_dockerfile.bash](../jenkins-scripts/docker/lib/docker_generate_dockerfile.bash).
2. **Dependency Aggregation:** The scripts collect dependency names from multiple sources into a master list (`$PACKAGES_CACHE_AND_CHECK_UPDATES`):
- **`packages.apt` files**: The build can also find `packages.apt` or `packages-<distro>.apt` files in the library's source code to include other required packages.
- **`BASE_DEPENDENCIES`**: A common list of packages needed for all builds.
3. **Dockerfile Generation:** The `docker_generate_dockerfile.bash` script writes a new `Dockerfile`, injecting the master package list into an `apt-get install` command.
4. **`build.sh` Generation:** The `_generic_linux_compilation_build.sh.bash` script writes a new `build.sh`, which is executed inside the docker container.
3. **Dockerfile Generation:** The [docker_generate_dockerfile.bash](../jenkins-scripts/docker/lib/docker_generate_dockerfile.bash) script writes a new `Dockerfile`, injecting the master package list into an `apt-get install` command.
4. **`build.sh` Generation:** The [_generic_linux_compilation_build.sh.bash](../jenkins-scripts/docker/lib/_generic_linux_compilation_build.sh.bash) script writes a new `build.sh`, which is executed inside the docker container.
5. **Build Execution:** The job builds this Docker image and runs the compilation inside the container, where all dependencies are already present.

Note that both the generated files `Dockerfile` and `build.sh` are stored as Jenkins artifacts and can be viewed on the jobs page (e.g. [gz_math-ci-pr_any-noble-amd64](https://build.osrfoundation.org/job/gz_math-ci-pr_any-noble-amd64/lastBuild/)).

### Windows: Building Dependencies from Source

The Windows strategy is a hybrid: it uses the `pixi` package manager and `conda-forge` packages for a baseline of third-party tools and then builds all Gazebo dependencies from source using `gazebodistro`. The process is orchestrated by `jenkins-scripts/lib/colcon-default-devel-windows.bat`.
The Windows strategy is a hybrid: it uses the `pixi` package manager and `conda-forge` packages for a baseline of third-party tools and then builds all Gazebo dependencies from source using `gazebodistro`. The process is orchestrated by [jenkins-scripts/lib/colcon-default-devel-windows.bat](../jenkins-scripts/lib/colcon-default-devel-windows.bat).

1. **Baseline via `pixi`/`conda`:** A `pixi` environment is created to install large, pre-built binary packages like Boost, OGRE, Qt, etc.
2. **Determine Target Version:** The script runs `tools/detect_cmake_major_version.py` on the PR's `CMakeLists.txt` to find the library's major version (e.g., `9`).
2. **Determine Target Version:** The script runs [tools/detect_cmake_major_version.py](../jenkins-scripts/tools/detect_cmake_major_version.py) on the PR's `CMakeLists.txt` to find the library's major version (e.g., `9`).
3. **Checkout Dependencies from `gazebodistro`:** A helper function uses `vcs import` to read the appropriate `.yaml` file from the `gazebodistro` repository (e.g., `gz-math9.yaml`). `vcs` then clones the source code for **all listed dependencies** into the local workspace.
4. **Build Workspace:** The `colcon build` command is used to compile the entire workspace from source.

### macOS: Pre-built Homebrew Packages ("Bottles")

The macOS strategy relies on the Homebrew package manager, preferring its pre-compiled binaries ("bottles"). This is orchestrated by [jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash](lib/project-default-devel-homebrew-amd64.bash).
The macOS strategy relies on the Homebrew package manager, preferring its pre-compiled binaries ("bottles"). This is orchestrated by [jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash](../jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash).

1. **Determine Formula Name:** The script uses `detect_cmake_major_version.py` to determine the correct Homebrew formula name (e.g., `gz-math9`).
1. **Determine Formula Name:** The script uses [`detect_cmake_major_version.py`](../jenkins-scripts/tools/detect_cmake_major_version.py) to determine the correct Homebrew formula name (e.g., `gz-math9`).
2. **Install Dependencies via Homebrew:** The script ensures the `osrf/simulation` Homebrew tap is active and then runs the central command:
```bash
brew install gz-math9 --only-dependencies
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ This document explains the continuous integration (CI) process for stable branch

## 1. Job Definition and Triggering

The jobs for stable branches are defined and generated using the same core infrastructure as the pull request (PR) jobs, but with important differences in naming and triggering. The entire process is orchestrated by `jenkins-scripts/dsl/gazebo_libs.dsl`, which reads its configuration from `jenkins-scripts/dsl/gz-collections.yaml`.
The jobs for stable branches are defined and generated using the same core infrastructure as the pull request (PR) jobs, but with important differences in naming and triggering. The entire process is orchestrated by [`jenkins-scripts/dsl/gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl), which reads its configuration from [`jenkins-scripts/dsl/gz-collections.yaml`](../jenkins-scripts/dsl/gz-collections.yaml).

### Job Generation

Within the `gazebo_libs.dsl` script, a specific block of code is responsible for creating stable branch jobs. It checks if the `stable_branches` category is enabled for a given CI configuration:
Within the [`gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl) script, a specific block of code is responsible for creating stable branch jobs. It checks if the `stable_branches` category is enabled for a given CI configuration:

```groovy
// In jenkins-scripts/dsl/gazebo_libs.dsl
Expand Down
9 changes: 8 additions & 1 deletion jenkins-scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

The release-tools repository uses the [DSL Jenkins plugin](https://plugins.jenkins.io/job-dsl/) to allow us to programmatically generate the job configuration (configuration as code). You can find the different job configs under the [`dsl`](./dsl/) folder.

## Workflow Documentation

For detailed explanations of how the CI/CD workflows operate, see:

- [Pull Request Build Workflow](../docs/pr_job_workflow.md) - End-to-end explanation of the PR CI process from trigger to build
- [Stable Branch CI Workflow](../docs/stable_branch_workflow.md) - Explanation of stable branch CI jobs and their triggers

## Conda local builder for Windows

### Prerequisites
Expand Down Expand Up @@ -42,7 +49,7 @@ git clone -b my-testing-branch C:\Users\foo\code\gz-math
python3 local_build.py gz_math-default-devel-windows-amd64.bat C:\Users\foo\code\gz-math
```

This command will run `gz_math-default-devel-windows-amd64.bat ` using the sources from `C:\User\foo\code\gz-math`. It will handle the installation of all the system dependencies
This command will run `gz_math-default-devel-windows-amd64.bat` using the sources from `C:\User\foo\code\gz-math`. It will handle the installation of all the system dependencies
using Pixi (it can take up to 10 minutes) and build all the Gazebo dependencies from source
using colcon. In a second build it builds gz-math with tests using colcon.

Expand Down
4 changes: 2 additions & 2 deletions release-repo-scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ first fork the `gz-rendering6-release` in GitHub and call the fork `gz-rendering
```bash
git clone https://github.com/gz-release/gz-rendering7-release
cd gz-rendering7-release
./path/to/release-tools/release-repo-scripts/bump_major_version 6 7
./path/to/release-tools/release-repo-scripts/bump_major_version.bash 6 7
```

### changelog_spawn.sh
Expand Down Expand Up @@ -141,7 +141,7 @@ change.

```bash
cd <path-to-release-repo>
./new_ubuntu_distribution <new_distro_name>
./new_ubuntu_distribution.bash <new_distro_name>
```

#### Example
Expand Down