diff --git a/jenkins-scripts/pr_job_workflow.md b/docs/pr_job_workflow.md similarity index 67% rename from jenkins-scripts/pr_job_workflow.md rename to docs/pr_job_workflow.md index 97137edec..60b913a24 100644 --- a/jenkins-scripts/pr_job_workflow.md +++ b/docs/pr_job_workflow.md @@ -15,8 +15,8 @@ The process begins when a developer opens a pull request on a Gazebo repository, The multitude of Jenkins jobs for every library, collection, and platform are not configured manually. They are generated programmatically using Jenkins Job DSL, which is a feature that allows you to define jobs in code. -- **`gazebo_libs.dsl`**: The heart of the job generation is the Groovy script [jenkins-scripts/dsl/gazebo_libs.dsl](dsl/gazebo_libs.dsl). This script runs as part of a "seed" job in Jenkins and is responsible for creating all the other CI jobs. -- **`gz-collections.yaml`**: The DSL script is powered by a single source of truth: [jenkins-scripts/dsl/gz-collections.yaml](dsl/gz-collections.yaml). This YAML file defines the entire Gazebo ecosystem for the CI system, including all collections, the libraries within them, their major versions, and their corresponding branches. +- **`gazebo_libs.dsl`**: The heart of the job generation is the Groovy script [jenkins-scripts/dsl/gazebo_libs.dsl](../jenkins-scripts/dsl/gazebo_libs.dsl). This script runs as part of a "seed" job in Jenkins and is responsible for creating all the other CI jobs. +- **`gz-collections.yaml`**: The DSL script is powered by a single source of truth: [jenkins-scripts/dsl/gz-collections.yaml](../jenkins-scripts/dsl/gz-collections.yaml). This YAML file defines the entire Gazebo ecosystem for the CI system, including all collections, the libraries within them, their major versions, and their corresponding branches. For our `gz-math` example in the `jetty` collection, the YAML looks like this: ```yaml @@ -45,9 +45,9 @@ The DSL script parses this file and generates a job for each library and CI conf A critical part of the process is ensuring a pull request for a specific library branch only triggers the correct set of jobs. This matching logic is explicitly coded in DSL helper scripts. -- **`*GitHub.groovy` Helper Scripts:** The logic is contained in files within `jenkins-scripts/dsl/_configs_/`, particularly `GenericAnyJobGitHub.groovy`. These scripts configure the "GitHub Pull Request Builder" (ghprb) plugin for each job generated by the DSL. +- **`*GitHub.groovy` Helper Scripts:** The logic is contained in files within `jenkins-scripts/dsl/_configs_/`, particularly [GenericAnyJobGitHub.groovy](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy). These scripts configure the "GitHub Pull Request Builder" (ghprb) plugin for each job generated by the DSL. -- **The `whiteListTargetBranches` Logic:** The `GenericAnyJobGitHub.groovy` script contains the precise matching logic: +- **The `whiteListTargetBranches` Logic:** The [GenericAnyJobGitHub.groovy](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy) script contains the precise matching logic: ```groovy // In GenericAnyJobGitHub.groovy ... @@ -64,9 +64,9 @@ A critical part of the process is ensuring a pull request for a specific library ### The End-to-End Matching Flow -1. **Blueprint:** `gazebo_libs.dsl` reads `gz-collections.yaml` and sees that `gz-math` in the `jetty` collection uses the branch `gz-math9`. +1. **Blueprint:** [`gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl) reads [`gz-collections.yaml`](../jenkins-scripts/dsl/gz-collections.yaml) and sees that `gz-math` in the `jetty` collection uses the branch `gz-math9`. 2. **Job Generation:** The DSL script generates the CI jobs (e.g., `gz_math-ci-pr_any-noble-amd64`). When doing so, it calls the helper scripts and passes `gz-math9` as a `supported_branch`. -3. **Job Configuration:** The `GenericAnyJobGitHub.groovy` script configures the Jenkins job, adding `gz-math9` to its "White-listed target branches". +3. **Job Configuration:** The [`GenericAnyJobGitHub.groovy`](../jenkins-scripts/dsl/_configs_/GenericAnyJobGitHub.groovy) script configures the Jenkins job, adding `gz-math9` to its "White-listed target branches". 4. **The Match:** When a developer opens a PR targeting the `gz-math9` branch, the ghprb plugin sees that the PR's target branch is in the job's whitelist, and **a match is made**, triggering the build. A PR targeting any other branch would be ignored by this job. ## 4. Dependency Resolution: A Platform-Specific Deep Dive @@ -77,30 +77,30 @@ Once the correct job is triggered, it begins executing. This is where dependenci The Linux strategy prioritizes speed by pre-installing all required `.deb` packages into a Docker image that is generated on-the-fly at the beginning of the build job. -1. **Script Chain:** The process follows a chain of scripts: `gz_math-compilation.bash` -> `generic-building-base.bash` -> `docker_generate_dockerfile.bash`. +1. **Script Chain:** The process follows a chain of scripts: [gz_math-compilation.bash](../jenkins-scripts/docker/lib/gz_math-compilation.bash) -> [generic-building-base.bash](../jenkins-scripts/docker/lib/generic-building-base.bash) -> [docker_generate_dockerfile.bash](../jenkins-scripts/docker/lib/docker_generate_dockerfile.bash). 2. **Dependency Aggregation:** The scripts collect dependency names from multiple sources into a master list (`$PACKAGES_CACHE_AND_CHECK_UPDATES`): - **`packages.apt` files**: The build can also find `packages.apt` or `packages-.apt` files in the library's source code to include other required packages. - **`BASE_DEPENDENCIES`**: A common list of packages needed for all builds. -3. **Dockerfile Generation:** The `docker_generate_dockerfile.bash` script writes a new `Dockerfile`, injecting the master package list into an `apt-get install` command. -4. **`build.sh` Generation:** The `_generic_linux_compilation_build.sh.bash` script writes a new `build.sh`, which is executed inside the docker container. +3. **Dockerfile Generation:** The [docker_generate_dockerfile.bash](../jenkins-scripts/docker/lib/docker_generate_dockerfile.bash) script writes a new `Dockerfile`, injecting the master package list into an `apt-get install` command. +4. **`build.sh` Generation:** The [_generic_linux_compilation_build.sh.bash](../jenkins-scripts/docker/lib/_generic_linux_compilation_build.sh.bash) script writes a new `build.sh`, which is executed inside the docker container. 5. **Build Execution:** The job builds this Docker image and runs the compilation inside the container, where all dependencies are already present. Note that both the generated files `Dockerfile` and `build.sh` are stored as Jenkins artifacts and can be viewed on the jobs page (e.g. [gz_math-ci-pr_any-noble-amd64](https://build.osrfoundation.org/job/gz_math-ci-pr_any-noble-amd64/lastBuild/)). ### Windows: Building Dependencies from Source -The Windows strategy is a hybrid: it uses the `pixi` package manager and `conda-forge` packages for a baseline of third-party tools and then builds all Gazebo dependencies from source using `gazebodistro`. The process is orchestrated by `jenkins-scripts/lib/colcon-default-devel-windows.bat`. +The Windows strategy is a hybrid: it uses the `pixi` package manager and `conda-forge` packages for a baseline of third-party tools and then builds all Gazebo dependencies from source using `gazebodistro`. The process is orchestrated by [jenkins-scripts/lib/colcon-default-devel-windows.bat](../jenkins-scripts/lib/colcon-default-devel-windows.bat). 1. **Baseline via `pixi`/`conda`:** A `pixi` environment is created to install large, pre-built binary packages like Boost, OGRE, Qt, etc. -2. **Determine Target Version:** The script runs `tools/detect_cmake_major_version.py` on the PR's `CMakeLists.txt` to find the library's major version (e.g., `9`). +2. **Determine Target Version:** The script runs [tools/detect_cmake_major_version.py](../jenkins-scripts/tools/detect_cmake_major_version.py) on the PR's `CMakeLists.txt` to find the library's major version (e.g., `9`). 3. **Checkout Dependencies from `gazebodistro`:** A helper function uses `vcs import` to read the appropriate `.yaml` file from the `gazebodistro` repository (e.g., `gz-math9.yaml`). `vcs` then clones the source code for **all listed dependencies** into the local workspace. 4. **Build Workspace:** The `colcon build` command is used to compile the entire workspace from source. ### macOS: Pre-built Homebrew Packages ("Bottles") -The macOS strategy relies on the Homebrew package manager, preferring its pre-compiled binaries ("bottles"). This is orchestrated by [jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash](lib/project-default-devel-homebrew-amd64.bash). +The macOS strategy relies on the Homebrew package manager, preferring its pre-compiled binaries ("bottles"). This is orchestrated by [jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash](../jenkins-scripts/lib/project-default-devel-homebrew-amd64.bash). -1. **Determine Formula Name:** The script uses `detect_cmake_major_version.py` to determine the correct Homebrew formula name (e.g., `gz-math9`). +1. **Determine Formula Name:** The script uses [`detect_cmake_major_version.py`](../jenkins-scripts/tools/detect_cmake_major_version.py) to determine the correct Homebrew formula name (e.g., `gz-math9`). 2. **Install Dependencies via Homebrew:** The script ensures the `osrf/simulation` Homebrew tap is active and then runs the central command: ```bash brew install gz-math9 --only-dependencies diff --git a/jenkins-scripts/stable_branch_workflow.md b/docs/stable_branch_workflow.md similarity index 84% rename from jenkins-scripts/stable_branch_workflow.md rename to docs/stable_branch_workflow.md index 1643f8ac0..b50c921c8 100644 --- a/jenkins-scripts/stable_branch_workflow.md +++ b/docs/stable_branch_workflow.md @@ -4,11 +4,11 @@ This document explains the continuous integration (CI) process for stable branch ## 1. Job Definition and Triggering -The jobs for stable branches are defined and generated using the same core infrastructure as the pull request (PR) jobs, but with important differences in naming and triggering. The entire process is orchestrated by `jenkins-scripts/dsl/gazebo_libs.dsl`, which reads its configuration from `jenkins-scripts/dsl/gz-collections.yaml`. +The jobs for stable branches are defined and generated using the same core infrastructure as the pull request (PR) jobs, but with important differences in naming and triggering. The entire process is orchestrated by [`jenkins-scripts/dsl/gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl), which reads its configuration from [`jenkins-scripts/dsl/gz-collections.yaml`](../jenkins-scripts/dsl/gz-collections.yaml). ### Job Generation -Within the `gazebo_libs.dsl` script, a specific block of code is responsible for creating stable branch jobs. It checks if the `stable_branches` category is enabled for a given CI configuration: +Within the [`gazebo_libs.dsl`](../jenkins-scripts/dsl/gazebo_libs.dsl) script, a specific block of code is responsible for creating stable branch jobs. It checks if the `stable_branches` category is enabled for a given CI configuration: ```groovy // In jenkins-scripts/dsl/gazebo_libs.dsl diff --git a/jenkins-scripts/README.md b/jenkins-scripts/README.md index 1a70f3acc..a8f96ec68 100644 --- a/jenkins-scripts/README.md +++ b/jenkins-scripts/README.md @@ -2,6 +2,13 @@ The release-tools repository uses the [DSL Jenkins plugin](https://plugins.jenkins.io/job-dsl/) to allow us to programmatically generate the job configuration (configuration as code). You can find the different job configs under the [`dsl`](./dsl/) folder. +## Workflow Documentation + +For detailed explanations of how the CI/CD workflows operate, see: + +- [Pull Request Build Workflow](../docs/pr_job_workflow.md) - End-to-end explanation of the PR CI process from trigger to build +- [Stable Branch CI Workflow](../docs/stable_branch_workflow.md) - Explanation of stable branch CI jobs and their triggers + ## Conda local builder for Windows ### Prerequisites @@ -42,7 +49,7 @@ git clone -b my-testing-branch C:\Users\foo\code\gz-math python3 local_build.py gz_math-default-devel-windows-amd64.bat C:\Users\foo\code\gz-math ``` -This command will run `gz_math-default-devel-windows-amd64.bat ` using the sources from `C:\User\foo\code\gz-math`. It will handle the installation of all the system dependencies +This command will run `gz_math-default-devel-windows-amd64.bat` using the sources from `C:\User\foo\code\gz-math`. It will handle the installation of all the system dependencies using Pixi (it can take up to 10 minutes) and build all the Gazebo dependencies from source using colcon. In a second build it builds gz-math with tests using colcon. diff --git a/release-repo-scripts/README.md b/release-repo-scripts/README.md index 28ee49ec4..927c0dcd7 100644 --- a/release-repo-scripts/README.md +++ b/release-repo-scripts/README.md @@ -41,7 +41,7 @@ first fork the `gz-rendering6-release` in GitHub and call the fork `gz-rendering ```bash git clone https://github.com/gz-release/gz-rendering7-release cd gz-rendering7-release -./path/to/release-tools/release-repo-scripts/bump_major_version 6 7 +./path/to/release-tools/release-repo-scripts/bump_major_version.bash 6 7 ``` ### changelog_spawn.sh @@ -141,7 +141,7 @@ change. ```bash cd -./new_ubuntu_distribution +./new_ubuntu_distribution.bash ``` #### Example