diff --git a/docs/dev_onborading.md b/docs/dev_onborading.md index a394beb4..0c2c241d 100644 --- a/docs/dev_onborading.md +++ b/docs/dev_onborading.md @@ -4,22 +4,21 @@ - **Goal**: Run and develop CI tests for **NFS-Ganesha** using the `sanity_dev` Jenkins job. - **Key pieces**: - - `jobs/distributed_test_runner/sanity_dev.groovy` – defines the Jenkins job & parameters. + - `jobs/jjb/dev_sanity.yaml` – defines the Jenkins job & parameters. - `jobs/Jenkinsfile.sanity_dev` – actual pipeline (checkout, install, run tests). - `tests/dev_space/` – dev-focused pytest suites (e.g. `test_fsal.py`). - `ci_utils/` – shared Python helpers. - `ci_utils/dev_space/` – dev-only helpers (dependencies, node reservation, etc.). -- **Sanity dev pipeline**: [`sanity_dev` Jenkins job](https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org/view/all/job/sanity_dev/) +- **Sanity dev pipeline**: [`dev-sanity-tests` Jenkins job](https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org/view/all/job/dev-sanity-tests/) --- ## How the Jenkins Job Works -- **Job definition (`sanity_dev.groovy`)** +- **Job definition (`dev_sanity.yaml`)** - **Defines parameters**: - **`TEST_SUITE`**: which test file to run (e.g. `test_fsal` → `test_fsal.py`). - **`SERVER_NODE_COUNT`**, **`CLIENT_NODE_COUNT`**: how many nodes to reserve. - - **`CI_REPO`/`CI_BRANCH`**: repo/branch for this CI framework (default is this repo). - **`GIT_REPO`/`GIT_BRANCH`**: NFS-Ganesha source to test. - **`CMAKE_FLAGS`**, **`CMAKE_OVERRIDE`**: extra/override CMake options. - **`CENTOS_VERSION`**, **`CENTOS_ARCH`**: OS/arch to use. @@ -72,7 +71,7 @@ Use this pattern (fixtures + managers from `ci_utils`) for any new dev tests. ## Typical Dev Workflow - **To run via Jenkins**: - - Open **`sanity_dev`** job. + - Open **`dev-sanity-tests`** job. - Set: - **`TEST_SUITE`**: e.g. `test_fsal`. - **Node counts**: `SERVER_NODE_COUNT`, `CLIENT_NODE_COUNT`. @@ -93,7 +92,7 @@ Use this pattern (fixtures + managers from `ci_utils`) for any new dev tests. - Reserve nodes, connect via SSH, setup backends, run tests. - **2. Wire it up in Jenkins** - - In `jobs/distributed_test_runner/sanity_dev.groovy`: + - In `jobs/jjb/dev_sanity.yaml`: - Add your test suite name to `TEST_SUITE` choices (e.g. `'test_newbackend'`). - In `jobs/Jenkinsfile.sanity_dev`: - Make sure `pytest` call matches the pattern: diff --git a/docs/github_actions_workflow.md b/docs/github_actions_workflow.md new file mode 100644 index 00000000..09e1b35f --- /dev/null +++ b/docs/github_actions_workflow.md @@ -0,0 +1,155 @@ +# GitHub Actions Workflow - Quick Guide + +## Overview + +This repository uses GitHub Actions to automatically validate and deploy Jenkins Job Builder (JJB) configurations. + +**Two Workflows:** +1. **Validation** (`test.yaml`) - Runs on Pull Requests +2. **Deployment** (`deploy.yaml`) - Runs on merge to `centos-ci` branch + +--- + +## Workflow Architecture + +![Workflow Diagram](../docs/images/github_actions_flow.png) + +**Simple Flow:** +``` +Developer → Pull Request → test.yaml (Validate) → Merge → deploy.yaml (Deploy) → Jenkins Updated +``` + +--- + +## 1. Validation Workflow (`test.yaml`) + +**Trigger:** Pull Request to any branch + +**What it does:** +- Builds Docker container with JJB +- Validates YAML syntax +- Checks macro references +- **Does NOT** connect to Jenkins + +**File:** `.github/workflows/test.yaml` + +```yaml +on: + pull_request: + branches: ['*'] +steps: + - Checkout code + - Build Docker image + - Run: jenkins-jobs test (validation only) +``` + +--- + +## 2. Deployment Workflow (`deploy.yaml`) + +**Trigger:** Push to `centos-ci` branch + +**What it does:** +- Builds Docker container with JJB +- Injects Jenkins API key +- Connects to Jenkins +- Creates/updates jobs + +**File:** `.github/workflows/deploy.yaml` + +```yaml +on: + push: + branches: ['centos-ci'] +steps: + - Checkout code + - Build Docker image + - Run: jenkins-jobs update (actual deployment) +``` + +--- + +## Supporting Files + +### `tests/Containerfile` +Docker image with JJB installed + +### `tests/verify-yaml.sh` +```bash +jenkins-jobs test globals/macros:jobs +``` +Validates syntax without Jenkins connection + +### `tests/deploy-nfs-ganesha-ci.sh` +```bash +sed "s/JENKINS_API_KEY/$JENKINS_API_KEY/g" tests/jenkins.conf > jobs/jenkins.ini +jenkins-jobs --conf jobs/jenkins.ini update jobs/:globals/macros/ +``` +Injects API key and deploys to Jenkins + +### `tests/jenkins.conf` +JJB configuration template with API key placeholder + +### `globals/macros/macros.yml` +Reusable JJB templates (SCM, parameters, triggers, etc.) + +### `jobs/jjb/*.yaml` +Jenkins job definitions using JJB format + +--- + +## How Files Work Together + +``` +Pull Request + ↓ +test.yaml → Containerfile → verify-yaml.sh → JJB (test mode) + ↓ +Validation Result → PR Status Check + +Merge to centos-ci + ↓ +deploy.yaml → Containerfile → deploy-nfs-ganesha-ci.sh + ↓ +jenkins.conf + API Key → JJB (update mode) + ↓ +macros.yml + jobs/jjb/*.yaml → Jenkins API + ↓ +Jobs Created/Updated in Jenkins +``` + +--- + +## Testing Locally + +```bash +# 1. Build container +docker build -t nfs-ganesha-ci-tests -f tests/Containerfile . + +# 2. Test validation (like test.yaml) +docker run --rm nfs-ganesha-ci-tests:latest + +# 3. Generate XML preview +docker run --rm -v $(pwd)/output:/output nfs-ganesha-ci-tests:latest sh -c \ + "jenkins-jobs test globals/macros:jobs/jjb -o /output --config-xml" + +# 4. View generated XML +cat output/*/config.xml + +# 5. Deploy to Jenkins (requires API key) +docker run --rm --env="JENKINS_API_KEY=your-token" \ + --entrypoint=tests/deploy-nfs-ganesha-ci.sh \ + nfs-ganesha-ci-tests:latest +``` + +--- + +## Quick Reference + +**Jenkins Server:** https://jenkins-nfs-ganesha.apps.ocp.cloud.ci.centos.org + +**Test locally:** `docker run --rm nfs-ganesha-ci-tests:latest` + +**Deploy:** Push to `centos-ci` branch + +**Monitor:** https://github.com/nfs-ganesha/ci-tests/actions \ No newline at end of file diff --git a/docs/images/github_actions_flow.png b/docs/images/github_actions_flow.png new file mode 100644 index 00000000..1203e0be Binary files /dev/null and b/docs/images/github_actions_flow.png differ diff --git a/globals/macros/macros.yml b/globals/macros/macros.yml index a8e18e8c..51af3972 100644 --- a/globals/macros/macros.yml +++ b/globals/macros/macros.yml @@ -8,7 +8,7 @@ skip-tag: true shallow-clone: true url: https://github.com/nfs-ganesha/ci-tests.git - basedir: "$WORKSPACE/ci-tests" + basedir: "ci-tests" - builder: name: get-node @@ -56,7 +56,7 @@ exclude-drafts: true exclude-no-code-change: true - comment-added-contains-event: - comment-contains-value: 'recheck {option}' + comment-contains-value: '(?i)recheck {option}' projects: - project-compare-type: PLAIN project-pattern: 'ffilz/nfs-ganesha' @@ -103,6 +103,10 @@ default: refs/heads/next description: Git reference to fetch from the Gerrit project. name: GERRIT_REFSPEC + - string: + name: GERRIT_USER + default: jenkins-glusterorg + description: Gerrit user to comment with - parameter: name: centos_variables diff --git a/jobs/Jenkinsfile.sanity_dev b/jobs/Jenkinsfile.sanity_dev index 7abeb225..9f729581 100644 --- a/jobs/Jenkinsfile.sanity_dev +++ b/jobs/Jenkinsfile.sanity_dev @@ -13,16 +13,6 @@ pipeline { stages { - stage('Checkout CI Tests') { - steps { - echo "Cloning CI repo: ${params.CI_REPO} (${params.CI_BRANCH})" - - dir('ci-tests') { - git url: params.CI_REPO, branch: params.CI_BRANCH - } - } - } - stage('Checkout NFS-Ganesha') { steps { echo "Cloning Ganesha repo: ${params.GIT_REPO} (${params.GIT_BRANCH})" diff --git a/jobs/distributed_test_runner/sanity_dev.groovy b/jobs/distributed_test_runner/sanity_dev.groovy deleted file mode 100644 index 993a4544..00000000 --- a/jobs/distributed_test_runner/sanity_dev.groovy +++ /dev/null @@ -1,93 +0,0 @@ -// ---------- PIPELINE JOB ---------- -pipelineJob('sanity_dev') { - - description('Dynamic CI Test Pipeline with parametrized test suite, topology, repos, and CMake options') - - parameters { - - // Test suite - choiceParam( - 'TEST_SUITE', - ['test_fsal'], - 'Select which test suite to run' - ) - - // Server nodes count — only allow 1 or 2 - choiceParam( - 'SERVER_NODE_COUNT', - ['1', '2'], - 'Number of server nodes to reserve' - ) - - // Client nodes count — only allow 1 or 2 - choiceParam( - 'CLIENT_NODE_COUNT', - ['1', '2'], - 'Number of client nodes to reserve' - ) - - // CI test repo + branch - stringParam( - 'CI_REPO', - 'https://github.com/nfs-ganesha/ci-tests.git', - 'Git repo containing CI test framework' - ) - stringParam( - 'CI_BRANCH', - 'centos-ci', - 'Branch for CI test framework' - ) - - // Ganesha repo + branch - stringParam( - 'GIT_REPO', - 'https://github.com/nfs-ganesha/nfs-ganesha.git', - 'NFS-Ganesha source repo' - ) - stringParam( - 'GIT_BRANCH', - 'next', - 'NFS-Ganesha branch' - ) - - // CMake - stringParam( - 'CMAKE_FLAGS', - '', - 'Extra CMake flags' - ) - booleanParam( - 'CMAKE_OVERRIDE', - false, - 'Override default CMake flags' - ) - - // OS Version - choiceParam( - 'CENTOS_VERSION', - ['9s', '10s'], - 'Select CentOS version' - ) - - choiceParam( - 'CENTOS_ARCH', - ['x86_64'], - 'Architecture' - ) - } - - definition { - cpsScm { - scm { - git { - remote { - // Static repo that stores ONLY the Jenkinsfiles - url('https://github.com/nfs-ganesha/ci-tests.git') - } - branch('centos-ci') - } - } - scriptPath('jobs/Jenkinsfile.sanity_dev') - } - } -} diff --git a/jobs/jjb/dev_sanity.yaml b/jobs/jjb/dev_sanity.yaml new file mode 100644 index 00000000..c97c254c --- /dev/null +++ b/jobs/jjb/dev_sanity.yaml @@ -0,0 +1,60 @@ +- job: + name: dev-sanity-tests + project-type: pipeline + description: | + Dynamic CI Test Pipeline with parametrized test suite, topology, repos, and CMake options + + properties: + - discarder + + parameters: + - centos_variables + + # Job-specific parameters + - choice: + name: TEST_SUITE + choices: + - test_fsal + description: Select which test suite to run + + - choice: + name: SERVER_NODE_COUNT + choices: + - '1' + - '2' + description: Number of server nodes to reserve + + - choice: + name: CLIENT_NODE_COUNT + choices: + - '1' + - '2' + description: Number of client nodes to reserve + + - string: + name: GIT_REPO + default: 'https://github.com/nfs-ganesha/nfs-ganesha.git' + description: NFS-Ganesha source repo + + - string: + name: GIT_BRANCH + default: 'next' + description: NFS-Ganesha branch + + - string: + name: CMAKE_FLAGS + default: '' + description: Custom cmake flags to append to the existing flags + + - bool: + name: CMAKE_OVERRIDE + default: false + description: > + If set, overrides existing cmake flags and only uses CMAKE_FLAGS. + If unset, CMAKE_FLAGS is appended to the existing flags. + + pipeline-scm: + scm: + - ci-tests + script-path: jobs/Jenkinsfile.sanity_dev + lightweight-checkout: true diff --git a/jobs/jjb/gatecheck.yaml b/jobs/jjb/gatecheck.yaml new file mode 100644 index 00000000..3787c068 --- /dev/null +++ b/jobs/jjb/gatecheck.yaml @@ -0,0 +1,39 @@ +- job: + name: nfs-ganesha-pipeline + project-type: pipeline + description: | + Automated GerritHub patch validation pipeline. Runs staged checks for every new patchset: + - Code checks (Checkpatch, Clang) + - FSAL tests (CephFS, GPFS, VFS, RGW) in parallel + - Functional tests (Cthon, Pynfs variants) in parallel + Posts consolidated results to Gerrit with links to full logs. + + properties: + - discarder + + parameters: + - gerrit_variables + - centos_variables + + # Job-specific parameters + - string: + name: CMAKE_FLAGS + default: '' + description: Custom cmake flags to append to the existing flags + + - bool: + name: CMAKE_OVERRIDE + default: false + description: > + If set, overrides existing cmake flags and only uses CMAKE_FLAGS. + If unset, CMAKE_FLAGS is appended to the existing flags. + + triggers: + - gerrithub-trigger: + option: gatecheck + + pipeline-scm: + scm: + - ci-tests + script-path: jobs/Jenkinsfile.gatecheck + lightweight-checkout: true diff --git a/jobs/jjb/views.yaml b/jobs/jjb/views.yaml new file mode 100644 index 00000000..88dccb13 --- /dev/null +++ b/jobs/jjb/views.yaml @@ -0,0 +1,82 @@ +- view: + name: Gatecheck Tests + view-type: list + description: 'Jobs related to patchset testing and validation' + job-name: + - gatecheck-trigger-gerrithub + columns: + - status + - weather + - job + - last-success + - last-failure + - last-duration + - build-button + - favorite + recurse: false + +- view: + name: Dev Tests + view-type: list + description: 'Development and testing jobs' + regex: 'dev-.*' + columns: + - status + - weather + - job + - last-success + - last-failure + - last-duration + - build-button + - favorite + recurse: false + + +- view: + name: Ceph Tests + view-type: list + description: 'Ceph and CephFS related test jobs' + regex: '.*ceph.*' + columns: + - status + - weather + - job + - last-success + - last-failure + - last-duration + - build-button + - favorite + recurse: false + +- view: + name: GPFS Tests + view-type: list + description: 'GPFS and Storage Scale related test jobs' + regex: '.*gpfs.*' + columns: + - status + - weather + - job + - last-success + - last-failure + - last-duration + - build-button + - favorite + recurse: false + + +- view: + name: WIP (Test only) + view-type: list + description: 'Work in progress and test-only jobs' + regex: 'test-.*' + columns: + - status + - weather + - job + - last-success + - last-failure + - last-duration + - build-button + - favorite + recurse: false diff --git a/tests/dev_space/test_fsal.py b/tests/dev_space/test_fsal.py index 34f73470..03026fa0 100644 --- a/tests/dev_space/test_fsal.py +++ b/tests/dev_space/test_fsal.py @@ -24,7 +24,9 @@ "CENTOS_VERSION", "CENTOS_ARCH", ] -NFS_GANESHA_REPO = "/tmp/workspace/sanity_dev/nfs-ganesha" +# Get the current working directory as WORKSPACE +WORKSPACE = os.getcwd() +NFS_GANESHA_REPO = os.path.join(WORKSPACE, "nfs-ganesha") @pytest.fixture(scope="session", autouse=True) def ci_params(): diff --git a/tests/jenkins.conf b/tests/jenkins.conf index 70b85014..ce4fe72b 100644 --- a/tests/jenkins.conf +++ b/tests/jenkins.conf @@ -1,7 +1,7 @@ [job_builder] ignore_cache=False keep_descriptions=False -recursive=False +recursive=True exclude=.*:manual:./development allow_duplicates=False #query_plugins_info=False diff --git a/tests/verify-yaml.sh b/tests/verify-yaml.sh index 48d37ede..7f658a10 100755 --- a/tests/verify-yaml.sh +++ b/tests/verify-yaml.sh @@ -1,3 +1,3 @@ #!/bin/bash -jenkins-jobs test globals/macros:jobs +jenkins-jobs test globals/macros:jobs --recursive