Skip to content

WORKFLOW.md

Latest commit

 

History

History
168 lines (127 loc) · 5.54 KB

WORKFLOW.md

File metadata and controls

168 lines (127 loc) · 5.54 KB

Build System Workflow Documentation

Overview

This system uses GitHub Actions workflows running on the main branch to orchestrate builds across multiple build branches. Each build cycle runs on its own dedicated branch.

Architecture

Main Branch

  • Contains: Workflow definitions and scripts
  • Runs: Scheduled dispatcher that triggers builds on active build branches
  • Never builds packages: Only orchestrates builds

Build Branches

  • Named: build/YYYY-MM-DD-HHMMSS-container-tag (e.g., build/2025-10-31-143027-bioconductor-release321)
  • Contains: Build state, logs, and results for a single build cycle
  • Runs: Package build workflows triggered by the dispatcher on main

Workflows

1. dispatch_builds.yaml (Main Branch Only)

Purpose: Runs on schedule on main branch, dispatches builds to incomplete build branches

Schedule: Every 5 minutes (configurable via cron)

Process:

  1. Fetches all build/* branches
  2. Checks each branch for completion (presence of PACKAGES file)
  3. Dispatches run.yaml workflow to incomplete branches
  4. Skips branches that have completed

Triggers:

  • schedule: Every 5 minutes
  • workflow_dispatch: Manual trigger

2. start_cycle.yaml (Main Branch)

Purpose: Creates a new build branch and starts the build cycle

Process:

  1. Creates new branch from main: build/YYYY-MM-DD-HHMMSS-container-tag
  2. Stores container image in CONTAINER_BASE_IMAGE.bioc
  3. Initializes empty state files (biocdeps.json, logs, etc.)
  4. Triggers initial build on the new branch

Triggers:

  • workflow_dispatch: Manual trigger with container image input

Inputs:

  • container_image: Full container image path (e.g., ghcr.io/bioconductor/bioconductor_docker:RELEASE_3_21)

3. run.yaml (Build Branches)

Purpose: Builds packages on a build branch

Process:

  1. First run: Sets up Kubernetes resources and dependencies
  2. Subsequent runs:
    • Processes completed jobs
    • Finds ready packages
    • Dispatches new build jobs
  3. Commits state changes (logs, counters, etc.)

Triggers:

  • workflow_dispatch: Dispatched by dispatch_builds.yaml or manual
  • push: When logs are updated (triggers next iteration)

4. finish_cycle.yaml (Build Branches)

Purpose: Finalizes a build cycle

Process:

  1. Creates PACKAGES index file
  2. Syncs binaries to final storage via rclone
  3. Marks cycle as complete

Triggers:

  • workflow_dispatch: Manual trigger when ready to finalize

Build Lifecycle

Starting a Build

# On GitHub: Go to Actions → "Start New Build Cycle" → Run workflow
# Provide container image, e.g.: ghcr.io/bioconductor/bioconductor_docker:RELEASE_3_21

Automatic Building

  • The dispatch_builds.yaml workflow on main runs every 5 minutes
  • It automatically dispatches builds to any incomplete build branches
  • No manual intervention needed

Monitoring Progress

# Checkout the build branch
git checkout build/2025-10-31-143027-bioconductor-release321

# Check logs
cat logs/successful-packages.txt
cat logs/failed-packages.txt

# Check remaining packages
cat remaining-packages.json

# View README for detailed status (generated by handle_k8s_jobs.sh)
cat README.md

Finishing a Build

# On GitHub: Switch to build branch → Actions → "Finish Build Cycle" → Run workflow

This creates the PACKAGES file and syncs to final storage. The dispatcher will stop triggering builds on this branch once it sees the PACKAGES file.

Key Files on Build Branches

  • CONTAINER_BASE_IMAGE.bioc: Container image used for this build
  • biocdeps.json: Package dependencies (created by setup_k8s.sh)
  • uniquedeps.json: Unique dependencies
  • ready_packages.txt: Packages ready to build
  • remaining-packages.json: Packages still to build
  • logs/dispatched-packages.txt: Packages that have been dispatched
  • logs/successful-packages.txt: Successfully built packages
  • logs/failed-packages.txt: Failed packages
  • null_push_counter: Tracks workflow runs with no changes
  • reset_attempts_counter: Tracks reset attempts
  • PACKAGES: Created when cycle is complete (marks branch as done)
  • README.md: Build status report with BBS comparison

Advantages of This Architecture

  1. Schedules work on main: GitHub only allows scheduled workflows on default branches
  2. Parallel builds: Multiple build branches can run simultaneously for different containers
  3. Branch isolation: Each build cycle is isolated in its own branch
  4. Simple cleanup: Delete build branch when done
  5. Easy monitoring: Check branch state for build progress
  6. Automatic orchestration: Dispatcher handles all active builds centrally
  7. No workflow editing: Build branches don't need modified workflows

Cleanup

After a build completes and you've verified the results:

# Delete the build branch
git push origin --delete build/2025-10-31-143027-bioconductor-release321

Troubleshooting

Build branch not getting dispatched

  • Check that PACKAGES file doesn't exist on the branch (if it exists, branch is marked complete)
  • Verify dispatch_builds.yaml is running on main branch (check Actions tab)

Manual dispatch to specific branch

# On GitHub: Switch to build branch → Actions → "Build R Packages" → Run workflow

Reset a stuck build

# Checkout the build branch
git checkout build/2025-10-31-143027-bioconductor-release321

# Clear dispatch logs to retry packages
> logs/dispatched-packages.txt
git add logs/dispatched-packages.txt
git commit -m "Reset dispatched packages"
git push