pypeline - Define Your CI/CD Pipeline Once, Run It Anywhere

Pypeline lets you define your build, test, and deployment pipeline in a single YAML file and run it consistently across your local development environment and any CI/CD platform (GitHub Actions, Jenkins, etc.). No more platform-specific configurations – write once, run anywhere.

Key Features

• Unified Pipeline Definition: Users can define their entire pipeline in a single YAML file, eliminating the need to switch between different syntaxes and configurations for different CI/CD tools.

• Extensibility: Pypeline supports execution steps defined not only through installed Python packages but also from local scripts.

• Execution Context: Allow sharing information and state between steps. Each step in the pipeline receives an execution context that can be updated during step execution.

• Dependency Handling: Every step can register its dependencies and will only be scheduled if anything has changed.

• Use as a Library: PipelineConfig and PipelineLoader[T] are generic — define your own step base class and use pypeline's infrastructure (config loading, step resolution, shared context) to build custom pipelines for any domain, not just CI/CD. See Use Pypeline as a Library.

Installation

Use pipx (or your favorite package manager) to install and run it in an isolated environment:

pipx install pypeline-runner

This will install the pypeline command globally, which you can use to run your pipelines.

Note

The Python package is called pypeline-runner because the name pypeline was already taken on PyPI. The command-line interface is pypeline.

Documentation: pypeline-runner.readthedocs.io

Quick Start

pipx install pypeline-runner
pypeline init --project-dir my-pipeline
cd my-pipeline
pypeline run

Walkthrough: Getting Started with Pypeline

To get started run the init command to create a sample project:

pypeline init --project-dir my-pipeline

The example project pipeline is defined in the pipeline.yaml file.

pipeline:
  - step: CreateVEnv
    module: pypeline.steps.create_venv
    config:
      bootstrap_script: .bootstrap/bootstrap.py
  - step: WestInstall
    module: pypeline.steps.west_install
    description: Download external modules
  - step: MyStep
    file: steps/my_step.py
    description: Run a custom script

This pipeline consists of three steps:

• CreateVEnv: This is a built-in step that creates a Python virtual environment.
• WestInstall: This is a built-in step that downloads external modules using the west tool.
• MyStep: This is a custom step that runs a script defined in the steps/my_step.py file.

You can run the pipeline using the run command:

pypeline run --project-dir my-pipeline

Contributing

The project uses Poetry for dependencies management and packaging. You can set up the development environment using one of the following methods:

Option 1: Using Poetry directly (minimal setup)

poetry install

Option 2: Using pypeline-runner (runs the full pipeline including tests and checks)

pipx install pypeline-runner
pypeline run

To execute the test suite, call pytest inside Poetry's virtual environment via poetry run:

poetry run pytest

For those using VS Code there are tasks defined for the most common commands:

• run tests
• run all checks configured for pre-commit
• generate documentation

See the .vscode/tasks.json for more details.

This repository uses commitlint for checking if the commit message meets the conventional commit format.

Credits

This package was created with Copier and the browniebroke/pypackage-template project template.