Contributing to Flyte 2

Thank you for your interest in contributing to Flyte 2! This guide will help you understand the contribution workflow, testing requirements, and release process.

Getting Started
Development Workflow
Making Changes
Testing and Verification
Submitting Changes
Release Process
Best Practices

Getting Started

Prerequisites

Before contributing, ensure you have:

Buf CLI installed
Go 1.26.3 or later
Node.js and npm (for TypeScript)
Python 3.10+ with uv package manager
Rust toolchain (if working with Rust bindings)
Git configured with your name and email
Docker (for building and running the devbox image)

Setting Up Your Environment

Fork the repository on GitHub

Clone your fork:

git clone https://github.com/YOUR_USERNAME/flyte.git
cd flyte/flyte

Add upstream remote:

git remote add upstream https://github.com/flyteorg/flyte.git

Verify your setup:

make docker-pull   # Pull the docker image for generation
make gen

Running Flyte Locally

The fastest way to run a full Flyte stack on your machine is the bundled devbox — a k3d-based Kubernetes cluster with all dependencies (TaskAction CRD, Knative, PostgreSQL, etc.) pre-installed — combined with a locally-running flyte-manager binary.

Start the Flyte Devbox

From the repo root:

# Build the devbox image (first time only, or after Dockerfile changes)
make devbox-build

# Start the devbox cluster in dev mode (required for running the manager locally)
make devbox-run FLYTE_DEV=true

# Stop the devbox when you're done
make devbox-stop

FLYTE_DEV=true is required when you intend to run the manager locally — it disables the in-cluster manager so your local process can take over. make devbox-run writes a kubeconfig pointing at the devbox cluster in global kubeconfig, so kubectl will target it automatically.

Build and Run the Manager

With the devbox running, start the manager locally:

# From the repo root
make -C manager run

# Or from manager/
make run

# Or build and run the binary directly
cd manager
make build
./bin/flyte-manager --config config.yaml

The manager will:

Connect to PostgreSQL and run database migrations
Start all services in parallel goroutines
Connect to your Kubernetes cluster
Begin reconciling TaskAction CRs

Configuration

Edit manager/config.yaml:

manager:
  # Single server port hosting all Connect services (Runs, Actions, DataProxy, Events, Cache, Secret, App).
  server:
    host: "0.0.0.0"
    port: 8090

  executor:
    healthProbePort: 8081

  kubernetes:
    namespace: "flyte"
    # Optional: specify custom kubeconfig path
    # kubeconfig: "/path/to/kubeconfig"

runs:
  storagePrefix: "s3://flyte-data"
  database:
    postgres:
      host: "localhost"
      port: 30001
      dbname: "runs"
      username: "postgres"
      password: "postgres"
      options: "sslmode=disable"

logger:
  level: 4  # Info level
  show-source: true

See manager/README.md for the full architecture, API endpoints, and troubleshooting tips.

Development Workflow

Creating a Feature Branch

Always create a new branch for your changes:

git checkout -b feature/your-feature-name

Use descriptive branch names:

feature/add-new-task-type - for new features
fix/workflow-literal-bug - for bug fixes
docs/improve-readme - for documentation
refactor/simplify-interface - for refactoring

Keeping Your Branch Updated

Regularly sync with upstream:

git fetch upstream
git rebase upstream/v2

CI Checks on Pull Requests from Forks

For security reasons, pull requests opened from a forked repository do not run the full CI suite that runs on branches in flyteorg/flyte. Fork PRs execute in an isolated context without access to repository secrets, so any job that builds or pushes an image — or comments on the PR — is skipped rather than run.

Checks that behave differently on fork PRs:

CI image build & push (build-ci-image.yml): skipped entirely — no image is built or pushed, and no build-status comment is posted to the PR.
Generated-code check (check-generate.yml): runs against the fallback base image instead of a PR-specific image, so it does not validate generated code against the changes in the PR.
Single-binary & devbox images (flyte-binary-v2.yml): both build-and-push jobs are skipped.

For contributors: this is expected — you don't need to do anything, and a skipped image job is not a failure. Run make gen and the checks under Testing and Verification locally to catch issues before requesting review.

For maintainers: treat a skipped or green image/generate check on a fork PR as "not run," not "passed." Before merging, validate the change under full CI by either checking out the contributor's branch locally and running the affected checks (e.g. make gen and confirming there is no diff), or pushing the branch to flyteorg/flyte (with the contributor's permission) to trigger the complete pipeline.

Making Changes

1. Modifying Protocol Buffers

When editing .proto files in flyteidl2/:

Naming Conventions:

Use snake_case for field names: task_id, execution_time
Use PascalCase for message names: TaskDefinition, WorkflowSpec
Use SCREAMING_SNAKE_CASE for enum values: TASK_STATE_RUNNING

Backward Compatibility:

Never change field numbers
Never change field types
Never remove required fields

Use reserved for removed fields:

message Example {
  reserved 2, 15, 9 to 11;
  reserved "old_field_name";
}

Documentation:

Add clear comments for all messages, fields, and enums
Include examples where helpful
Document any constraints or validation rules

Example:

// TaskDefinition defines the structure and configuration of a task.
message TaskDefinition {
  // Unique identifier for the task
  string task_id = 1;

  // Human-readable name for the task
  string name = 2;

  // Optional description explaining the task's purpose
  string description = 3;
}

2. Generate and Format Code

After making changes:

# Format proto files
make buf-format

# Lint proto files
make buf-lint

# Generate all language bindings
make buf

3. Update Language-Specific Utilities (if needed)

If you need to customize generation for a specific language, update files in flyteidl2/gen_utils/:

flyteidl2/gen_utils/go/ - Go-specific utilities
flyteidl2/gen_utils/ts/ - TypeScript utilities
flyteidl2/gen_utils/python/ - Python package configuration
flyteidl2/gen_utils/rust/ - Rust crate configuration

Testing and Verification

1. Lint and Format Checks

make buf-lint
make buf-format

All proto files must pass linting without errors.

2. Verify Generated Code Builds

Go:

make buf-go
make go-tidy
cd gen/go
go build ./...
go test ./...

TypeScript:

make buf-ts
cd gen/ts
npm install
npm run build  # if there's a build script

Python:

make buf-python
cd gen/python
uv lock
uv sync

Rust:

make buf-rust
make build-crate

3. Generate Mocks (Go)

If you've modified Go interfaces:

make mocks

4. Integration Testing

Test your changes in a downstream project that uses flyte:

Use replace directive in go.mod to point to your local changes
Generate code and verify it works as expected
Run integration tests in the consuming project

Submitting Changes

1. Commit Your Changes

Write clear, descriptive commit messages:

git add .
git commit -s -m "feat: add new task execution state

- Add TASK_STATE_CACHED for cached task results
- Update state transitions to support caching
- Add documentation for caching behavior

Fixes #123"

Commit message format:

Use conventional commits: feat:, fix:, docs:, refactor:, test:
Include -s flag to sign your commits (required)
Reference issue numbers with Fixes #123 or Closes #456

2. Push Your Changes

git push origin feature/your-feature-name

3. Create a Pull Request

Go to the Flyte repository
Click "New Pull Request"
Select your fork and branch
Target the v2 branch (not main)
Fill out the PR template with:
- Description of changes
- Motivation and context
- Testing performed
- Screenshots (if UI changes)
- Breaking changes (if any)

4. Code Review Process

Address reviewer feedback promptly
Push new commits to the same branch
Use git commit --amend for small fixes (then git push --force)
Engage in constructive discussion
Be patient - reviews may take time

Release Process

Releases are created by maintainers following these steps:

Version Management

Flyte 2 follows Semantic Versioning 2.0.0:

MAJOR version (v2.0.0 → v3.0.0): Breaking changes
MINOR version (v2.1.0 → v2.2.0): New features, backward compatible
PATCH version (v2.1.1 → v2.1.2): Bug fixes, backward compatible

Creating a Release (Maintainers Only)

Ensure all changes are merged into main branch

Determine the version number based on changes:

# View changes since last release
git log v2.X.Y..HEAD --oneline

Create and push a tag:

git checkout main
git pull upstream main
git tag -a v2.X.Y -m "Release v2.X.Y"
git push upstream v2.X.Y

Create GitHub Release:
- Go to Releases
- Click "Draft a new release"
- Select the tag you just created
- Generate release notes
- Add any additional context or highlights
- Publish release
Verify Artifacts: The release triggers automated publishing to:
- Go modules: github.com/flyteorg/flyte/v2
- NPM: @flyteorg/flyte
- PyPI: flyteidl2
- Crates.io: flyte

Post-Release

Verify published packages:

# Go
GOPROXY=https://proxy.golang.org go list -m github.com/flyteorg/flyte/v2@v2.X.Y

# NPM
npm view @flyteorg/flyte@2.X.Y

# PyPI
pip index versions flyteidl2

# Crates.io
cargo search flyte

Update dependent projects with the new version
Announce the release in community channels

Best Practices

Protocol Buffer Design

Keep messages small and focused - Single responsibility principle

Use oneof for mutually exclusive fields:

message Task {
  oneof task_type {
    PythonTask python = 1;
    ContainerTask container = 2;
  }
}

Use optional for truly optional fields (proto3)
Use repeated for arrays/lists
Provide sensible defaults where appropriate

Documentation

Document all public APIs
Include usage examples in comments
Keep README.md and CONTRIBUTING.md up to date
Add inline comments for complex logic

Git Workflow

Commit early and often
Keep commits atomic and focused
Write meaningful commit messages
Squash small "fix" commits before PR
Rebase instead of merge when updating your branch

Communication

Be respectful and professional
Ask questions when unclear
Provide context in discussions
Update PR descriptions as scope changes
Respond to reviews in a timely manner

Getting Help

Documentation: Flyte Documentation
Community Slack: Flyte Slack
GitHub Issues: Report Issues
GitHub Discussions: Ask Questions

License

By contributing to Flyte 2, you agree that your contributions will be licensed under the Apache License 2.0.

Uh oh!

FilesExpand file tree

CONTRIBUTING.md

Latest commit

History