@async/pipeline

Local-first TypeScript pipelines for projects that want one task graph to run on a laptop, in GitHub Actions, and eventually on isolated or remote runners.

@async/pipeline is intentionally small: a typed pipeline.ts, a task graph, a local run/cache store under .async/, and runner adapters for host and Lima execution.

What You Get

• Typed pipeline definitions with definePipeline, task, job, trigger, source, and sh.
• Task fields for dependsOn, inputs, outputs, cache, retry, timeout, requires, environment, steps, and run.
• Explicit many-repo sources with namespaced task refs such as storefront:test.
• Deferred shell commands with sh((ctx) => sh\...`)` for runtime values like the candidate repo path.
• Local execution records in .async/runs/<run-id>/.
• Local task cache in .async/cache/tasks/.
• CLI commands for running jobs, running one task, listing, graphing, explaining, and doctor checks.
• Thin GitHub Actions setup that runs the same pipeline used locally.

Package Split

Package	Purpose
@async/pipeline	Public convenience package and CLI bin.
@async/pipeline-core	Pipeline, task, job, graph, and type contracts.
@async/pipeline-node	CLI, filesystem store, scheduler, host runner, and doctor checks.
@async/pipeline-adapter-lima	Lima runner adapter using limactl.

Get Started In This Repo

cd /Users/patrickjs/code/async-framework/async-pipeline
pnpm install --frozen-lockfile
pnpm build
pnpm async-pipeline list
pnpm async-pipeline run verify

Inspect the run that was created:

ls .async/runs
cat .async/runs/<run-id>/summary.md
cat .async/runs/<run-id>/execution.json

Run local health checks:

pnpm async-pipeline doctor

The self pipeline lives in pipeline.ts. It defines typecheck, test, build, pack, and the verify job.

Use It In Another Project

After the packages are published, install the public package:

pnpm add -D @async/pipeline

Add a pipeline.ts:

import { definePipeline, job, sh, task } from "@async/pipeline";

export default definePipeline({
  name: "app",
  namedInputs: {
    source: ["src/**/*.ts", "package.json", "pnpm-lock.yaml", "tsconfig.json"]
  },
  tasks: {
    typecheck: task({
      inputs: ["source"],
      cache: true,
      run: sh`pnpm typecheck`
    }),
    build: task({
      dependsOn: ["typecheck"],
      inputs: ["source"],
      outputs: ["dist/**"],
      cache: true,
      timeout: "2m",
      run: sh`pnpm build`
    })
  },
  jobs: {
    verify: job({ target: "build" })
  }
});

Add package scripts:

{
  "scripts": {
    "async-pipeline": "async-pipeline",
    "verify": "async-pipeline run verify"
  }
}

Then run:

pnpm async-pipeline run verify

Node 24 is the current recommended runtime for pipeline.ts because the CLI imports TypeScript config files directly. Node 20+ can use pipeline.mjs or pipeline.js.

CLI

Use async-pipeline as the explicit command in CI and docs.

async-pipeline run <job>
async-pipeline run-task <task>
async-pipeline list
async-pipeline graph --format json
async-pipeline graph --format dot
async-pipeline explain <task>
async-pipeline sources list
async-pipeline sources sync
async-pipeline metadata --format json
async-pipeline matrix <job> --format github
async-pipeline doctor

Short aliases and smart runner dispatch belong in @async/run, not this package.

GitHub Actions

The workflow should stay thin: install dependencies, build the CLI, and invoke the same pipeline you run locally.

permissions:
  contents: read

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<pinned-sha>
      - uses: actions/setup-node@<pinned-sha>
        with:
          node-version: 24
      - run: |
          corepack enable
          corepack prepare pnpm@10.20.0 --activate
      - run: pnpm install --frozen-lockfile
      - run: pnpm build
      - run: pnpm async-pipeline run verify
        env:
          CI: true

The checked-in workflow is .github/workflows/ci.yml.

Docs

• Getting started
• How it works
• Running locally
• GitHub Actions setup
• API reference
• Many-repo impact runs

Current Limits

• Task execution is deterministic and sequential today. Parallel scheduling is part of the next tranche.
• The CLI uses the host runner by default. The Lima adapter is available programmatically and doctor checks for limactl.
• Cache is local-first only. Remote cache and shared cache backends are not implemented yet.
• Many-repo sources are explicit. @async/pipeline does not infer reverse dependencies from package manifests or lockfiles.
• Deno and Ollama are optional future requirements, not runtime dependencies.