Proposal: “CWL → Prefect Flow” converter for simple CWL workflows

[brunifrancesco]

Proposal: CWL → Prefect Flow adapter (strict subset, Docker & Kubernetes backends)

Before beginning

The Common Workflow Language (CWL) is an open, community-driven specification for describing command-line tools and workflows in a portable and reproducible way. CWL focuses on explicit inputs, containerized execution, and well-defined filesystem semantics.
For more details, see the official specification

Summary

This proposal describes a strict, deterministic adapter that converts a single CWL workflow into a Prefect flow, where:

• the Prefect flow acts as an orchestrator
• each CWL step runs in its own container image
• execution is strictly sequential
• Docker and Kubernetes are supported as step execution backends
• filesystem semantics are enforced via a shared workspace contract

The goal is not to implement the full CWL specification, but to provide a reliable bridge for “simple CWL” workflows into Prefect’s execution and deployment model.
Other orchestration tools are out there that support more features coming from CWL, but we love Prefect and we think it could be useful to have this CWL support, even limited :)

Scope (intentionally strict)

Supported

• Single class: Workflow
• Simple scalar inputs (string, int, float, bool, optional)
• CommandLineTool steps only
• Docker images only
• No parallelism (forced sequential execution)
• One CWL step → one Prefect task
• Deterministic command construction
• Shared filesystem workspace

Explicitly not supported (v1)

• Scatter / parallelism
• Subworkflows
• Expressions / JavaScript requirements
• Complex CWL types
• Secondary files
• Non-Docker runtimes

Prefect-aligned execution model

Execution is modeled in two layers, consistent with Prefect’s architecture.

1. Parent flow (orchestrator)

   A standard Prefect flow run, executed via:

   • .serve() (local development), or
   • a deployment + work pool + worker (Docker worker or Kubernetes worker)

   The parent flow:

   • validates the CWL document
   • parses it into an internal representation (IR)
   • executes steps sequentially
   • spawns and waits for step runtimes

2. Child runtimes (CWL steps)

   Each CWL step runs in its own isolated runtime:

   Backend	Runtime per step
   Docker	Docker container
   Kubernetes	Kubernetes Job

This preserves CWL’s “container-per-step” semantics while using Prefect strictly as an orchestrator.

Execution architecture (Mermaid)

flowchart TD
  A[Prefect Deployment] --> B[Worker]
  B --> C[Parent Flow Run<br/>]

  C --> S1[Step 1<br/>Container / Job]
  S1 --> S2[Step 2<br/>Container / Job]
  S2 --> S3[Step N<br/>Container / Job]

  V[(Shared Workspace<br/>/workspace)] --- S1
  V --- S2
  V --- S3

Loading

Backend model

The library distinguishes step execution backend, not parent runtime.

Docker backend

• Parent flow runs locally or in a Docker worker
• Each step is executed via prefect-docker
• One Docker container per CWL step
• Shared workspace mounted into each container

Kubernetes backend

• Parent flow runs as a Kubernetes Job (Kubernetes worker)
• Each step is executed as a separate Kubernetes Job
• Jobs are created and awaited via prefect-kubernetes
• Shared workspace provided via PVC

Workspace / volume contract (core design)

Filesystem semantics are handled as infrastructure, not CWL semantics.

Canonical workspace layout

All steps see the same workspace:

/workspace
/workspace/<flow_run_id>/
  inputs/
  outputs/
  tmp/

Environment variables (always set)

WORKDIR=/workspace/<flow_run_id>

All CWL commands are resolved relative to WORKDIR.

Volume implementation by backend

Backend	Volume implementation
Docker	Bind mount: host directory → /workspace. If the parent flow runs in a Docker worker, access to the Docker daemon (e.g. via socket mount or remote daemon) is required to spawn step containers.
Kubernetes	Shared PVC mounted at /workspace in all step Jobs

Internal architecture

The internal architecture resolves around multiple components. Idea is simple: validating the CWL provided, parsing, creating the flow&tasks, allow user to deploy it:

Component	Role
Validator	Takes the CWL document and validates it
Parser	Parse the CWL to extract all the relevant sections (env vars, volumes, arguments, commands, docker images..)
Flow builder	Build the tasks and flow dynamically, setting the correct signature for the flow as well
Step Runner	Embed the backend selection logic inside the flows

As diagram:

flowchart TD
    A[CWL document] --> B[Validator]
    B -->|valid| C[Parser]
    B -->|invalid| E1[Validation error]

    C -->|parsed| D[Flow builder]
    C -->|parse error| E2[Parse error]

    D -->|built| F[Step runner wiring]
    D -->|build error| E3[Build error]

    F -->|ok| G[Generated flow and tasks]
    F -->|config error| E4[Runner configuration error]

    G --> N[Ready for deployment]

Loading

How each CWL step executes

• Steps are modeled as Prefect tasks that orchestrate external runtimes
• Retries, logging, state, and observability remain fully Prefect-native (CWL got support for it)
• Full CWL compliance, even limited in terms of supported features

Open questions (just a few)

• How this fit into Prefect exec model?
• Should Kubernetes support require a pre-existing PVC (v1)? Or we can create on the fly?
• Should step logs be streamed back into Prefect logs or linked externally?
• Should this live as:
  • an experimental Prefect integration, or
  • a community-maintained plugin?
• Shall we support all the worker pool types?

Conclusion

This approach:

• fits cleanly into Prefect’s deployment and worker model (I suppose so)
• preserves CWL’s container-per-step semantics
• keeps the implementation small, explicit, and maintainable (even if parsing can be tricky and somehow complicated for CWL newbies)
• avoids re-implementing a full CWL runtime and rely on Prefect to run the orchestration logic

Feedback on architecture, scope, and backend expectations is very welcome.

[desertaxle]

Thanks for the write up @brunifrancesco! I think a Prefect adapter for CWL would be great as a community-maintained package. What you've described makes sense at a high level, and I think some of our existing integrations in prefect-docker and prefect-kubernetes will be helpful in implementing this adapater. Let us know in this dicussion if you need any roadblocks on Prefect specifics as you're working on implementation and we can provide guidance!

[brunifrancesco]

Hello @desertaxle , let's get practical: at the current time, the implementation is fairly easy.
We got a function that returns a flow with some inner tasks emdedded, one per each step.
So basically, we run:

• CWL validation
• CWL parsing into some enriched internal representation
• for each runnable step (some steps can depend from something else), we pick a backend (Docker only, for now), and run a dedicated container(s), wait for it/them, move to the next step 'till the end.
• We've relied on the create_docker_container in the prefect-docker library.

Even working, this is pretty different from Prefect idea of pool type. At the same time, since a CWL can be composed by multiple docker images, how can I fill this gap? Can we do better?