NetflixConductorCompare

POP & Netflix Conductor Comparison

Purpose

This document is to focus on the problems to consider when building workflow orchestration solutions and identify the differences in products such as POP and Netflix Conductor.

Technologies at a Glance

POP

• A workflow orchestration system where the workflow is scheduled as a unit giving resource priority once selected

• Priority queuing and customizable scheduling algorithms

• Customer aware for multi-tenant

• Authentication via AWS Authorizer

• Data object visibility for allowed customers

• Interactive workflow (retry)

• A JSON DAG based blueprint defines the execution flow

• Workflow template storage

• Executes operations within a workflow by spinning up on-demand Kubernetes Pods (dynamic resource allocation)

• Http UI for workflow processing and data object management

Technologies

• API Gateway

• AWS Authorizer

• Lambda

• SQS

• Dynamo (temporary)

• CloudFormation

• CloudWatch

• Kubernetes

• Kubernetes Annotations

• Docker

Netflix Conductor

• A workflow orchestration system where tasks within a workflow are queued when ready

• Limited queuing segregation

• A JSON DSL based blueprint defines the execution flow

• UI for workflow processing and control

• Interactive workflow (stop/retry/pause)

• Workflow template storage

Technologies

• Elastic Search

• Dynomite

• Servlet Container: Tomcat, Jetty, or similar running JDK 1.8 or higher

• Casandra (optional)

• AWS S3 (optional)

Feature Comparison For Workflow Products

Feature	POP	POP Notes	Netflix Conductor	Netflix Conductor Notes
Authentication	Y	AWS Authorizer	N	No Authorization at this time.
Multi-tenant	Y	Customer aware	N
Data object visibility	Y	Yes, only allowed customers can see and perform operations on their data objects	N
byQueries	Y	Search data byCID, byTitle, byCustomer ... etc	???	I believe there is no filtering on data in Conductor
Scheduling Priority of Workflow	Y	Customizable by Queue type	ie: FIFO, Customer Fairness	N
Queuing	Y	Workflow as an entire unit	Y	Tasks within a workflow only when ready. No priority or customer fairness Option1: One Queue for all Tasks. Homogeneous task queuing means homogeneous Task Workers. Option2: Predefine isolationGroupIds to delineate Queues & Workers ie: isolationGroupId=encode. Workflow service coupling with workers. Task decision execution is in separate thread but same JVM allocation as all other task decision making causing vertical scaling. Option3: Predefine executor namespaces.A separate executor is used with separate JVM allocation. (Still same machine so it's limited to Jetty allocation)
Variable Assignment in Workflow	Y	Y
Custom Workflow On-Demand	Y	Y
Pre-Configured Workflow Storage	Y	Agenda Template	Y
Resource Priority for Workflows	Y	Once a workflow is selected it is given resource priority before next in line	N	FIFO Task processing. Workers will pull tasks off non-priority queues (FIFO).Workflow tasks are not run as a unit. Partial queuing leads to larger wait times, no dedicated resource priority for a workflow
Pause, Restart, Terminate Functionality	Partial (restart)	Y
Workers		Typically 1 workflow puller per Kubernetes cluster.		1-n worker polling per task queue. Worker Agents adding load to service when no work is available. Worker Agent Resource consumption when unnecessary. Worker Agents would still need to spin up on-demand resources so they don't allocate a large amount of cpu for just polling.
Dynamic Resource Allocation	Y	We allocate resources based on the work load and processing needs	TBD
Progress	Y	Progress is stored separately. Progress is available for all Workflow tasks/operations	Y	Progress is within the workflow document. Concerns: No inter-task meta-data storage or individual task output
Service Configuration for Workers	Y	Simple Kubernetes lookup table for Docker Image name	Y	Service JVM configuration if it's a new task type. Could need server bounce.
Parallel processing of Tasks	Y	If a task/operation has no dependencies it runs. Optional: You can specify a hard dependency	N	Only if the workflow was written to do so and they happen to be tasks that get picked up near the same time (remember one would have to queue right after the other).
Webclient UI	Y	Simple CRUD webclient	Y
Workflow Control UI	Y	Submission and restart	Y
Workflow chaining	Y	Workflows would be chained by a task/operation submitting a new workflow	Y	Event Task within a workflow to trigger another workflow
In Memory Tasks	Y	The Executor can run operations within itself. Some are preconfigured like HTTP caller.	Y	Configured with Conductor as a System Task. Runs in the memory footprint of the service.
Decision Task	N	Not within the workflow itself	Y	Done via a system task