βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SQL QUERY β
β "SELECT a + 10 FROM table" β
βββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. PARSE & ANALYZE (DataFusion) β
β QueryAnalyzer::analyze(sql) β LogicalPlan β
β - Parses SQL, validates schema, builds AST β
βββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 2. TRANSLATE TO PHYSICAL PLAN β
β translate_plan(logical_plan) β Differential Dataflow Graph β
β - Maps LogicalPlan operators to Differential Dataflow operators β
β - Compiles Expr trees into closures over Row β
β - Converts Arrow RecordBatch β Vec<Row> β
βββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 3. EXECUTE (Timely Dataflow) β
β timely::execute(|worker| { ... }) β
β β
β a) Create InputSession, feed Row data β
β b) Build Collection<G, Row, isize> from input β
β c) Apply operators: .map(), .filter(), .join(), .reduce() β
β d) Attach .inspect() or .consolidate() to observe results β
β β
β Data flows as: (Row, Time, Diff) β
βββββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 4. OUTPUT β
β Stream of (Row, Time, Diff) updates emitted incrementally β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ-
Notifications
You must be signed in to change notification settings - Fork 0
lgingerich/dataflow
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Β | Β | |||
Β | Β | |||
Β | Β | |||
Β | Β | |||
Β | Β | |||
Β | Β | |||
Β | Β | |||
Repository files navigation
About
Incremental Stream Processing Engine Built on Differential Dataflow
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published