You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+14-11Lines changed: 14 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,14 +14,18 @@ Differential dataflow is a powerful data-parallel programming framework that ena
14
14
## Key Features
15
15
16
16
-**Incremental Processing**: Efficiently process changes to input data without recomputing everything
17
-
-**Rich Operators**: Supports common operations like:
18
-
-`map()`: Transform elements
19
-
-`filter()`: Filter elements based on predicates
20
-
-`join()`: Join two collections
21
-
-`reduce()`: Aggregate values by key
17
+
-**Rich Operators**: Supports common operations with a pipeline API:
18
+
-`concat()`: Concatenate two streams
19
+
-`consolidate()`: Consolidates the elements in the stream
22
20
-`count()`: Count elements by key
23
21
-`distinct()`: Remove duplicates
22
+
-`filter()`: Filter elements based on predicates
24
23
-`iterate()`: Perform iterative computations
24
+
-`join()`: Join two collections
25
+
-`map()`: Transform elements
26
+
-`reduce()`: Aggregate values by key
27
+
-`output()`: Output the results of the stream
28
+
-`pipe()`: Build a pipeline of operators
25
29
-**SQLite Integration**: Optional SQLite backend for managing operator state
26
30
-**Type Safety**: Full TypeScript type safety and inference
27
31
@@ -71,7 +75,7 @@ input.sendData(v([0, 0]), new MultiSet([
71
75
input.sendFrontier(v([0, 1]))
72
76
73
77
// Process the data
74
-
graph.step()
78
+
graph.run()
75
79
76
80
// Output will show:
77
81
// 6 (from 1 + 5)
@@ -107,7 +111,7 @@ See the `examples/` directory for more complex scenarios including:
107
111
108
112
## Implementation Details
109
113
110
-
The implementation follows the structure outlined in the Materialize blog post, with some TypeScript-specific adaptations:
114
+
The implementation is based on the the one outlined in the [Materialize blog post](https://materialize.com/blog/differential-from-scratch/), with some TypeScript-specific adaptations, along with using a pipeline rather than builder api pattern.
111
115
112
116
1. Core data structures:
113
117
@@ -118,12 +122,11 @@ The implementation follows the structure outlined in the Materialize blog post,
118
122
119
123
2. Operators:
120
124
121
-
- Base operator classes in `src/operators.ts`
122
-
- SQLite variants in `src/operators-sqlite.ts`
123
-
- Graph construction in `src/pipe.ts`
125
+
- Base operator classes in `src/operators/`
126
+
- SQLite variants in `src/sqlite/operators/`
124
127
125
128
3. Graph execution:
126
-
- Dataflow graph management in `src/graph.ts`
129
+
- Dataflow graph management in `src/graph.ts` and `src/D2.ts`
0 commit comments