Skip to content

Commit c16e229

Browse files
committed
Update readme
1 parent 540051b commit c16e229

File tree

1 file changed

+14
-11
lines changed

1 file changed

+14
-11
lines changed

README.md

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,18 @@ Differential dataflow is a powerful data-parallel programming framework that ena
1414
## Key Features
1515

1616
- **Incremental Processing**: Efficiently process changes to input data without recomputing everything
17-
- **Rich Operators**: Supports common operations like:
18-
- `map()`: Transform elements
19-
- `filter()`: Filter elements based on predicates
20-
- `join()`: Join two collections
21-
- `reduce()`: Aggregate values by key
17+
- **Rich Operators**: Supports common operations with a pipeline API:
18+
- `concat()`: Concatenate two streams
19+
- `consolidate()`: Consolidates the elements in the stream
2220
- `count()`: Count elements by key
2321
- `distinct()`: Remove duplicates
22+
- `filter()`: Filter elements based on predicates
2423
- `iterate()`: Perform iterative computations
24+
- `join()`: Join two collections
25+
- `map()`: Transform elements
26+
- `reduce()`: Aggregate values by key
27+
- `output()`: Output the results of the stream
28+
- `pipe()`: Build a pipeline of operators
2529
- **SQLite Integration**: Optional SQLite backend for managing operator state
2630
- **Type Safety**: Full TypeScript type safety and inference
2731

@@ -71,7 +75,7 @@ input.sendData(v([0, 0]), new MultiSet([
7175
input.sendFrontier(v([0, 1]))
7276

7377
// Process the data
74-
graph.step()
78+
graph.run()
7579

7680
// Output will show:
7781
// 6 (from 1 + 5)
@@ -107,7 +111,7 @@ See the `examples/` directory for more complex scenarios including:
107111

108112
## Implementation Details
109113

110-
The implementation follows the structure outlined in the Materialize blog post, with some TypeScript-specific adaptations:
114+
The implementation is based on the the one outlined in the [Materialize blog post](https://materialize.com/blog/differential-from-scratch/), with some TypeScript-specific adaptations, along with using a pipeline rather than builder api pattern.
111115

112116
1. Core data structures:
113117

@@ -118,12 +122,11 @@ The implementation follows the structure outlined in the Materialize blog post,
118122

119123
2. Operators:
120124

121-
- Base operator classes in `src/operators.ts`
122-
- SQLite variants in `src/operators-sqlite.ts`
123-
- Graph construction in `src/pipe.ts`
125+
- Base operator classes in `src/operators/`
126+
- SQLite variants in `src/sqlite/operators/`
124127

125128
3. Graph execution:
126-
- Dataflow graph management in `src/graph.ts`
129+
- Dataflow graph management in `src/graph.ts` and `src/D2.ts`
127130
- Message passing between operators
128131
- Frontier tracking and advancement
129132

0 commit comments

Comments
 (0)