Commit 51411af
authored
feat: add streaming mode for large files (#58)
Adds --stream flag for element-by-element processing of JSON arrays,
JSONL, and CSV input. In streaming mode, each element is parsed,
optionally transformed via mappings, and written to the output format
immediately rather than buffering the entire dataset in memory.
Supported streaming pipelines:
- JSONL → JSONL/JSON/CSV (true line-by-line streaming)
- CSV → JSONL/JSON/CSV (true row-by-row streaming)
- JSON array → JSONL/JSON/CSV (parse + stream output)
Features:
- StreamWriter abstraction for incremental output
- Mapping support (per-element transforms via -e/-m)
- CSV delimiter support in streaming mode
- Graceful fallback: --stream is silently ignored for unsupported
format combinations (e.g. YAML→JSON)
- Periodic flushing via BufWriter
Includes 24 unit tests and 7 integration tests covering all streaming
combinations, edge cases, and mapping integration.
Fixes #291 parent ca37b8a commit 51411af
4 files changed
+853
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
153 | 157 | | |
154 | 158 | | |
155 | 159 | | |
| |||
419 | 423 | | |
420 | 424 | | |
421 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
422 | 431 | | |
423 | 432 | | |
424 | 433 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
0 commit comments