Commit 6f72418
feat: add automatic streaming detection for piped input
- Automatically use streaming mode when stdin is piped
- No need to specify 'stream' command explicitly
- Simplifies usage: cat file.txt | tokenizer llama3
- Still supports explicit stream command for clarity
- Updates documentation to show both methods
The tokenizer now intelligently detects its context:
- With args: encode mode
- With piped input: stream mode
- Neither: show usage
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 72a1350 commit 6f72418
1 file changed
+15
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
5 | 7 | | |
6 | 8 | | |
7 | 9 | | |
| |||
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | | - | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
37 | 42 | | |
38 | 43 | | |
39 | 44 | | |
| |||
61 | 66 | | |
62 | 67 | | |
63 | 68 | | |
64 | | - | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
65 | 78 | | |
66 | 79 | | |
67 | 80 | | |
| |||
0 commit comments