Skip to content

Commit 6f72418

Browse files
jackspirouclaude
andcommitted
feat: add automatic streaming detection for piped input
- Automatically use streaming mode when stdin is piped - No need to specify 'stream' command explicitly - Simplifies usage: cat file.txt | tokenizer llama3 - Still supports explicit stream command for clarity - Updates documentation to show both methods The tokenizer now intelligently detects its context: - With args: encode mode - With piped input: stream mode - Neither: show usage 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 72a1350 commit 6f72418

File tree

1 file changed

+15
-2
lines changed

1 file changed

+15
-2
lines changed

llama3/cmd/llama3/command.go

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
package llama3cmd
33

44
import (
5+
"os"
6+
57
"github.com/spf13/cobra"
68
)
79

@@ -31,9 +33,12 @@ Available commands:
3133
# Decode tokens
3234
tokenizer llama3 decode 128000 9906 11 1917 0 128001
3335
34-
# Stream from stdin
36+
# Stream from stdin (explicit)
3537
cat large_file.txt | tokenizer llama3 stream
3638
39+
# Stream from stdin (implicit - automatic)
40+
cat large_file.txt | tokenizer llama3
41+
3742
# Show tokenizer info
3843
tokenizer llama3 info`,
3944
Args: cobra.ArbitraryArgs,
@@ -61,7 +66,15 @@ Available commands:
6166
return encodeCmd.Execute()
6267
}
6368

64-
// No args provided, show usage
69+
// No args provided - check if stdin is piped
70+
stat, _ := os.Stdin.Stat()
71+
if (stat.Mode() & os.ModeCharDevice) == 0 {
72+
// Data is being piped to stdin, use streaming mode
73+
streamCmd := newStreamCmd()
74+
return streamCmd.Execute()
75+
}
76+
77+
// No args and no piped input, show usage
6578
return cmd.Usage()
6679
},
6780
}

0 commit comments

Comments
 (0)