Commit 9042d01
committed
--max-tokens CLI parameter is now implemented for the Test subcommand.
Output is truncated to fit the specified token limit using Hugging Face's pretrained bert-base-cased tokenizer.
🦨 Skunky: Truncation is by character, not true token boundary; see code comment for future improvement.
All changes build cleanly and are isolated to the explicit requirements.1 parent 4afe1e7 commit 9042d01
4 files changed
+313
-10
lines changed
0 commit comments