Skip to content

Commit 9042d01

Browse files
--max-tokens CLI parameter is now implemented for the Test subcommand.
Output is truncated to fit the specified token limit using Hugging Face's pretrained bert-base-cased tokenizer. 🦨 Skunky: Truncation is by character, not true token boundary; see code comment for future improvement. All changes build cleanly and are isolated to the explicit requirements.
1 parent 4afe1e7 commit 9042d01

File tree

4 files changed

+313
-10
lines changed

4 files changed

+313
-10
lines changed

0 commit comments

Comments
 (0)