Commit d42081b
committed
feat: add token counting, markdown format, and YAML escaping fixes
- Add token counting module with tiktoken support and fallback approximation
- Add o200k_harmony encoding for newer models
- Add warning when --token-encoding used without --tokens
- Fix YAML escaping for \n, \r, \0, \x85, \u2028, \u2029 in filenames
- Add markdown output format with language-aware code fences
- Add comprehensive tests for tokens (23), markdown (56), YAML escaping (11)1 parent f490671 commit d42081b
2 files changed
+5
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| |||
0 commit comments