Commit d5247ab
committed
feat: add token counting, markdown format, and YAML escaping fixes
- Add token counting module with tiktoken support and fallback approximation
- Add o200k_harmony encoding for newer models
- Add warning when --token-encoding used without --tokens
- Fix YAML escaping for \n, \r, \0, \x85, \u2028, \u2029 in filenames
- Add markdown output format with language-aware code fences
- Add comprehensive tests for tokens (23), markdown (56), YAML escaping (11)1 parent f490671 commit d5247ab
2 files changed
+2
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
0 commit comments