Commit 0397bcd
committed
feat: add token counting, markdown format, and YAML escaping fixes
- Add token counting module with tiktoken support and fallback approximation
- Add o200k_harmony encoding for newer models
- Add warning when --token-encoding used without --tokens
- Fix YAML escaping for \n, \r, \0, \x85, \u2028, \u2029 in filenames
- Add markdown output format with language-aware code fences
- Add comprehensive tests for tokens (23), markdown (56), YAML escaping (11)1 parent f490671 commit 0397bcd
2 files changed
+2
-9
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
72 | 73 | | |
73 | 74 | | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | 75 | | |
81 | 76 | | |
82 | 77 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
| 73 | + | |
73 | 74 | | |
74 | 75 | | |
75 | 76 | | |
| |||
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | 83 | | |
86 | 84 | | |
87 | 85 | | |
| |||
0 commit comments