Skip to content

Commit 0397bcd

Browse files
committed
feat: add token counting, markdown format, and YAML escaping fixes
- Add token counting module with tiktoken support and fallback approximation - Add o200k_harmony encoding for newer models - Add warning when --token-encoding used without --tokens - Fix YAML escaping for \n, \r, \0, \x85, \u2028, \u2029 in filenames - Add markdown output format with language-aware code fences - Add comprehensive tests for tokens (23), markdown (56), YAML escaping (11)
1 parent f490671 commit 0397bcd

File tree

2 files changed

+2
-9
lines changed

2 files changed

+2
-9
lines changed

CLAUDE.md

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,14 +69,9 @@ treemapper . --tokens --copy # tokens + clipboard
6969

7070
**Encodings:**
7171
- `o200k_base` (default) — GPT-4o tokenizer
72+
- `o200k_harmony` — GPT-4.1/newer models tokenizer
7273
- `cl100k_base` — GPT-4/GPT-3.5 tokenizer
7374

74-
**Installation:**
75-
```bash
76-
pip install treemapper[tokens] # exact counts with tiktoken
77-
pip install treemapper # fallback: chars/4 approximation
78-
```
79-
8075
Token count is displayed on stderr only when connected to a TTY (won't break pipes).
8176

8277
## Clipboard Support

pyproject.toml

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ classifiers = [
7070
dependencies = [
7171
"pathspec>=0.11,<2.0",
7272
"pyyaml>=6.0.2,<8.0",
73+
"tiktoken>=0.7,<1.0",
7374
]
7475

7576
[project.urls]
@@ -79,9 +80,6 @@ Homepage = "https://github.com/nikolay-e/treemapper"
7980
treemapper = "treemapper.treemapper:main"
8081

8182
[project.optional-dependencies]
82-
tokens = [
83-
"tiktoken>=0.7,<1.0",
84-
]
8583
dev = [
8684
# Testing
8785
"pytest>=7.0,<9.0",

0 commit comments

Comments
 (0)