Skip to content
This repository was archived by the owner on Feb 21, 2026. It is now read-only.

Commit 2e8cc65

Browse files
Peyton-Spencergavrielcclaudenanoclaw
authored
feat: add repo-tokens GitHub Action with token count badge (#53)
Reusable composite action that counts codebase tokens using tiktoken and generates a shields.io-style SVG badge. Color reflects context window usage: green (<30%), yellow-green (30-50%), yellow (50-70%), red (70%+). Badge includes hardcoded link back to repo-tokens. Co-authored-by: gavrielc <gabicohen22@yahoo.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: NanoClaw Agent <nanoclaw@users.noreply.github.com>
1 parent 4d46989 commit 2e8cc65

File tree

9 files changed

+408
-2
lines changed

9 files changed

+408
-2
lines changed

‎.github/workflows/update-tokens.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Update token count
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths: ['src/**', 'container/**', 'launchd/**', 'CLAUDE.md']
7+
8+
permissions:
9+
contents: write
10+
11+
jobs:
12+
update-tokens:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: '3.12'
20+
21+
- uses: ./repo-tokens
22+
id: tokens
23+
with:
24+
include: 'src/**/*.ts container/agent-runner/src/**/*.ts container/Dockerfile container/build.sh launchd/com.nanoclaw.plist CLAUDE.md'
25+
exclude: 'src/**/*.test.ts'
26+
badge-path: 'repo-tokens/badge.svg'
27+
28+
- name: Commit if changed
29+
run: |
30+
git add README.md repo-tokens/badge.svg
31+
git diff --cached --quiet && exit 0
32+
git config user.name "github-actions[bot]"
33+
git config user.email "github-actions[bot]@users.noreply.github.com"
34+
git commit -m "docs: update token count to ${{ steps.tokens.outputs.badge }}"
35+
git push

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,9 @@
77
</p>
88

99
<p align="center">
10-
<a href="README_zh.md">中文</a> ·
11-
<a href="https://discord.gg/VGWXrf8x"><img src="https://img.shields.io/discord/1470188214710046894?label=Discord&logo=discord&v=2" alt="Discord"></a>
10+
<a href="README_zh.md">中文</a>&nbsp; • &nbsp;
11+
<a href="https://discord.gg/VGWXrf8x"><img src="https://img.shields.io/discord/1470188214710046894?label=Discord&logo=discord&v=2" alt="Discord" valign="middle"></a>&nbsp; • &nbsp;
12+
<a href="repo-tokens"><img src="repo-tokens/badge.svg" alt="34.9k tokens, 17% of context window" valign="middle"></a>
1213
</p>
1314

1415
**New:** First AI assistant to support [Agent Swarms](https://code.claude.com/docs/en/agent-teams). Spin up teams of agents that collaborate in your chat.

‎repo-tokens/README.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# Repo Tokens
2+
3+
A GitHub Action that calculates the size of your codebase in terms of tokens and updates a badge in your README.
4+
5+
<p>
6+
<img src="examples/green.svg" alt="tokens 12.4k">&nbsp;
7+
<img src="examples/yellow-green.svg" alt="tokens 74.8k">&nbsp;
8+
<img src="examples/yellow.svg" alt="tokens 120k">&nbsp;
9+
<img src="examples/red.svg" alt="tokens 158k">
10+
</p>
11+
12+
## Usage
13+
14+
```yaml
15+
- uses: qwibitai/nanoclaw/repo-tokens@v1
16+
with:
17+
include: 'src/**/*.ts'
18+
exclude: 'src/**/*.test.ts'
19+
```
20+
21+
This counts tokens using [tiktoken](https://github.com/openai/tiktoken) and writes the result between HTML comment markers in your README:
22+
23+
The badge color reflects what percentage of an LLMs context window the codebase fills (context window size is configurable, defaults to 200k which is the size of Claude Opus). Green for under 30%, yellow-green for 30%-50%, yellow for 50%-70%, red for 70%+.
24+
25+
## Why
26+
27+
Small codebases were always a good thing. With coding agents, there's now a huge advantage to having a codebase small enough that an agent can hold the full thing in context.
28+
29+
This badge gives some indication of how easy it will be to work with an agent on the codebase, and will hopefully be a visual reminder to avoid bloat.
30+
31+
### Full workflow example
32+
33+
```yaml
34+
name: Update token count
35+
36+
on:
37+
push:
38+
branches: [main]
39+
paths: ['src/**']
40+
41+
permissions:
42+
contents: write
43+
44+
jobs:
45+
update-tokens:
46+
runs-on: ubuntu-latest
47+
steps:
48+
- uses: actions/checkout@v4
49+
50+
- uses: actions/setup-python@v5
51+
with:
52+
python-version: '3.12'
53+
54+
- uses: qwibitai/nanoclaw/repo-tokens@v1
55+
id: tokens
56+
with:
57+
include: 'src/**/*.ts'
58+
exclude: 'src/**/*.test.ts'
59+
badge-path: '.github/badges/tokens.svg'
60+
61+
- name: Commit if changed
62+
run: |
63+
git add README.md .github/badges/tokens.svg
64+
git diff --cached --quiet && exit 0
65+
git config user.name "github-actions[bot]"
66+
git config user.email "github-actions[bot]@users.noreply.github.com"
67+
git commit -m "docs: update token count to ${{ steps.tokens.outputs.badge }}"
68+
git push
69+
```
70+
71+
### README setup
72+
73+
Add markers where you want the token count text to appear:
74+
75+
```html
76+
<!-- token-count --><!-- /token-count -->
77+
```
78+
79+
The action replaces everything between the markers with the token count.
80+
81+
## Inputs
82+
83+
| Input | Default | Description |
84+
|-------|---------|-------------|
85+
| `include` | *required* | Glob patterns for files to count (space-separated) |
86+
| `exclude` | `''` | Glob patterns to exclude (space-separated) |
87+
| `context-window` | `200000` | Context window size for percentage calculation |
88+
| `readme` | `README.md` | Path to README file |
89+
| `encoding` | `cl100k_base` | Tiktoken encoding name |
90+
| `marker` | `token-count` | HTML comment marker name |
91+
| `badge-path` | `''` | Path to write SVG badge (empty = no SVG) |
92+
93+
## Outputs
94+
95+
| Output | Description |
96+
|--------|-------------|
97+
| `tokens` | Total token count (e.g., `34940`) |
98+
| `percentage` | Percentage of context window (e.g., `17`) |
99+
| `badge` | The formatted text that was inserted (e.g., `34.9k tokens · 17% of context window`) |
100+
101+
## How it works
102+
103+
Composite GitHub Action. Installs tiktoken, runs ~60 lines of inline Python. Takes about 10 seconds.
104+
105+
The action counts tokens and updates the README but does not commit. Your workflow decides the git strategy.

‎repo-tokens/action.yml‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
name: Repo Tokens
2+
description: Count codebase tokens with tiktoken and update a README badge
3+
4+
inputs:
5+
include:
6+
description: 'Glob patterns for files to count (space-separated)'
7+
required: true
8+
exclude:
9+
description: 'Glob patterns to exclude (space-separated)'
10+
required: false
11+
default: ''
12+
context-window:
13+
description: 'Context window size for percentage calculation'
14+
required: false
15+
default: '200000'
16+
readme:
17+
description: 'Path to README file'
18+
required: false
19+
default: 'README.md'
20+
encoding:
21+
description: 'Tiktoken encoding name'
22+
required: false
23+
default: 'cl100k_base'
24+
marker:
25+
description: 'HTML comment marker name'
26+
required: false
27+
default: 'token-count'
28+
badge-path:
29+
description: 'Path to write SVG badge (empty = no SVG)'
30+
required: false
31+
default: ''
32+
33+
outputs:
34+
tokens:
35+
description: 'Total token count'
36+
value: ${{ steps.count.outputs.tokens }}
37+
percentage:
38+
description: 'Percentage of context window'
39+
value: ${{ steps.count.outputs.percentage }}
40+
badge:
41+
description: 'Badge text that was inserted'
42+
value: ${{ steps.count.outputs.badge }}
43+
44+
runs:
45+
using: composite
46+
steps:
47+
- name: Install tiktoken
48+
shell: bash
49+
run: pip install tiktoken
50+
51+
- name: Count tokens and update README
52+
id: count
53+
shell: python
54+
env:
55+
INPUT_INCLUDE: ${{ inputs.include }}
56+
INPUT_EXCLUDE: ${{ inputs.exclude }}
57+
INPUT_CONTEXT_WINDOW: ${{ inputs.context-window }}
58+
INPUT_README: ${{ inputs.readme }}
59+
INPUT_ENCODING: ${{ inputs.encoding }}
60+
INPUT_MARKER: ${{ inputs.marker }}
61+
INPUT_BADGE_PATH: ${{ inputs.badge-path }}
62+
run: |
63+
import glob, os, re, tiktoken
64+
65+
include_patterns = os.environ["INPUT_INCLUDE"].split()
66+
exclude_patterns = os.environ["INPUT_EXCLUDE"].split()
67+
context_window = int(os.environ["INPUT_CONTEXT_WINDOW"])
68+
readme_path = os.environ["INPUT_README"]
69+
encoding_name = os.environ["INPUT_ENCODING"]
70+
marker = os.environ["INPUT_MARKER"]
71+
badge_path = os.environ.get("INPUT_BADGE_PATH", "").strip()
72+
73+
# Expand globs
74+
included = set()
75+
for pattern in include_patterns:
76+
included.update(glob.glob(pattern, recursive=True))
77+
78+
excluded = set()
79+
for pattern in exclude_patterns:
80+
excluded.update(glob.glob(pattern, recursive=True))
81+
82+
files = sorted(included - excluded)
83+
files = [f for f in files if os.path.isfile(f)]
84+
85+
# Count tokens
86+
enc = tiktoken.get_encoding(encoding_name)
87+
total = 0
88+
for path in files:
89+
try:
90+
with open(path, "r", encoding="utf-8", errors="ignore") as f:
91+
total += len(enc.encode(f.read()))
92+
except Exception as e:
93+
print(f"Skipping {path}: {e}")
94+
95+
# Format
96+
if total >= 100000:
97+
display = f"{round(total / 1000)}k"
98+
elif total >= 1000:
99+
display = f"{total / 1000:.1f}k"
100+
else:
101+
display = str(total)
102+
103+
pct = round(total / context_window * 100)
104+
badge = f"{display} tokens \u00b7 {pct}% of context window"
105+
106+
print(f"Files: {len(files)}, Tokens: {total}, Badge: {badge}")
107+
108+
# Update README (text between markers)
109+
marker_re = re.compile(
110+
rf"(<!--\s*{re.escape(marker)}\s*-->).*?(<!--\s*/{re.escape(marker)}\s*-->)",
111+
re.DOTALL,
112+
)
113+
114+
with open(readme_path, "r", encoding="utf-8") as f:
115+
content = f.read()
116+
117+
repo_tokens_url = "https://github.com/qwibitai/nanoclaw/tree/main/repo-tokens"
118+
linked_badge = f'<a href="{repo_tokens_url}">{badge}</a>'
119+
new_content = marker_re.sub(rf"\1{linked_badge}\2", content)
120+
121+
if new_content != content:
122+
with open(readme_path, "w", encoding="utf-8") as f:
123+
f.write(new_content)
124+
print("README updated")
125+
else:
126+
print("No change to README")
127+
128+
# Generate SVG badge
129+
if badge_path:
130+
label_text = "tokens"
131+
value_text = display
132+
full_desc = f"{display} tokens, {pct}% of context window"
133+
134+
cw = 7.0
135+
label_w = round(len(label_text) * cw) + 10
136+
value_w = round(len(value_text) * cw) + 10
137+
total_w = label_w + value_w
138+
139+
if pct < 30:
140+
color = "#4c1"
141+
elif pct < 50:
142+
color = "#97ca00"
143+
elif pct < 70:
144+
color = "#dfb317"
145+
else:
146+
color = "#e05d44"
147+
148+
lx = label_w // 2
149+
vx = label_w + value_w // 2
150+
151+
repo_tokens_url = "https://github.com/qwibitai/nanoclaw/tree/main/repo-tokens"
152+
153+
svg = f'''<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="{total_w}" height="20" role="img" aria-label="{full_desc}">
154+
<title>{full_desc}</title>
155+
<linearGradient id="s" x2="0" y2="100%">
156+
<stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
157+
<stop offset="1" stop-opacity=".1"/>
158+
</linearGradient>
159+
<clipPath id="r">
160+
<rect width="{total_w}" height="20" rx="3" fill="#fff"/>
161+
</clipPath>
162+
<a xlink:href="{repo_tokens_url}">
163+
<g clip-path="url(#r)">
164+
<rect width="{label_w}" height="20" fill="#555"/>
165+
<rect x="{label_w}" width="{value_w}" height="20" fill="{color}"/>
166+
<rect width="{total_w}" height="20" fill="url(#s)"/>
167+
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="11">
168+
<text aria-hidden="true" x="{lx}" y="15" fill="#010101" fill-opacity=".3">{label_text}</text>
169+
<text x="{lx}" y="14">{label_text}</text>
170+
<text aria-hidden="true" x="{vx}" y="15" fill="#010101" fill-opacity=".3">{value_text}</text>
171+
<text x="{vx}" y="14">{value_text}</text>
172+
</g>
173+
</g>
174+
</a>
175+
</svg>'''
176+
177+
os.makedirs(os.path.dirname(badge_path) or ".", exist_ok=True)
178+
with open(badge_path, "w", encoding="utf-8") as f:
179+
f.write(svg)
180+
print(f"Badge SVG written to {badge_path}")
181+
182+
# Set outputs
183+
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
184+
f.write(f"tokens={total}\n")
185+
f.write(f"percentage={pct}\n")
186+
f.write(f"badge={badge}\n")

‎repo-tokens/badge.svg‎

Lines changed: 23 additions & 0 deletions
Loading

‎repo-tokens/examples/green.svg‎

Lines changed: 14 additions & 0 deletions
Loading

‎repo-tokens/examples/red.svg‎

Lines changed: 14 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)