ai-token-counter

A small, dependency-free Node.js utility for rough token estimation across OpenAI, Gemini, and Claude model families.

ai-token-counter helps developers estimate prompt size before making API calls. It is built for simple scripts, backend services, and internal tooling where you need a fast approximation without pulling in a full tokenizer.

It is intentionally lightweight and fast. The tradeoff is that estimates are approximate, not exact. Real provider token counts may differ.

The estimator now uses a chunk-based heuristic that weighs words, numbers, punctuation, CJK characters, and emoji separately, which produces more realistic results than a plain character count.

Highlights

• Dependency-free Node.js package
• Simple API for both plain text and chat-style messages
• Small CLI for fast prompt checks in scripts and terminals
• TypeScript typings included
• Supports common OpenAI, Gemini, and Claude naming patterns
• Built for lightweight prompt budgeting and preflight validation

Quick Start

Install:

npm install ai-token-counter

Estimate text:

const { countTokens } = require("ai-token-counter");

console.log(countTokens("Summarize this support issue.", "gpt-4o"));
console.log(countTokens("Summarize this support issue.", "gemini-2.0-flash"));

Estimate chat messages:

const { countMessages } = require("ai-token-counter");

const messages = [
  { role: "system", content: "You are a concise assistant." },
  { role: "user", content: "Summarize this outage report." }
];

console.log(countMessages(messages, "sonnet-4"));
console.log(countMessages(messages, "gemini-1.5-pro"));

TypeScript works out of the box:

import { countMessages, countTokens } from "ai-token-counter";

const textCount = countTokens("Summarize this support issue.", "gpt-4o");
const messageCount = countMessages(
  [{ role: "user", content: "Review this changelog." }],
  "sonnet-4"
);

Check model metadata and prompt fit:

const { getModelInfo, fitsContextWindow } = require("ai-token-counter");

console.log(getModelInfo("sonnet-4"));
console.log(fitsContextWindow("Summarize this issue.", "gpt-4o", 2000));

What problem it solves

When building with LLM APIs, it is common to need a quick estimate of prompt size before sending a request. Exact tokenization can be overkill for:

• prompt budget checks before an API call
• request validation in backend services
• logging and analytics for prompt sizes
• simple CLI tools or internal automation scripts
• rough cost estimation during development

ai-token-counter gives you a simple way to do that with a single function call.

It now supports both raw text estimation and chat-style message arrays. It also includes model metadata, context-window checks, and approximate token cost estimation.

Installation

Install with npm:

npm install ai-token-counter

Or with other package managers:

yarn add ai-token-counter

pnpm add ai-token-counter

Example usage

const { countTokens } = require("ai-token-counter");

const prompt = [
  "You are a helpful assistant.",
  "Summarize the following customer feedback in three bullet points,",
  "highlight the main risk, and suggest a next action."
].join(" ");

const openAiEstimate = countTokens(prompt, "gpt-4o");
const geminiEstimate = countTokens(prompt, "gemini-2.0-flash");
const claudeEstimate = countTokens(prompt, "claude-3-5-sonnet");
const claudeAliasEstimate = countTokens(prompt, "sonnet-4");
const claudeOpusAliasEstimate = countTokens(prompt, "opus 4.6");

console.log("OpenAI estimate:", openAiEstimate);
console.log("Gemini estimate:", geminiEstimate);
console.log("Claude estimate:", claudeEstimate);
console.log("Claude alias estimate:", claudeAliasEstimate);
console.log("Claude Opus alias estimate:", claudeOpusAliasEstimate);

Chat-style payloads are also supported:

const { countMessages } = require("ai-token-counter");

const messages = [
  {
    role: "system",
    content: "You are a concise coding assistant."
  },
  {
    role: "user",
    content: "Review this pull request summary and list the top two risks."
  }
];

console.log(countMessages(messages, "gpt-4o"));
console.log(countMessages(messages, "sonnet-4"));

You can also use the estimate to enforce a rough prompt limit before making a request:

const { countTokens } = require("ai-token-counter");

function canSendPrompt(text, model, maxTokens) {
  return countTokens(text, model) <= maxTokens;
}

console.log(canSendPrompt("Write a short product description.", "gpt-4.1-mini", 100));

The library exports:

• countTokens(text, model)
• countMessages(messages, model)
• getModelInfo(model)
• fitsContextWindow(input, model, maxOutputTokens?)
• estimateCost(input, model, options?)

CLI

The package also includes a small command-line interface.

Count inline text:

npx ai-token-counter "Summarize this pull request and call out the main risk." --model gpt-4o

Count text from a file:

npx ai-token-counter --model gemini-1.5-pro --file ./prompt.txt

Count chat-style messages from a JSON file:

npx ai-token-counter --model sonnet-4 --messages-file ./messages.json

Pipe plain text via stdin:

echo "Summarize this issue" | npx ai-token-counter --stdin --model gpt-4o

Pipe JSON message arrays via stdin:

cat messages.json | npx ai-token-counter --stdin --json --model sonnet-4

Show help:

npx ai-token-counter --help

Estimate cost directly from the CLI:

npx ai-token-counter --cost --model gpt-4o "Explain Kubernetes in 2 sentences"

Get machine-readable JSON output for scripts/CI:

npx ai-token-counter --json --model gpt-4o "Summarize this issue"

npx ai-token-counter --json --cost --model gpt-4o --output-tokens 500 "Summarize this issue"

--messages-file expects a JSON array:

[
  { "role": "system", "content": "You are a concise assistant." },
  { "role": "user", "content": "Summarize this release note." }
]

API

countTokens(text, model)

Returns a rough integer token estimate for the supplied text and model.

• text: string input to estimate
• model: model name string, such as gpt-4o, gpt-4.1-mini, gemini-2.0-flash, claude-3-opus, claude-3-5-sonnet, sonnet-4, or opus 4.6

Returns a rounded integer estimate. The function throws for unsupported model names or invalid input types.

countMessages(messages, model)

Returns a rough token estimate for a chat-style message array.

• messages: an array of objects with role and string content
• model: model name string, such as gpt-4o, gemini-1.5-pro, o3-mini, claude-3-5-sonnet, sonnet-4, or opus 4.6

The function sums the estimated content tokens and adds a small overhead for message structure.

getModelInfo(model)

Returns normalized metadata for a supported model family.

• provider: recognized provider family
• normalizedModel: normalized model string used internally
• contextWindow: default context window estimate for the provider family
• supportsMessages: whether message arrays are supported

fitsContextWindow(input, model, maxOutputTokens?)

Checks whether a prompt likely fits within the model's estimated context window.

• input: either a raw text string or a chat-style message array
• model: model name string
• maxOutputTokens: optional output-token budget to reserve

Returns an object with:

• fits
• inputTokens
• reservedOutputTokens
• availableInputTokens
• contextWindow
• provider

estimateCost(input, model, options?)

Estimates approximate input and output token cost for text or chat messages.

• input: either a raw text string or a chat-style message array
• model: model name string
• options.outputTokens: optional reserved output tokens

Returns:

• provider
• model
• inputTokens
• outputTokensReserved
• totalTokensEstimated
• estimatedInputCost
• estimatedOutputCost
• estimatedTotalCost

Cost Estimation

const { estimateCost } = require("ai-token-counter");

const result = estimateCost("Explain Kubernetes.", "gpt-4o", {
  outputTokens: 500
});

console.log(result);

The library uses a lightweight pricing map and falls back to provider defaults when an exact model price is not found.

Playground

A lightweight browser playground is available in this repository.

Open it locally:

open playground/index.html

The playground lets you:

• paste prompt text
• select a model
• view token estimates
• view estimated cost
• view context window usage

Supported Models

The library uses provider-family heuristics based on the model name string.

OpenAI

Matches model names containing:

• gpt
• o1
• o3
• o4
• text-embedding
• openai

Examples:

• gpt-4o
• gpt-4.1-mini
• o1
• o3-mini
• text-embedding-3-small

Default family context window used by this package: 128000

Gemini

Matches model names containing:

• gemini
• google

Examples:

• gemini-1.5-pro
• gemini-1.5-flash
• gemini-2.0-flash
• google/gemini-2.0-flash-thinking

Default family context window used by this package: 1000000

Claude

Matches model names containing:

• claude
• anthropic

Examples:

• claude-3-haiku
• claude-3-opus
• claude-3-5-sonnet
• anthropic/claude-3-7-sonnet
• sonnet-4
• haiku-3.5
• opus-4
• opus 4.6

Default family context window used by this package: 200000

If a model name does not match one of these families, countTokens throws a RangeError.

Why it is approximate

This package uses heuristic estimation rather than provider-native tokenizers. That keeps it lightweight and easy to drop into any Node.js project, but it also means:

• counts are best used as rough planning values
• provider-side totals may differ from local estimates
• exact billing or hard token limits should still be validated against the provider when precision matters

Example

Run the included example:

node example.js

Test

Run the built-in test suite:

npm test

Repository

This repository includes:

• README.md for package documentation
• CONTRIBUTING.md for contribution guidelines
• CHANGELOG.md for release notes
• CODE_OF_CONDUCT.md for community expectations
• SECURITY.md for security reporting guidance
• LICENSE for licensing terms

Contributing

Contributions are welcome, especially if they improve the estimation heuristics while keeping the library simple and dependency-free.

To contribute:

1. Fork the repository.
2. Create a feature branch for your change.
3. Make your changes and keep the package dependency-free unless there is a clear reason not to.
4. Run npm test.
5. Update the README or examples if behavior changes.
6. Open a pull request with a clear explanation of what changed and why.

Good contribution ideas:

• Improve estimation heuristics for additional OpenAI or Claude model naming patterns
• Add edge-case tests for mixed-language or punctuation-heavy prompts
• Improve docs and examples for real-world prompt budgeting workflows

Publishing

The package includes the required metadata to publish on npm. Before publishing:

1. Update the version in package.json.
2. Run npm test.
3. Ensure you are logged in with npm login.
4. Run npm publish --access public.

License

MIT