Token counter and tokenizer for GPT, Claude and Gemini

What is a token?

A token is the unit a language model reads, writes and bills: a word, part of a word, a punctuation mark, a run of spaces or a few bytes of a character. Prices, rate limits and context windows are all set in tokens, not words.

Tokenizers build their vocabulary with byte-pair encoding: frequent sequences such as " the" become one token, rare ones are spelled out from smaller pieces. OpenAI's o200k_base has 200,019 token IDs, the older cl100k_base 100,277, and a bigger vocabulary usually means fewer tokens, above all outside English.

How many tokens is a word?

In English, about 1.15 tokens per word for plain prose with OpenAI's o200k_base, and more when the text carries numbers, JSON or code. Other languages need more tokens for the same meaning.

Measured with this tool on 17 October 2026, the English Universal Declaration of Human Rights, 1,753 words, is 2,017 o200k_base tokens (1.15 per word); the tool's opening prompt, with a line of JSON, is 98 tokens for 67 words (1.46); the Python example is 84 for 40 (2.10). Translations say the same thing, so they compare languages fairly. Words come from the browser's word segmenter, which splits Chinese and Thai into dictionary words.

Vietnamese needs 3.4 times the tokens of English, and cl100k_base is far worse outside Latin script: Tamil takes four times as many tokens there as on o200k_base. Vendors publish rules of thumb too: Google says 100 Gemini tokens are about 60 to 80 English words, and Anthropic says 1M tokens on Claude's current tokenizer is roughly 555,000 words.

Why do token counts differ between models?

Each model family has its own tokenizer, so the same text counts differently on each.

The OpenAI rows come from tiktoken's model map, which stops at GPT-5; OpenAI has not said which encoding GPT-6 uses. o200k_harmony adds the special tokens of gpt-oss's harmony format to o200k_base, so only markers such as <|start|> split differently. Claude changed tokenizer with Opus 4.7, which Anthropic says produces about 30% more tokens for the same text. API counts also include formatting tokens for message roles, as OpenAI's guide notes.

Can you count Claude and Gemini tokens offline?

Not in a browser today. Anthropic publishes no tokenizer for current models (its old TypeScript tokenizer says it stopped being accurate with Claude 3), and Google's Gen AI SDK counts locally only after downloading tokenizer data this page does not ship.

So the tool shows a range from the vendors' published equivalences: Claude's current tokenizer, characters ÷ 2.5 and words ÷ 0.555; earlier Claude models, words ÷ 0.75; Gemini, characters ÷ 4 and words ÷ 0.6 to 0.8. The cost panel uses the upper end. These are English rules of thumb, so there is no estimate when most letters are in another script.

How do you count tokens in code?

Use tiktoken locally for OpenAI encodings, and each vendor's counting endpoint for a whole request.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # or tiktoken.encoding_for_model("gpt-4o")
print(len(enc.encode("tiktoken is great!")))  # 6

from openai import OpenAI

client = OpenAI()
response = client.responses.input_tokens.count(
    model="gpt-6-astra",
    input="Tell me a joke.",
)
print(response.input_tokens)

import anthropic

client = anthropic.Anthropic()
response = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": "Hello, Claude"}],
)
print(response.input_tokens)

from google import genai

client = genai.Client()
response = client.models.count_tokens(
    model="gemini-3.8-flash",
    contents="The quick brown fox jumps over the lazy dog.",
)
print(response.total_tokens)

Checked 17 October 2026 against OpenAI's tiktoken cookbook, OpenAI's token counting guide, Anthropic's docs (counting is free within rate limits) and Google's tokens guide.

How do token counts drive cost and context limits?

Cost is input tokens times the input price plus output tokens times the output price, and output usually costs several times more. The context window caps both together.

Worked example: the tool's opening prompt is 98 o200k_base tokens. On GPT-5.4 mini at US$0.75 per 1M input and US$4.50 per 1M output tokens (OpenAI's list price, 17 October 2026), a 500-token reply costs US$0.00225 against US$0.00007 for the prompt: about US$0.0023 per request, US$2.32 per 1,000, with the reply 97% of the bill. It uses 598 of the 400,000-token window, 0.15%.

The cost panel also applies long-prompt rates: OpenAI bills prompts over 272,000 input tokens on GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna at twice the input and 1.5 times the output rate for the whole request, and Claude Haiku 5.5 costs more above 100,000 prompt tokens. Reasoning counts as output: Google's price list says output prices include thinking tokens.

Common mistakes when counting tokens

• Counting Claude or Gemini text with tiktoken. Their tokenizers differ; treat the figure as an estimate.
• Counting only the user message. System prompt, tool schemas and history are input, resent every turn.
• Pricing only the input. Output and reasoning tokens usually dominate.
• Assuming English ratios. Vietnamese needs 3.4 times the tokens of English.
• Reusing counts after a model change. Anthropic advises recounting for Claude Opus 4.7 and later.
• Leaving no room for the reply. Prompt plus maximum output must fit the window.