The AI meter you can't fully see

I kept watching AI coding sessions burn through way more tokens than I expected. So I went down a rabbit hole to figure out what I was actually paying for.

The short version: "token" is doing two jobs. It's a unit of information the model works with, and it's a unit providers bill by. Those overlap, but they aren't the same thing. A token count tells you how much material got processed. It can't tell you how much reasoning happened, how much compute it took, or what a specific app will charge.

And some of the expensive part happens where you can't see it.

Here's what I found, plus the rules I'm using now.

1. A token isn't a word

• A token can be a whole word, part of a word, a single character, or punctuation.
• Spaces, capitalization, and the tokenizer itself all change how text gets split.
• OpenAI's English rule of thumb is about 4 characters, or roughly three-quarters of a word, per token. Good for a quick guess. Not a conversion formula.

"cat" might map to one token ID. A rare name, a weird string, or a chunk of code might split into several. Under the hood, the model works with sequences of numbers that point to learned representations, not tiny dictionary cards with one fixed meaning.

It's not only text, either. Google's Gemini docs count images, audio, and video in tokens too, with different rules for each. "Token" is the accounting unit. What it actually represents depends on the input and the provider.

2. Why split words up at all?

This started as a translation problem, not a billing one. In 2016, Sennrich, Haddow, and Birch tackled models that choke on unfamiliar names, compounds, and rare terms because they only know a fixed list of words. Their fix: break words into smaller subword pieces, adapting byte pair encoding (BPE) to build the vocabulary.

The tradeoff in one line each:

• Every word gets its own entry: the vocabulary gets ridiculous.
• Everything becomes single characters: sequences get really long.
• Subwords: the middle ground.

In 2018, Kudo and Richardson's SentencePiece pushed it further by learning straight from raw sentences, with no language-specific word splitting first. That matters because the word boundaries that feel obvious in English aren't universal rules of language.

Takeaway: a token isn't a natural unit hiding in your sentence. It's a design choice made by a specific tokenizer.

3. What happens when you hit Send

Stripped down, for a Transformer-style text model:

• Your text becomes token IDs.
• The IDs become number vectors called embeddings, plus info about their order.
• Layers of computation let each position use the surrounding context.
• The model scores the next token, picks one, adds it to the context, and repeats. And repeats.

That's the idea from the original Transformer paper, with GPT-2 as an early OpenAI example of a model that generates one token at a time. Architectures vary, but the point holds: tokens aren't just counted and tossed in a bucket. Each one takes part in the computation, and an answer takes many steps.

Quick side note: this isn't training. Training is when the model's parameters are learned. Your prompt asks an already-trained model to respond, and it doesn't retrain the model on the spot. Whether a company later uses API data to improve models is a separate data-policy question.

4. Why the bill is in tokens

OpenAI, Google, Anthropic, and xAI all publish API pricing by token category. Input, output, and cached input can each have their own rate, and long context, tools, and other features can add their own rules. So "a token costs X" is already too simple.

Why tokens? They're countable units of input and generation, which makes them a practical meter for model activity. OpenAI even describes token-based credits for some ChatGPT and Codex products as a direct mapping between model activity and credit use.

What I couldn't find: a clear source on why the industry settled on tokens instead of requests, characters, seconds, or raw compute. So that part is my inference, not a proven fact.

And tokens aren't interchangeable dollar chips. A million cached input tokens can cost less than a million fresh ones. Output can cost something different from input. The model, the tools, and the provider all matter. "How many tokens did I use?" is only the start of the billing question.

5. The tokens you don't see

This is the part that sent me down the hole: a short answer on screen, a surprisingly big usage report.

Reasoning models can generate internal thinking before or between the words you actually see:

• OpenAI: reasoning tokens count toward output and are billed as output tokens.
• Anthropic: Claude's thinking tokens are billed as output, even when the full thinking text isn't returned.
• Google: Gemini reports thinking tokens separately in API usage.
• xAI: shows reasoning-token counts for supported Grok models and describes high and xhigh effort as deeper-reasoning settings.

Different details, same lesson: the neat paragraph on your screen may not be the whole output meter.

Effort isn't a "higher is better" switch, either. Anthropic calls it soft guidance: it nudges how readily and deeply Claude thinks, with no promised number of extra tokens. xAI says reasoning support and behavior vary by model. Higher effort can mean more reasoning, more time, and a bigger API bill. Often useful. Not a guarantee of a better answer. I haven't run a controlled low-vs-high test yet, so I'm not going to fake a conclusion.

One more wrinkle: app credits aren't API billing. OpenAI's current ChatGPT rate card lists some experiences at a fixed credit cost per message, and for the listed GPT-6 settings, moving from Medium to High or Extra High doesn't change that per-message rate. If credits seem to vanish inside a specific app, check that app's rules. API token math alone may not explain it.

My playbook for spending less

1. Read the whole usage record, not just the answer. Check input, output, reasoning or thinking, cached input, and tool use wherever the provider shows them. OpenAI warns that a plain-text tokenizer view can miss message formatting, schemas, files, and images. If you're seriously comparing cost, inspect the actual request and response.
2. Give it what it needs, then stop. Cut duplicated instructions and irrelevant documents. Try a lower effort level on simple tasks and save deep thinking for work that benefits from it. Keep the requested output focused.
3. Use caching on purpose. If you keep sending the same prompt opening, a provider may reuse earlier processing and charge a lower cached-input rate. Per OpenAI's prompt-caching guide, what gets reused is the computed key/value state, not just a saved copy of your text. Stable instructions and reference docs can get cheaper across repeated requests.
4. Watch for cache traps. Two prompts that look identical to us don't guarantee a cache hit. Change earlier content or certain settings and you break the reusable prefix. Shortening a prompt can even cost more if it loses enough discounted cached input. And in a chained API conversation, OpenAI says prior inputs are still billed as input. Look at the cost of the whole conversation, not one turn.
5. Measure cost per successful task. Count retries. Count tools. Count time. Then ask whether the answer was actually useful. A cheap answer you retry three times can cost more than getting it right once.

The rule in one line: measure cost per working answer, not cost per visible paragraph.

What I'm testing next

Same task, same model, several effort levels, repeated runs, then grade the answers blind. Until that's done, here's the cleanest conclusion I've got: tokens are what models work with and what providers bill around, and some of the expensive work happens where you can't see it.

How are you tracking AI cost on your own projects?

Sources

1. OpenAI: Understanding and counting tokens. Core definition and token accounting.
2. Sennrich et al. (2016): Neural Machine Translation of Rare Words with Subword Units. Subword byte pair encoding.
3. Kudo & Richardson (2018): SentencePiece. Tokenization from raw text and alternative segmentation methods.
4. Vaswani et al.: Attention Is All You Need, and OpenAI: GPT-2 paper. Model processing and next-token generation.
5. OpenAI: Reasoning models. Reasoning effort, hidden reasoning tokens, and API usage.
6. Anthropic: Extended thinking, and Thinking, effort, and cost. Thinking tokens and effort controls.
7. Google: Understand and count tokens, and Gemini thinking. Multimodal token accounting and thinking-token usage.
8. xAI: Reasoning, and Best practices. Model-specific reasoning levels.
9. OpenAI: Prompt caching. Repeated-input caching and its cost effects.
10. OpenAI: API pricing, and ChatGPT Business/Enterprise/Edu rate card. API price categories and the separate ChatGPT credit model.

Originally published on X.