AI 101: LLM Token Types: Input, Output, Reasoning, Cached & More

Last time (AI 101: What is a Token?) we covered what a token is and how tokenization turns text/video/audio into something a model can process. That was the foundation. Now let's talk about what actually happens in production – because in production, there is no such thing as just "a token" anymore.

Jensen Huang recently said that the AI business is about transforming electrons into tokens. That framing is exactly right – but it is incomplete. Because "tokens" is no longer one thing. A single API call can now involve input tokens, output tokens, reasoning tokens, cached tokens, tool-use tokens, and vision tokens – each billed differently, each consuming compute in a different way. If you want to make good decisions about AI systems and how you are spending your electrons, you need a taxonomy: a clear map of what all these tokens actually are and why they behave so differently.

So thank you, Jensen, for the inspiration. Here is my taxonomy. Let me walk all of you through the full token zoo, species by species. I’m pretty sure you’ll find some you’ve never heard before (and still, you are paying for them).

Surprising result: you'll never look at an API pricing page the same way.

In today’s episode:

Every API call has two sides: what you send in (input tokens) and what comes back (output tokens). And as you might guess, generating text is computationally harder than reading it.

When the model processes your input, it can do it in parallel. All the tokens in your prompt get processed more or less at once, in what is called the prefill phase. The model builds up its internal representation of everything you said in a single forward pass.

Output is different. The model generates tokens one at a time, each one depending on the one before it. This is autoregressive generation, and it is inherently sequential. Each new token requires a separate forward pass through the model – or at least through the decoder layers. That is why output tokens are more expensive. They require more compute per token.

This is also why you see the price difference on every API pricing page. As of mid-2026, output tokens typically cost 2x to 6x more than input tokens, depending on the provider and model. That ratio reflects a real difference in how the hardware is used.

A practical implication: if you can restructure a task to reduce output length without losing quality – for example, by asking for structured JSON instead of verbose explanations – you are already saving.

This is the category that has emerged most dramatically since 2024. Reasoning tokens – also called thinking tokens – are tokens the model generates internally as part of a chain-of-thought process before producing its final answer.

When you use models with extended thinking, the model does not jump directly to the answer. It first "thinks through" the problem, producing an internal monologue that may be partially or fully hidden from you. Those intermediate tokens still consume compute. They still cost money. And they can massively inflate the total token count of a response.

Here is what makes reasoning tokens interesting from an economic standpoint:

Some providers are now exposing reasoning token counts as a separate line item in their API responses. Others fold them into the output token price. This lack of standardization makes cost comparison between providers much harder.

Jensen Huang explicitly named reasoning tokens as a distinct pricing category in a recent conversation with Dwarkesh Patel. And the interesting thing here is that the token has become a segmented product, not a single commodity.

Here is some practical tips from Boris Cherny who builds Claude Code:

If reasoning tokens are expensive because they add thinking, speculative tokens are strange for the opposite reason: they are generated specifically so that most of them can be thrown away.

This is one of the most counterintuitive ideas in the token zoo, and by 2026 it has become production-standard at most major inference providers.

Here is the problem it solves →

Autoregressive generation, as we described above, is sequential: one token at a time, each depending on the last. That is slow. The hardware sits partially idle waiting for each step to finish before starting the next.

Speculative decoding attacks this bottleneck with a clever trick. Instead of generating one token at a time with the full, expensive target model, a smaller and cheaper draft model races ahead and proposes several candidate tokens in parallel – sometimes 5, sometimes 10 or more. The target model then verifies them all at once in a single forward pass, keeping only the tokens that match its own distribution and discarding the rest.

The result is often 2–3× lower latency with only modest extra compute overhead. The target model's output quality is mathematically preserved – you get the same distribution of answers, just faster – because any draft token that does not match gets rejected and regenerated properly.

What makes speculative tokens interesting as a category is that they are real tokens. They are generated, they consume compute, they flow through the pipeline. But they are invisible to the user, and many of them are intentionally discarded. They exist as a speed optimization, not as content.

Several variants have emerged in the area of speculative decoding. EAGLE uses the target model's own hidden states to draft more accurately. Tree-based methods like SpecInfer propose branching candidate sequences rather than a single linear draft, increasing the chance that at least one path survives verification. Medusa adds extra prediction heads directly onto the target model, eliminating the need for a separate draft model entirely.

For users, speculative decoding is mostly invisible – you just notice faster responses. But for anyone managing inference infrastructure, it is one of the most important levers for latency and throughput. It also complicates the notion of "tokens generated," because the system may generate and discard 30% more tokens than the user ever receives.

On the other end from reasoning tokens – which add compute – cached tokens subtract it. Prompt caching (or context caching) lets the model reuse work it has already done on identical input.

Here is how it works. When you send a prompt, the model processes it through its layers and builds a set of internal representations called the KV cache (key-value cache). If you send the same prompt prefix again – for example, the same system prompt, the same long document, the same set of instructions – the model can skip the computation for that part and reuse the cached KV values.

This matters enormously for production workloads. Consider an agentic system where every call includes a 4,000-token system prompt, a 10,000-token document, and a variable 200-token user query. Without caching, you recompute the system prompt and document on every single call. With caching, those 14,000 tokens are processed once and reused.

How providers price cached tokens:

The savings compound fast. A company running 10,000 API calls per day with a shared 8,000-token prefix can cut input token costs by an order of magnitude.

The catch: caches expire. They are typically held for minutes to hours, depending on the provider and the load on their serving infrastructure. If your traffic is bursty or infrequent, caching may not help much.

There is also cross-user KV caching, which is newer and more aggressive. If a thousand agents share the same system prompt, the serving infrastructure computes the KV cache once and makes it available to all of them. This is where the real infrastructure-level efficiency gains are happening, and it is one reason inference providers are racing to differentiate on caching strategy.

This is the category most people underestimate. When a model uses tools – function calling, API integration, code execution, web search – it is consuming tokens you never see directly.

A concrete example: an agent with a 3,000-token system prompt, accessing 8 tools (2,500 tokens of schemas), processes a user query in 6 loops. By the final loop, the context might contain 30,000+ tokens. Total tokens consumed across all loops? Potentially 100,000+, even though the user sent a 50-token question and received a 300-token answer. This is the hidden cost of agentic AI. The token bill for an agent workflow can be 50x–200x what a single-turn conversation costs. Companies scaling agent deployments are learning this the hard way.

Cognition's Devin, for example, prices its work in Agent Compute Units (ACUs) — a normalized measure of resource consumption per task — rather than per token directly, reflecting exactly this reality: agentic workloads cost orders of magnitude more than single-turn calls. See how Cognition AI prices Devin

Tokens are NLP natives, but they have escaped into other modalities. Every modality gets tokenized differently, and those differences create dramatic cost variations.

When you send an image to GPT, Claude, or Gemini, the model does not "see" the image the way you do. The image is divided into a grid of patches – small rectangular regions – and each patch becomes a token (or a small number of tokens). A high-resolution image can easily produce 1,000–2,000 tokens. A detailed infographic might push past 3,000.

This means uploading a screenshot for a simple question can cost more than a full page of text input. The image resolution directly controls token count: higher resolution = finer grid = more patches = more tokens. Most providers now offer a "low detail" mode that downscales the image first, producing fewer tokens at the cost of fidelity.

The architectural trick behind vision tokens is the vision encoder (often a ViT – Vision Transformer). It processes the image into patch embeddings, which are then projected into the same embedding space the language model uses. From the model's perspective, an image patch and a text token are the same kind of object once they enter the transformer.

Audio tokenization takes a different path. Models like Whisper first convert audio into a spectrogram – a visual representation of frequencies over time – and then chop that spectrogram into frames. Each frame becomes a token. For speech, this typically means one token per 20–40 milliseconds of audio.

A 30-second audio clip might produce 750–1,500 tokens. A one-hour meeting recording? 90,000–180,000 tokens. This is why audio-native AI is expensive and why most real-world systems transcribe first and process text second.

Native multimodal models like Gemini can process audio tokens directly alongside text, which preserves tone, emphasis, and non-verbal cues but at a much higher token cost than a transcript.

Video is the most token-hungry modality. A video is essentially a sequence of images (frames) plus an audio track. Naive tokenization would produce frame_count × patches_per_frame tokens, which is enormous. A 10-second video at 30fps with 256 patches per frame would be 76,800 tokens – for ten seconds.

In practice, models use temporal compression. They sample keyframes, track changes between frames, and only tokenize the differences. But even with compression, video input remains orders of magnitude more expensive than text.

This is one reason video understanding is still an emerging capability. The token budget required makes it impractical for many applications at current prices.

Code is text, but it tokenizes differently than natural language. Whitespace, indentation, brackets, and syntax characters are common in code but rare in prose. Many tokenizers handle them inefficiently, splitting what a human sees as a single indent into multiple tokens.

This has practical consequences: the same logic expressed in Python (significant whitespace) and C++ (braces) can produce noticeably different token counts. Languages with verbose syntax (Java, for example) tend to be more token-expensive than terse ones (Python, Go).

Some newer models have tokenizers explicitly trained on code, which improves efficiency for programming tasks. But it is still common for a 500-line Python script to consume 3,000–5,000 tokens.

Every model uses special tokens that control behavior but carry no semantic content. These are rarely discussed, but they are always present.

You do not normally see these tokens. But they are why a "128K context window" does not mean you get exactly 128,000 tokens for your content. Some of that budget is consumed by structural tokens that the system requires.

All of these token types create a landscape where the same question – "How much does it cost to run this AI system?" – has a complicated answer. A few key implications:

And there is a deeper architectural point. The token types we have today – reasoning, speculative, cached, multimodal, tool-use – are not fixed. They are an artifact of current architectures. If a future model processes text and images through a fundamentally different mechanism, the token taxonomy will change with it. What will not change is the core reality: AI computation is metered, and the meter runs in tokens.

Understanding what a token is matters. But in production, what matters even more is knowing how to budget for the different species. Reasoning tokens multiply your bill in exchange for better answers. Speculative tokens are born and discarded by the millions, invisible but essential for speed. Cached tokens shrink costs when you can reuse context. Tool-use and retrieval tokens hide in the background, quietly dominating agentic workloads. Multimodal tokens let models see and hear, but at wildly different cost-per-information rates. And structural tokens consume context window space just to keep the system functioning.

Picking the best model is one thing. But the smartest teams are going further: into managing their token portfolio – routing tasks to the right model with the right token profile, caching aggressively, compressing where they can, and measuring what each token actually buys them. The next frontier may be tokens that delete themselves: dynamic pruning of low-value tokens mid-inference, freeing up context window space and compute. Still early, but it points toward a future where the token zoo is managed by the model itself.

We can see a future where the same model serves the same basic unit, tokens, at different price points depending on latency. Fast tokens for code completions cost more. Slow tokens for batch jobs cost less. That is closer to a commodity market, where pricing depends on delivery conditions and performance tier, than to traditional software, where the product is usually sold as a fixed package. As that happens, AI product design becomes partly a market design problem: you have to decide which interactions deserve premium real-time inference and which can be pushed into cheaper, delayed workflows. For anyone building AI products, that means architecture stops being only a technical question and becomes an economic one too. That is genuinely new.

Sources and further reading

Turing Post

Papers:

Docs: