AI 101: The Hidden Decisions Behind an AI Answer

TL;DR: AI routing decides which model or tool should do the work, balancing quality, cost, and speed. The hard part is predicting the best choice before seeing the answer and passing along the right context. New research also asks how to verify the route and the model’s computation. Neither proof, on its own, guarantees a good answer.

When you ask an AI a question, you might have already chosen the model you want to talk to: Opus 5.5, GPT 6.1-Terra, Gemini Flash etc. But depending on the system, that doesn’t necessarily mean all the decisions have been made.

Your request might be simple enough for a quick response, or it might need more time and reasoning. It might call for a web search, a calculation, or a piece of code. And if you’ve given an agent a larger task, it may have to decide which tools to use and which parts of the work to hand off.

These choices fall under a concept called routing, and the more you look at it, the more interesting it becomes. Because how do you know which model, tool, or agent is right for a job before you’ve actually done the job?

The answer is that you don’t, at least not with perfect certainty. You can make an educated guess based on the request, past performance, cost, speed, and a few other signals. But a question that looks easy can turn out to be surprisingly difficult, while an intimidating wall of text might only need a simple answer. AI systems, it turns out, also have to deal with appearances being deceiving.

In this guide, we’ll explore how good that guess can get. We’ll start with Route Anything, where researchers use past results to predict which model will do best. And then we will throw some world models into the mix and how they can affect desicion-making. It is super interesting, and we will also answer this question: If you paid for one model and a cheaper one answered, how would you know?

Let’s go!

In today’s episode:

AI routing is the process of choosing which model, tool, or policy should handle a particular input. Depending on the system, that choice might follow a fixed rule or a prediction about which candidate is likely to do the job well within the time and budget available.

The basic path looks simple:

Input → router → selected model or tool → output

The difficult part is deciding which path to take, especially when the router has to make that call before it has seen the result.

A router is the component that makes this choice. In model routing, it sits outside the candidate models and chooses which one should handle a request. It can be a set of rules, a classifier, a specialized neural network, or another language model.

For example, a system could send an image to a model that can process images, a coding task to a model that performs well on code, and a straightforward classification task to a smaller model. Routing can also happen between image generators, speech models, or robot policies, which turn observations into actions.

Suppose you upload a photo and ask, “Why are my tomato leaves turning yellow?” A router might start by narrowing the list to models that can process the photo. A classifier estimates what kind of task this is and how difficult it looks. The router then weighs expected answer quality against cost and sends the request to its first choice. If that model’s provider is unavailable, the router tries the next model on the list.

You may remember that Mixture-of-Experts models also have routers. Those work at a different level: a network inside the model selects which experts process each token. Here, we’ll mostly focus on choices between whole models, tools, or policies.

There are two big reasons. First, different models and tools have different strengths, so choosing a suitable one can improve the result. Second, the most expensive option isn’t always necessary, and using a smaller or faster model when it can do the job well may save money and time.

But there’s an obvious trap here. Suppose a cheaper model costs one-tenth as much as a more expensive one and succeeds 15% of the time. If the expensive model succeeds every time, the cheaper one produces 50% more correct answers per dollar. Sounds efficient, until you remember that it also fails 85% of the time.

Most people would not like that.

That’s why a router needs more than a cost target. It may also need a quality floor: a minimum level of performance the system is expected to maintain. The interesting question is how to meet that requirement without sending every request to the most expensive candidate.

To make a choice, the router needs something to go on. That could be a task category, the full prompt, an image, the state of a robot, or feedback from an earlier attempt.

The simplest approach is to route by task type. First, classify the request, then send it to the candidate that usually performs well in that category. This is relatively easy to inspect, but it also has an obvious limitation: two coding tasks can be very different. One might need a small edit; the other might involve finding a bug that has been hiding from an entire engineering team for three weeks.

A more detailed approach is to predict each candidate’s outcome on the specific input. The router estimates how well each option is likely to perform, compares those predictions, and makes its choice.

The Route Anything paper explores this approach by training routers on recorded outcomes. The training data shows how different candidates performed on different inputs. The router then learns to predict those scores for a new input, before paying to run all the candidates on it.

When cost is part of the decision, the trade-off can be expressed like this:

Routing score = predicted quality − λ × predicted cost

Here, λ determines how much weight the system gives to cost. When it’s zero, the router considers predicted quality alone. Increase it, and a more expensive candidate needs to offer a larger expected quality advantage to justify its selection.

But even before we decide how much cost matters, there’s another question: what should we train the router to predict?

It might seem natural to train a router to identify the candidate most likely to win. But that can throw away useful information.

Can the router make the right call, and can you trust it? In the full guide, we follow context between models, explore why trained routers fall short, and look at world-model simulations, FlashProve, and VeriRouter. Choosing a route is only half the story.

Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI. 

AI routing chooses which model, tool, or policy handles a task. Depending on the system, it may use rules or predictions to balance the task’s requirements with expected quality, cost, and speed.

It can follow rules, classify the task, predict each candidate’s performance, or try a cheaper model and escalate after checking the result. The decision and any verification steps have costs of their own.

No. In a typical API-based system, the application must supply the relevant history, a summary, current files, or a way to retrieve them. Choosing the next model and preparing its context are separate jobs.

A benchmark oracle chooses after seeing the candidates’ results. A trained router has to predict before seeing them. Some oracle gains also come from run-to-run variation that may be impossible to predict reliably.

No. FlashProve checks consistency with a committed model’s computation. VeriRouter checks whether a router followed agreed rules and relayed the request and response faithfully, assuming an honest model provider. Answer correctness still needs separate evaluation.