Choosing a Model

A team builds a feature that sorts incoming support tickets into five categories. They pick the most capable model their provider offers, because it is "the best". It works well. Then the first monthly bill arrives, and users complain that the ticket form hangs for five seconds after they press Send.

They try a smaller model on fifty old tickets. It labels them almost as well, answers in about a second, and costs a fraction as much per ticket. Meanwhile, the team next door used a small, fast model to review contracts and kept missing unusual clauses. Both teams made the same mistake in opposite directions: they chose a model before they knew what the task needed.

#There is no best model, only a best fit

Every model choice trades a few things against each other. Bigger, more capable models tend to follow long instructions better, handle subtle cases and reason through harder problems. They are also slower and cost more per token. Smaller models are fast and cheap and often perfectly good at narrow, well-defined tasks.

Some models also offer a mode where they "think" longer before answering (see Give the Model Room to Think). That moves the same trade-off along a dial: better answers on hard problems, at the price of more time and more tokens. For easy tasks it often adds cost without improving anything.

Cost is easy to underestimate because it multiplies. A model that costs five times more per request barely matters for ten requests a day. For a feature that runs on every one of 30,000 tickets a month, the same factor of five is the difference between a rounding error and a budget line someone will ask about.

#Let your own tests decide

Public rankings measure models on someone else's tasks. Your tickets, your documents and your users are different. The reliable way to choose is to build a small test set from real examples with known right answers, run each candidate on it, and compare. Evals covers how to build a good one; fifty labelled examples is a reasonable start for a simple task.

For the ticket team, the bar was: at least 90% labelled correctly, and a reply in at most 2 seconds. Here is the choice as code.

// Results from running each candidate on your own 50 test tickets.
const candidates = [
  { name: "large", accuracy: 0.96, seconds: 4.8, cost: 20 },
  { name: "small", accuracy: 0.84, seconds: 0.6, cost: 1 },
  { name: "medium", accuracy: 0.92, seconds: 1.4, cost: 4 },
];
 
function chooseModel(candidates, minAccuracy, maxSeconds) {
  const byCost = [...candidates].sort((a, b) => a.cost - b.cost);
  for (const model of byCost) {
    if (model.accuracy >= minAccuracy && model.seconds <= maxSeconds) {
      return model.name; // cheapest one that clears every bar
    }
  }
  return null; // nothing is good enough: rethink the task or the bar
}
 
chooseModel(candidates, 0.9, 2); // "medium"

# Results from running each candidate on your own 50 test tickets.
candidates = [
    {"name": "large", "accuracy": 0.96, "seconds": 4.8, "cost": 20},
    {"name": "small", "accuracy": 0.84, "seconds": 0.6, "cost": 1},
    {"name": "medium", "accuracy": 0.92, "seconds": 1.4, "cost": 4},
]
 
def choose_model(candidates, min_accuracy, max_seconds):
    for model in sorted(candidates, key=lambda m: m["cost"]):
        if model["accuracy"] >= min_accuracy and model["seconds"] <= max_seconds:
            return model["name"]  # cheapest one that clears every bar
    return None  # nothing is good enough: rethink the task or the bar
 
choose_model(candidates, 0.9, 2)  # "medium"

The names are placeholders for whatever your provider offers, and cost is in relative units: "small" costs 1 per ticket, "large" costs 20. Step through the decision.

sort candidates by cost, cheapest firstfor each model:  if accuracy >= 0.90 and seconds <= 2:    return modelreturn none

If nothing passes, do not just lower the bar. Ask whether the task can be made easier: clearer instructions, a few examples in the prompt, or splitting one hard request into two simple ones. A better prompt often moves a cheaper model over the line.

#Mixing models

You do not have to pick one model for everything. A common pattern is routing: a small model handles the easy, high-volume cases, and anything it marks as unsure goes to a bigger one. Another is to use a capable model to write a plan and a cheaper one to carry out the simple steps. Each split adds moving parts, so do it when cost or speed really demands it.

#Try it yourself

#Common mistakes

#Check yourself

#What to practise next

Pick one task you use AI for regularly and write its bar in numbers: how often it must be right, how fast it must be, and what data it touches. Then read Common Mistakes, which pulls together the habits from this whole module.