Evals

You run a small online shop, and an agent reads incoming support messages. It picks a category, decides whether the message is urgent, and drafts a reply. You tweak its instructions so it stops being so formal, try three messages, and the replies look great. You ship it.

A week later you find out that double charges have been landing in the normal queue since the change. Your three test messages never included one. You did not test the agent; you glanced at it.

#The idea: a test set and a grader

An eval is a test for an AI system. It has the same parts as any software test:

• Cases: inputs the system will really see.
• Expectations: for each case, what a good output must have.
• A grader: something that compares the output to the expectation and says pass or fail.

Unlike normal tests, the output is not fixed: the wording changes between runs, and many answers are acceptable. So you do not compare the reply with a stored string. You check the properties that matter: the category, the urgency, whether the reply mentions the order number.

#Build a small test set

Ten to twenty good cases are enough to start. Take them from three places:

1. Real inputs. Messages the system has actually received, with personal details removed. They have the typos and odd phrasing you would never invent.
2. Known failures. Every bug a user reports becomes a case, so it can never come back unnoticed.
3. Edge cases. An empty message. A message in another language. A message that tries to give the agent orders. A message that fits two categories.

Write down each expected result before you run anything, or you will tend to accept whatever comes out.

#A worked example

Here is the shop's eval: six cases, graded on three checks. The category must match, the urgent flag must match, and if the message names an order number the reply must mention it. A case passes only if every check passes.

Two lessons hide in the last step. A single score hides what changed, so always look at which cases failed. And case 6 failed in both versions: that is prompt injection, found by a test before an attacker found it.

#The grader in code

Checks like these are plain code: fast, free and the same every time. Here is a grader and a runner, with canned replies standing in for the agent.

function grade(output, expect) {
  const failures = [];
  if (output.category !== expect.category) {
    failures.push(`category is ${output.category}, expected ${expect.category}`);
  }
  if (output.urgent !== expect.urgent) {
    failures.push(`urgent is ${output.urgent}, expected ${expect.urgent}`);
  }
  if (expect.mention && !output.reply.includes(expect.mention)) {
    failures.push(`reply does not mention ${expect.mention}`);
  }
  return failures;
}
 
function runEval(agent, cases) {
  let passed = 0;
  for (const { input, expect } of cases) {
    const failures = grade(agent(input), expect);
    if (failures.length === 0) passed++;
    else console.log(`FAIL "${input}": ${failures.join("; ")}`);
  }
  console.log(`${passed} of ${cases.length} passed`);
  return passed;
}

def grade(output, expect):
    failures = []
    if output["category"] != expect["category"]:
        failures.append(f"category is {output['category']}, expected {expect['category']}")
    if output["urgent"] != expect["urgent"]:
        failures.append(f"urgent is {output['urgent']}, expected {expect['urgent']}")
    if expect.get("mention") and expect["mention"] not in output["reply"]:
        failures.append(f"reply does not mention {expect['mention']}")
    return failures
 
def run_eval(agent, cases):
    passed = 0
    for case in cases:
        failures = grade(agent(case["input"]), case["expect"])
        if not failures:
            passed += 1
        else:
            print(f'FAIL "{case["input"]}": {"; ".join(failures)}')
    print(f"{passed} of {len(cases)} passed")
    return passed

const cases = [
  { input: "Charged twice for order 2210", expect: { category: "billing", urgent: true, mention: "2210" } },
  { input: "Mug arrived cracked, order 1042", expect: { category: "damaged", urgent: false, mention: "1042" } },
  { input: "hi", expect: { category: "other", urgent: false } },
];
// Canned replies standing in for the real agent.
const canned = {
  "Charged twice for order 2210": { category: "billing", urgent: false, reply: "Sorry about order 2210..." },
  "Mug arrived cracked, order 1042": { category: "damaged", urgent: false, reply: "Sorry the mug in order 1042..." },
  "hi": { category: "other", urgent: false, reply: "Hi, how can I help?" },
};
runEval((input) => canned[input], cases);
// FAIL "Charged twice for order 2210": urgent is false, expected true
// 2 of 3 passed

cases = [
    {"input": "Charged twice for order 2210", "expect": {"category": "billing", "urgent": True, "mention": "2210"}},
    {"input": "Mug arrived cracked, order 1042", "expect": {"category": "damaged", "urgent": False, "mention": "1042"}},
    {"input": "hi", "expect": {"category": "other", "urgent": False}},
]
# Canned replies standing in for the real agent.
canned = {
    "Charged twice for order 2210": {"category": "billing", "urgent": False, "reply": "Sorry about order 2210..."},
    "Mug arrived cracked, order 1042": {"category": "damaged", "urgent": False, "reply": "Sorry the mug in order 1042..."},
    "hi": {"category": "other", "urgent": False, "reply": "Hi, how can I help?"},
}
run_eval(lambda text: canned[text], cases)
# FAIL "Charged twice for order 2210": urgent is False, expected True
# 2 of 3 passed

The grader says why each case failed, which points at the instruction to fix. This works because the agent returns structured output; free text would be much harder to check with code.

#Grading what code cannot check

Some qualities do not fit an if: is the reply polite, does it answer the actual question, does it promise something the shop cannot do? For those, write a rubric: a few yes-or-no questions, each about one thing. A person can grade with it, or a model can.

Yes-or-no questions tend to grade more consistently than "rate this from 1 to 10". A model grader makes mistakes too. Grade about 20 replies yourself, compare, and trust it only on the questions where it agrees with you.

#Evals for agents

Agents add two twists.

• Grade the outcome, not the path. An agent may reach the right result in 3 steps or in 6. Check what happened in the world, like whether the ticket ended up in the right queue, rather than demanding one exact sequence of tool calls. Do check the path for things that must never happen, like a refund nobody asked for.
• Run each case more than once. Agents take different paths on different runs. A case that passes 3 times out of 5 is a problem, even if it passed when you looked. Report a pass rate per case.

#Try it yourself

#Common mistakes

#Check yourself

#What to practise next

Pick one AI feature you use or build, and write ten cases for it today, including three edge cases. Then continue with Guardrails: evals tell you how often an agent goes wrong, and guardrails limit how much damage it can do when it does.