Why a demonstration is not evidence
AI systems fail in a way ordinary software does not. A calculator that works for one sum works for all of them. A language model can answer nine questions well and the tenth confidently and wrongly, in the same fluent tone. Someone trying it for ten minutes sees the nine. Customers will find the tenth.
Evaluation is the discipline of finding the tenth before they do. It does not require a data science team. It requires a list of questions, an agreed idea of what a good answer is, and the patience to score results the same way every time. Business staff who know the subject are the best evaluators there are.
Build a test set from real questions
- 01
Collect real questions
From support tickets, emails, search logs and the people who answer questions today. Use the wording people really use, with its typos and vagueness.
- 02
Write the expected answer for each
A short reference answer and where it comes from. Have a second expert confirm it.
- 03
Cover the range
Common questions, rare but important ones, questions with several parts, questions in each language you support and questions phrased badly.
- 04
Include questions that should be refused
Topics outside the system's scope, requests for advice it must not give and questions the content cannot answer. Saying so is the correct behaviour.
- 05
Include attempts to misuse it
Requests to ignore instructions, reveal confidential material or produce harmful content.
- 06
Keep some questions unseen
Hold back a portion that is never used while the system is being adjusted, so that the final score is honest.
Fifty to a hundred well-chosen questions is a sound start for a focused use. The set should grow: every failure found after launch becomes a new test question.
Define what a good answer is
People disagree about quality until criteria are written down. Agree them with the business owner before scoring begins, and keep them few.
| Criterion | Question to ask | Failure looks like |
|---|---|---|
| Correct | Does it agree with the reference answer? | A wrong figure, date, rule or step |
| Grounded | Is every claim supported by the source shown? | Plausible detail that appears in no document |
| Complete | Does it include what the person needs to act? | A true answer that omits the condition that matters |
| Relevant | Does it answer the question asked? | A correct paragraph about something nearby |
| Appropriate refusal | Does it decline when it should, and only then? | Answering outside scope, or refusing a fair question |
| Tone and format | Is it clear, concise and in the house style? | Rambling, over-apologetic or oddly formal text |
| Safe | Does it avoid harmful, confidential or prohibited content? | Leaking another customer's data, giving regulated advice |
Not all criteria weigh the same. For a policy assistant, correctness and grounding dominate. For a drafting tool, tone matters more. Decide which failures are tolerable and which are never acceptable.
Score the same way every time
Each answer is scored against each criterion, as pass or fail, or on a short scale with written definitions. Two methods are used together.
- Human review. People who know the subject score the answers. This is the standard everything else is measured against. Have two reviewers score a shared sample first, compare, and refine the definitions until they agree.
- Automated scoring. Software, often using another model, compares each answer with the reference and the sources. It is fast enough to run on every change. It must be checked against human scores before it is trusted, and rechecked from time to time.
- Retrieval checks. Where the system looks up documents, check separately whether it found the right passage. If it did not, the fault lies in content or search, and no amount of adjusting the model will help.
- Exact checks. For structured outputs such as an extracted invoice number or a chosen category, a simple comparison with the expected value is enough.
Keep every run: the questions, the answers, the scores and the version of the system. A history of results is what turns evaluation from an event into a control.
Set the bar before you see the results
Decide in advance what result permits launch. Deciding afterwards invites the answer that whatever was achieved is good enough. The bar depends on what a wrong answer costs and on what stands between the answer and the customer.
| Situation | Reasonable bar | Safeguard |
|---|---|---|
| Drafts reviewed by staff before sending | Most answers usable with light editing | The reviewer |
| Internal answers shown with sources | High correctness, with clear refusal when unsure | Sources visible, feedback button |
| Answers sent directly to customers | Very high correctness, and no failures on the never-acceptable list | Narrow scope, handoff to a person, monitoring |
| Actions taken automatically | Near-perfect on tested cases, with limits on what can be done | Approval steps, limits, audit trail |
Compare the result with the present situation too. If people answering the same questions today are right most of the time, that is the fair benchmark, not perfection.
A worked example
Evaluating a returns policy assistant
This example is invented to show an evaluation from start to decision. A retailer is preparing an assistant that answers customers' questions about returns and refunds.
- Test set120 questions from last year's support tickets, with reference answers confirmed by two team leaders. 20 are held back unseen. 15 should be refused, including requests for legal advice.
- CriteriaCorrect, grounded and appropriate refusal are mandatory. The never-acceptable list: inventing a refund period, and promising a refund the policy does not allow.
- BarAgreed before testing: at least 95 of the 100 working questions fully correct, and zero never-acceptable failures.
- First run88 correct. Three never-acceptable failures, all on items bought in a sale, where two policy documents disagree.
- Fix and rerunThe policy documents are reconciled. The second run scores 96 with no never-acceptable failures, and the held-back set scores 19 of 20.
The assistant launches for returns questions only, with handoff to a person on request. The three sale-item failures would have reached customers within days. They were caused by the documents, and the evaluation is what exposed them.
Retest after every change
Changes that require a rerun
- A new model or model version
- Any edit to the instructions
- New, changed or removed documents
- A change to how documents are searched
- A new language, channel or user group
Habits that hide problems
- Testing only the questions that used to fail
- Adjusting the system until the held-back set passes
- Trusting automated scores never checked by people
- Evaluating once, at launch, and never again
- Dropping hard questions from the set
A change that fixes one answer can break another. Rerunning the whole set each time is the only way to see that, which is why automated scoring is worth setting up. After launch, the same set becomes the core of ongoing monitoring, described on our AI evaluation and monitoring page.
Evaluation checklist
AI answer evaluation checklist
0 of 11 checkedWhat to do next
Collect thirty real questions this week and write the correct answer beside each. If you have a prototype, run them through it and score the results with a colleague. If you do not, you have just created the most useful document for whoever builds one.
Our guide to planning an AI pilot with measurable success criteria shows how evaluation fits into a pilot. Send us your question list if you would like a view on how to test it.
Common questions
How many test questions do we need?
Enough to cover the range of what users will ask, which for a focused use is typically fifty to a hundred to begin with. Coverage matters more than volume: a hundred variations of one easy question prove little. Add questions for each topic, language and kind of refusal, and grow the set with every failure discovered after launch.
Can AI be used to grade AI answers?
Yes, as a fast first pass, and only after its scores have been compared with those of human experts on a sample. Automated grading is good at spotting clear mismatches and makes it practical to retest after every change. It can share blind spots with the system under test, so people should review a sample of each run and all borderline cases.
What accuracy is good enough to launch?
It depends on what a wrong answer costs and what safeguards exist. A drafting aid reviewed by staff can launch at a lower level than an assistant answering customers directly. Set the bar before testing, define the failures that are never acceptable, and compare the result with how accurately people answer the same questions today.
Who should do the evaluation?
People who know the subject and were not responsible for building the system. They write reference answers, score results and decide whether the bar has been met. The development team runs the tests and fixes what fails. Keeping those roles apart avoids the natural tendency to mark your own work generously.
Do we have to evaluate again after launch?
Yes. Content changes, users ask new kinds of question and model providers update or retire model versions. Rerun the test set on a schedule and after every change, review a sample of real conversations and add new failures to the set. Evaluation before launch tells you the system was good on that day only.