What a pilot is for
A pilot exists to answer one question with evidence. Should we invest in this, change it or drop it? Everything in the plan serves that decision. A pilot that ends with it looks promising has failed, whatever the technology did.
A pilot is
- One workflow, with real data and real users
- Limited in time, with a decision date
- Measured against today's performance
- Run with people reviewing the output
- Designed so that stopping is an acceptable result
A pilot is not
- A demonstration built to impress a meeting
- A test of everything the technology could do
- An open-ended experiment with no end date
- A launch to every user at once
- A commitment to continue whatever it shows
Choose a scope you can measure
| Decision | A good choice | A poor choice |
|---|---|---|
| Workflow | One task with clear inputs and outputs, done many times a week | A whole department, or a task that happens a few times a year |
| Users | A small group who do the work daily and are willing to give feedback | Everyone, or only managers |
| Data | Real cases from your own operations, including the awkward ones | Clean samples chosen because they work |
| Length | Long enough to cover a normal cycle of the work, with an end date | Until we are satisfied |
| Systems | The minimum connections needed, with manual steps where that is safer | Full integration before anything is proven |
| Owner | One person from the business who can decide | A committee, or the vendor |
If you have not chosen the workflow yet, how to identify business workflows suitable for AI gives a scoring method.
Record how the workflow performs today
Without a baseline you can only say that the pilot produced numbers. You cannot say that it helped. Measure the current process first, even roughly.
- 01
Count the volume
Cases per day or week, and how it varies.
- 02
Time the work
How long one case takes a person, measured on a sample and not estimated from memory.
- 03
Measure current quality
Review a sample of finished cases and record the error rate. People make mistakes too, and the pilot should be compared with reality, not with perfection.
- 04
Note the waiting time
How long a case sits before someone picks it up.
- 05
Record the cost
Time multiplied by the cost of that time, plus any rework that errors cause.
Define the success criteria
Choose a small number of measures, each tied to something the business cares about. Then set a pass mark for each one, in numbers, before the pilot starts. Deciding afterwards what counts as good invites everyone to see what they hoped for.
| Dimension | A measure | How to collect it |
|---|---|---|
| Quality | Share of outputs a reviewer accepts without correction | Reviewers mark each output as accepted, corrected or rejected |
| Serious errors | Number of outputs that would have caused harm if not caught | Reviewers flag them, and each one is examined |
| Time | Time for a person to complete a case with the system, against the baseline | Timed samples, during the pilot |
| Coverage | Share of cases the system attempts, as opposed to passing to a person | System logs |
| Cost | Running cost per case | Usage reports from the model provider and the hosting |
| Adoption | Whether the pilot users choose to keep using it | Usage logs, and a short interview at the end |
Include at least one measure that can fail the pilot by itself, usually serious errors. A system that is fast and cheap but occasionally does damage has not passed.
A worked example
Pilot plan for supplier invoice entry
A distribution company pilots a system that reads supplier invoices and prepares the entries for its accounting system. The company and all figures are invented to show how a plan reads.
- QuestionShould we use AI to prepare invoice entries for all suppliers?
- ScopeInvoices from the twenty largest suppliers. Two accounts clerks. The system prepares the entry and a clerk approves it.
- BaselineMeasured over two weeks before the pilot. Time per invoice, and errors found in the month-end check.
- Pass marksNine in ten entries accepted without correction. No wrong supplier or wrong amount posted. Clerk time per invoice at least halved. Running cost per invoice below the agreed figure.
- Test setThree hundred past invoices with the entries that were finally posted, including credit notes and invoices in a second language.
- ReviewEvery entry is approved by a clerk during the pilot. Corrections are recorded with a reason.
- Decision rulesAll pass marks met, extend to all suppliers. Quality met and time not met, improve the review screen and repeat. A serious error not caught by review, stop and reassess.
Everything in this plan was written before the pilot started. At the decision meeting, nobody has to argue about what the numbers mean.
Build the test set from real cases
The test set is a collection of real past cases with their correct outcomes. The system is run against it before any user sees it, and again after every change. It is the most valuable thing the pilot produces, so agree in writing who keeps it.
- Take cases from normal operations, not hand-picked easy ones
- Include the difficult and unusual cases in realistic proportion
- Have a person who knows the work confirm the correct outcome for each
- Keep part of the set aside, unseen during development, for the final check
- Add every failure found during the pilot, so that the same mistake is caught next time
Run it with people in the loop
During the pilot
0 of 7 checkedDecide whether to scale, adjust or stop
| What the pilot shows | Decision | What happens next |
|---|---|---|
| All pass marks met | Scale | Extend to more cases or users in steps, moving from full review to sampling |
| Quality met, time or cost not met | Adjust | Improve the workflow around the model, such as the review screen, the routing or the choice of model |
| Quality close but below the mark | Adjust once | Fix the main causes of failure and rerun against the test set, with a limit on the attempts |
| Serious errors that review did not catch | Stop or redesign | Lower the level of autonomy, or choose a safer workflow |
| Users avoid it despite good numbers | Investigate | The measures missed something. Find out what before spending more |
Write the outcome down with the evidence, whichever it is. A clear stop saves the cost of a full rollout, and the method you built carries over to the next workflow.
What to do next
Write the one-page plan, with the question, scope, baseline, pass marks, test set, review and decision rules. If you are choosing an outside team to run the pilot with you, how to choose an AI development company explains what to look for.
A controlled pilot is one of the steps in our AI integration and AI agent projects. Send us your draft plan and we will tell you which pass marks we think are missing.
Common questions
How long should an AI pilot run?
Long enough to cover a normal cycle of the work and to collect enough cases for the measures to mean something. For a daily workflow that is a matter of weeks, not months. Set the end date at the start. A pilot without one tends to become a permanent experiment that nobody is willing to judge.
How many test cases do we need?
Enough to include the variety of real work. A few hundred cases is a common starting point for a document or message workflow. Fifty well-chosen cases that cover the difficult situations are worth more than a thousand easy ones. The set should keep growing as new failures are found.
Who should own the pilot?
Someone from the business side of the workflow, with the authority to decide what happens at the end. Technical people and outside partners run the system, but the owner sets the pass marks and makes the call. Without that person, a pilot produces a report and no decision.
What if the pilot fails?
Then it has done its job at a fraction of the cost of a failed rollout. Look at why it failed. A different workflow, a lower level of autonomy or better data may change the result. Keep the baseline, the test set and the review process, because they apply to the next attempt.
What is the difference between a pilot and a proof of concept?
A proof of concept shows that something is technically possible, usually on sample data and without real users. A pilot shows whether it works in your operation, with real cases, real users and measures. The gap between the two is where a project can fail, which is why the pilot is the step that matters.
