The gap between a demo and a product
AI prototypes are quick to build and persuasive to watch. In an afternoon, someone connects a model to a few documents and it answers questions impressively. The natural conclusion is that the project is nearly finished. It has barely started, and understanding why saves a great deal of disappointment.
| In the prototype | In production |
|---|---|
| A handful of chosen examples | Every question real users think of |
| Sample or copied data | Live data, with each user's permissions |
| One person using it | Many at once, at busy times |
| Failures shrugged off | Failures seen by customers |
| Cost unnoticed | Cost multiplied by real volume |
| Nobody trying to misuse it | Some people trying to |
| Built and watched by its author | Run by a team for years |
Our comparison of a prototype and a production-ready MVP describes the same gap for software in general. With AI it is wider, because quality cannot be assumed from a few good results.
Quality: measured, not demonstrated
The first requirement is evidence. Build a test set of real questions or cases with known correct results, including difficult ones and ones the system should decline, and set the result needed for launch before running it. Score consistently and keep the results, so that every later change can be compared.
Then look at how uncertainty is shown to users. A production system should say when it does not know, show the sources behind an answer and make it easy to report a poor result. A prototype that always answers confidently has not yet been designed for the cases where it is wrong. Our guide to evaluating AI answers before production sets out the method step by step.
Data and access: the real thing, properly controlled
- 01
Is it connected to live sources?
Why it mattersA copy of the data made for the prototype goes stale at once. Production needs a maintained connection or a scheduled refresh.
- 02
Are each user's permissions enforced?
Why it mattersPrototypes typically read everything through one account. Production must return only what the person asking may see.
- 03
Is the data fit for use?
Why it mattersDuplicates, old versions and contradictions that the demo avoided will surface with real questions.
- 04
What data is sent to the model provider?
Why it mattersUnder which terms, retained for how long, processed where, and has your data protection adviser reviewed it?
- 05
How does content stay current?
Why it mattersWho adds, changes and removes sources, and how quickly does the system reflect it?
Reliability and cost at real volume
- 01
Handle the provider being slow or down
Model services have outages and rate limits. Decide what the user sees, whether a request is retried, and whether a second model or a non-AI fallback exists.
- 02
Set time limits
A response that takes too long is a failure to the user. Measure response times under load and set a ceiling for each kind of request.
- 03
Check that outputs are usable
When the system expects structured output from the model, validate it and handle the occasions when it is malformed.
- 04
Calculate cost per use
Multiply by expected volume, then by the busiest realistic day. Long documents, long conversations and repeated attempts raise it sharply.
- 05
Put limits on spending
Caps per user, per task and per day, with alerts. A loop or an abusive user should not produce a surprising invoice.
- 06
Test with realistic load
Many users at once, long inputs, several languages. Find the limits before your users do.
Providers change prices, retire model versions and alter behaviour between versions. Keep the model behind a layer of your own so that it can be replaced, and keep your test set ready to judge any replacement.
Security, misuse and compliance
- Injected instructions. Test with content written to redirect the system, in user messages and in documents it reads. Limit what it can do, so that a successful attempt achieves little.
- Data leakage. Confirm that one user cannot obtain another's data, and that system instructions and internal details are not revealed on request.
- Abuse and off-topic use. Decide what the system declines, and apply rate limits and sign-in where appropriate.
- Harmful or prohibited content. Define what must never be produced, test for it, and filter outputs where the risk warrants.
- Legal review. Privacy, sector rules, consumer law and, in some jurisdictions, rules specific to AI systems may apply. Have the use reviewed by a legal adviser before launch.
- Disclosure. Tell users they are dealing with an AI system, and what it should and should not be relied on for.
Where the system takes actions, the controls in our guide to AI agent access permissions and audit trails apply in full.
A worked example
A proposal drafting tool goes from demo to production
This example is invented to show what the gap contained in one case. An engineering consultancy builds a prototype that drafts proposal sections from past proposals, and the partners want it rolled out.
- QualityA test on 40 past requests shows strong drafts for standard services and weak ones for a newer practice area with few examples. The launch scope is limited to the standard services.
- AccessThe prototype read every past proposal. Some are confidential to particular clients. Production reads only proposals marked reusable, and respects each user's client access.
- ReliabilityDuring a test, the model provider is slow for an hour. A queue with a clear waiting message is added, with a daily spending cap.
- CostLong proposals make each draft costlier than assumed. Sending only the relevant sections cuts the cost per draft to a third.
- OwnershipThe knowledge manager owns the content. A partner owns quality and reviews a sample monthly. A support agreement covers the system.
The work between demo and launch took several times longer than the demo itself. It also turned up the confidentiality problem, which would have been serious if the prototype had simply been opened to all staff.
Monitoring, support and ownership
Ready to operate
- Quality, usage, cost and errors visible on one screen
- User feedback collected and read by someone
- The test set rerun after every change
- A named owner for content, for quality and for the system
- A rollback plan and an off switch
Not ready
- Only the person who built it knows how it works
- No alert when it fails or costs spike
- Changes go live without a test run
- Feedback goes nowhere
- No plan for a retired model version
Launch in stages: a pilot group first, then wider, with the measures watched at each step. Our guide to AI usage monitoring and ongoing support covers what happens after launch.
The production readiness checklist
AI production readiness checklist
0 of 11 checkedWhat to do next
Go through the checklist with whoever built your prototype and mark each line as done, partly done or not started. The pattern of marks is your plan, and it gives management an honest view of the distance left.
Our AI prototype to production page explains how we take over prototypes and harden them. Send us a description of yours and we will say which lines we would look at first.
Common questions
Why does production take so much longer than the prototype?
Because the prototype handles the cases its author chose, and production must handle every case, for every user, under load, with real data, safely, for years. Evaluation, permissions, failure handling, cost control, security, monitoring and ownership are each modest pieces of work. Together they are usually several times the effort of the demonstration.
Can we just give the prototype to a few users and improve it as we go?
A controlled pilot is a good idea, provided the essentials are in place first: real permissions, a known cost limit, disclosure that it is experimental, a way to collect feedback and an off switch. Opening a prototype without those risks exposing confidential data or producing errors that damage trust in the whole project.
Should we rebuild the prototype or extend it?
It depends on how it was built. Prototypes made in notebooks or no-code tools are often rebuilt, keeping the instructions, test cases and lessons, which are the valuable parts. A prototype built with sound structure can be extended. An independent review of the code before deciding is quick and avoids both unnecessary rebuilding and building on sand.
What if the model we used is retired or changed?
Providers publish schedules for retiring model versions, and newer versions can behave differently. Design for this from the start: keep the model behind your own interface, pin a specific version, and maintain a test set so that a replacement can be judged in hours. Then a retirement notice becomes a scheduled task.
Who should own an AI system once it is live?
Three responsibilities need names: the content it draws on, the quality of what it produces and the running of the system. In a small organization one person may hold two of them. What matters is that someone reviews real outputs regularly and has the authority to pause the system if quality falls.