Why an AI system does not stay as good as it was
Conventional software, left alone, keeps doing what it did. An AI system can decline while nobody touches it. A policy changes and the documents it reads are not updated. Customers start asking about a new product it has never seen. The provider replaces the model version with a newer one that phrases things differently or follows instructions less strictly. Costs creep up as conversations grow longer.
None of these trips an alarm in ordinary technical monitoring, which only reports whether the system is running. The system is running. It is simply giving worse answers. Looking after an AI system therefore means watching what it says, not only whether it responds.
What to monitor
| Area | Signals | What it tells you |
|---|---|---|
| Quality | Test set score, sampled review scores, user ratings, reported errors | Whether answers are still right |
| Usage | Active users, requests per user, topics asked, abandoned sessions | Whether people find it useful, and what they want from it |
| Unanswered and declined | Questions it could not answer, handoffs to people, repeated rephrasing | What content or capability is missing |
| Cost | Spend per day, per user and per task, against budget | Whether the economics hold |
| Speed and failures | Response time, timeouts, provider errors, malformed outputs | Whether the experience is acceptable |
| Safety and misuse | Blocked requests, attempts to extract data or bypass rules, policy violations | Whether someone is probing it, and whether the controls work |
Conversations often contain personal data. Decide what is stored, who may read it for review, how long it is kept and how users are told, with your data protection adviser, before monitoring begins.
Review real outputs on a schedule
Numbers show that something changed. Reading shows what. The most valuable monitoring activity is a person who knows the subject reading a sample of real interactions every week.
- 01
Sample deliberately
A random set, plus everything rated poorly, everything handed to a person and a selection of the longest conversations.
- 02
Score with the launch criteria
Use the same definitions of correct, grounded and complete that were used before launch, so that results are comparable over time.
- 03
Find the cause of each failure
Missing content, wrong content, a retrieval miss, a model error, an unclear question or a request that was out of scope.
- 04
Fix at the source
Most failures are fixed in the content or the instructions, not the model. Assign each one to its owner.
- 05
Add failures to the test set
Every real failure becomes a permanent test question, so that it cannot quietly return.
Manage model and provider changes
Model providers release new versions frequently and retire old ones on published schedules. A newer model is often better in general and still worse on a particular task of yours. Treat every model change as a release of your own system.
- 01
A pinned version
What to have in placeUse a specific model version, not an alias that silently moves to the latest, so that behaviour changes only when you decide.
- 02
A watch on retirement notices
What to have in placeSomeone subscribed to the provider's announcements, with dates in a calendar.
- 03
A test set ready to run
What to have in placeThe same questions and scoring used at launch, able to compare a candidate model with the current one within a day.
- 04
A staged switch
What to have in placeTry the new version on a share of traffic or with a pilot group before moving everyone.
- 05
A layer of your own
What to have in placeThe model called through an interface you control, so that changing version or provider does not mean rewriting the application.
- 06
A record
What to have in placeWhich version served which period, so that a change in quality can be traced.
Keep the content current
Where the system answers from documents or data, its upkeep is mostly the upkeep of those sources. New and changed documents must reach it, withdrawn ones must leave it, and someone must act on the list of questions it could not answer.
Content under control
- Each source has an owner and a review date
- Changes at the source reach the system automatically
- Unanswered questions are reviewed and lead to new content
- Product and policy launches include an update to the system's sources
- Old versions are removed, not left beside the new
Content adrift
- Documents uploaded once at launch and never since
- Nobody knows which version the system is reading
- The same question fails every week
- Staff quietly stop using it and go back to asking colleagues
- Content changes are discovered through customer complaints
A worked example
Six months of running an internal IT help assistant
This example is invented to show the kinds of event that monitoring catches. A company of 600 staff runs an assistant that answers IT questions from its help articles.
- Month oneWeekly review of 50 conversations finds that questions about the new expense tool fail, because its guide was never added. The guide is added, and three test questions with it.
- Month twoDaily cost doubles in a week. The usage view shows a team pasting entire log files into the assistant. A length limit and a short notice solve it.
- Month fourThe provider announces retirement of the model version in use. The test set is run against the successor: overall score is slightly higher, but it now answers two questions it should decline. The instructions are adjusted and retested before the switch.
- Month fiveUser ratings dip. Review traces it to a change in the remote access procedure that left two contradictory articles. The old one is withdrawn.
- Month sixThe quarterly report shows questions answered, the topics with most failures and the cost per answered question, and proposes adding hardware requests as a new topic.
None of these events was a system outage, and ordinary monitoring would have shown green throughout. Each was caught within days because someone was reading, counting and testing.
Roles, support agreement and reporting
- Business owner. Accountable for whether the system is worth having. Receives the report, decides on scope and budget, and can pause the system.
- Content owner. Keeps sources correct and acts on unanswered questions.
- Quality reviewer. Reads the weekly sample, scores it and maintains the test set. Someone who knows the subject.
- Technical support. Watches cost, speed, failures and security signals, applies changes and manages model versions. Often the company that built the system, under a support agreement.
- A regular report. Monthly or quarterly: usage, quality against the launch bar, cost, notable failures and what was done, and recommendations.
A support agreement for an AI system should name the monitoring that is included, response times for faults, who reruns the test set and when, how model changes are handled and what is reported. Our AI evaluation and monitoring page describes how we provide this.
Monitoring and support checklist
AI monitoring and support checklist
0 of 11 checkedWhat to do next
If you have an AI system running, read twenty of last week's real interactions yourself. Then ask who would notice if its answers got worse next month. If the answer is nobody, that is the first role to assign.
If the system is not yet live, our AI prototype production readiness checklist covers what to have in place first. Tell us what you are running and we will suggest what to watch.
Common questions
Why would an AI system get worse after launch?
Because the things around it change. Documents go out of date, users ask about new subjects, providers update models and usage patterns shift. The software may be untouched while the quality of its answers falls. Ordinary uptime monitoring cannot see this, which is why regular review of real outputs and a maintained test set are needed.
How often should we review the system's outputs?
Weekly in the first months, with a sample large enough to cover the main topics, and at least monthly once it is stable. Review immediately after any change to the model, the instructions or important content. The frequency should reflect what a wrong answer costs: a system that speaks to customers deserves closer attention than an internal drafting aid.
What does it cost to run an AI system?
Running costs include model usage, which scales with volume and with the length of inputs and outputs, hosting, monitoring tools and the time of the people who review and maintain it. Model usage is the most variable. Measure cost per task from the start, set limits and alerts, and review the figures monthly, since both prices and usage change.
What happens when the AI model is retired by its provider?
Providers announce retirements in advance. With a pinned version, a watch on announcements and a test set, the change is routine: test the successor against your cases, adjust instructions if results differ, switch a pilot group, then everyone. Without those, the first sign may be an error message or a sudden change in behaviour.
Do we need a support contract for an AI system?
If you lack in-house skills to manage model changes, cost, failures and security, yes. Compared with ordinary software support, the agreement should also cover quality: who reruns the test set, who reviews outputs and what is reported. Content ownership and subject review usually stay inside your organization whatever the technical arrangement.
