Why an agent needs more care than a chatbot
A chatbot answers. An agent acts: it looks things up in your systems, sends messages, creates records, moves a task forward across several steps, choosing each step itself. That usefulness comes from access, and access is where the risk lies. A wrong answer can be ignored. A wrong action has already happened.
Two properties make agents different from ordinary automation. Their behaviour is not fully predictable, because a language model decides what to do next. And they can be influenced by what they read. An email, a web page or a document may contain text written to redirect the agent, a technique known as prompt injection. No instruction to the model reliably prevents this, so the design must assume it can happen and make sure the damage would be small.
Give the agent an identity
Every action in your systems should be attributable. That starts with deciding who the agent is when it acts.
| Model | How it works | Use when |
|---|---|---|
| Acts as the user | The agent uses the rights of the person who asked, and can do only what they could | An assistant working for one person at a time, such as answering questions over their records |
| Its own service identity | The agent has an account of its own, with rights set for its task | Background work with no single requesting user, such as processing incoming documents |
| Both | Its own identity, further narrowed by the rights of whoever triggered the task | Agents that serve many users and also perform system tasks |
Whichever model applies, avoid a shared administrator account, never reuse a person's credentials, and make sure actions appear in each system's own history as performed by the agent, on behalf of a named person where there is one. Credentials belong in a secrets store, not in the agent's instructions, and should be rotated like any other.
Grant the least access the task needs
- 01
List the task's steps
For each step, the system it touches and whether it reads or writes. Anything not on the list is not granted.
- 02
Expose narrow tools
The agent is given specific functions, such as look up an order by number or create a draft reply. It is not given a general connection to a database or an inbox.
- 03
Separate reading from writing
Many useful agents need to read widely and write almost nowhere. Grant them separately.
- 04
Restrict by scope
Limit to the records, folders, customers or mailboxes the task concerns, not the whole system.
- 05
Validate inputs in the tool
Each tool checks what it is asked to do against rules, whatever the model requested. An agent asking to refund more than was paid is refused by the tool.
- 06
Review regularly
Access granted for a pilot tends to stay. Check the list on a schedule and remove what is unused.
A useful test: if an outsider took full control of the agent for an hour, what is the worst they could do with the access it holds? Design the permissions so that the answer is tolerable.
Set limits, and require approval for what cannot be undone
| Control | Example | Protects against |
|---|---|---|
| Value limits | Refunds up to a set amount, discounts within policy | A costly single error |
| Rate limits | A maximum number of emails, orders or updates per hour | Runaway loops and mass mistakes |
| Approval steps | A person confirms before a payment, an external message or a deletion | Irreversible or outward-facing actions |
| Draft mode | Records and messages are created as drafts for a person to release | Errors in content |
| Allowed recipients and destinations | Email only to addresses on the customer record, files only to approved locations | Data being sent somewhere it should not go |
| Time and budget limits | A cap on steps, running time and model spend per task | Tasks that never finish |
| Off switch | One control that stops the agent and revokes its access | Anything unexpected |
Our guide on AI workflow approval and human review design explains how to decide which actions need a person and how to keep that review meaningful.
Treat what the agent reads as untrusted
Anything that arrives from outside your control can carry instructions aimed at the agent: an email asking it to forward the inbox, a supplier document with hidden text, a web page it was asked to summarize. The agent may follow them, because to a language model instructions and data look alike.
Designs that contain the risk
- An agent that reads outside content has no access to sensitive data or outbound channels
- Actions triggered by incoming content need a person's approval
- Outbound messages go only to known recipients
- Tools enforce rules regardless of what the model asks
- Suspicious requests are logged and reviewed
Designs that invite trouble
- One agent that reads the public web, sees private data and can send email
- Relying on an instruction telling the model to ignore injected text
- Automatically acting on instructions found in documents
- Showing agent output containing links or images from untrusted sources without checks
- No testing with deliberately hostile content
The dependable pattern is separation. An agent exposed to untrusted content should not also hold private data and a way to send it out. Where a task seems to need all three, split it into steps with a person or a strict rule between them.
Keep an audit trail someone can use
- What was asked. The request or event that started the task, and who or what triggered it.
- What the agent saw. The records, documents and messages it retrieved, or references to them.
- What it did. Each tool call with its inputs and results, in order, with times.
- What it produced. The final answer, message or record, and where it went.
- Who approved. Each approval or rejection, by whom, with any changes they made.
- Which version. The model, instructions and tool versions in use, so that behaviour can be reproduced and compared.
- What was refused. Requests blocked by a rule or a limit. These show both attacks and missing permissions.
Store the trail where the agent itself cannot alter it, protect it as you would the underlying data, since it may contain personal information, and set a retention period with your legal adviser. Then make sure someone reads it: a weekly sample of tasks, and every blocked or unusual action. A log nobody looks at is storage, not control.
A worked example
Permissions for a customer email agent
This example is invented to show access cut down to the task. An insurance broker wants an agent that reads incoming customer emails and prepares replies.
- First designThe agent is given the shared mailbox, full access to the client system and the ability to send email. A review asks what a malicious email could make it do, and the answer is: send any client's policy details to anyone.
- IdentityThe agent receives its own service identity. In the client system it appears as the agent, with the handler's name attached once they approve.
- AccessIt can read the incoming email and look up only the client whose address sent it. It can create a draft reply. It cannot send, and it cannot search clients by name.
- ApprovalA handler reviews each draft beside the original email and the client record, edits and sends. Replies can go only to the address on file.
- TrailEvery email, lookup, draft and handler decision is recorded. In testing with hostile emails, one asks the agent to include another client's renewal date. The lookup tool refuses, and the refusal appears in the log.
The agent still saves handlers most of their drafting time. The difference is that the worst outcome of a successful attack is now a strange draft that a handler deletes.
Permissions and audit checklist
AI agent permissions and audit checklist
0 of 11 checkedWhat to do next
For any agent you run or plan, write three lists: what it can read, what it can change and where it can send things. Then ask what the worst use of that combination would be. Where the answer is uncomfortable, remove an item from one of the lists.
Our AI integration services page describes how we connect agents to business systems with these controls, and you can browse task-specific examples in our AI agent solutions catalogue. Tell us what your agent needs to reach.
Common questions
What is prompt injection, in plain terms?
It is text written to make an AI system do something its operator did not intend. Because a language model reads instructions and content in the same way, a sentence hidden in an email or a web page can be followed as if it were a command. It cannot be fully prevented by wording, so the defence is to limit what the agent is able to do.
Should an AI agent use an employee's account?
It should not borrow a person's sign-in. It can act with a person's rights through a proper delegation mechanism, so that it can do only what that person could and the action is recorded as performed by the agent on their behalf. For background tasks, a dedicated service identity with narrow rights is the right choice.
Which actions should always need human approval?
Those that cannot be undone or that leave the organization: payments, messages to customers or third parties, deletions, changes to access rights, contractual commitments and changes to bank or identity details. Lower-risk actions can run within limits and be sampled afterwards. The list should be agreed with the process owner and written down.
How long should agent logs be kept?
Long enough to investigate problems, answer disputes and meet any legal or contractual duty, and no longer than privacy rules allow, since logs often contain personal data. The right period differs by sector and country. Decide it with your legal or compliance adviser, and apply the same access controls to the logs as to the data they describe.
Can we trust an agent more once it has worked well for a while?
You can widen what it does, step by step, on evidence. Good behaviour over time shows the agent handles normal cases. It does not show how it behaves under a deliberate attack or an unusual input. Keep limits, approval for irreversible actions and the audit trail in place however reliable the agent appears.
