The guardrails an AI agent needs before it can act

AI agent guardrails are the controls that limit what an agent can do, check each action before it runs, and record what happened afterwards. An agent that only answers can be wrong. An agent that acts can be wrong expensively: it refunds the wrong order, emails the wrong customer, or overwrites a record nobody thinks to check.

Before an agent touches a live system, it needs seven guardrails. In the order to add them: scoped tools, an eval set with attack tests, replayable traces, a staged rollout, human approval for high-stakes actions, limits on retries and spend, and versioned prompts and models. Each one costs build time, latency or reviewer hours, and we name which below.

Autonomy is easy to treat as one decision: the agent can act, or it cannot. It works better as a sequence, with each guardrail in place before the risk it covers arrives.

Give the agent the narrowest tools that do the job, and enforce their limits outside the model. A refund tool should refund one order, up to a set value, for the customer the agent is acting for. It should not be a general database connection with a polite instruction in the prompt.

The OWASP Gen AI Security Project's 2025 Top 10 for LLMs and Gen AI apps calls this risk excessive agency (LLM06:2025). It names three usual root causes: excessive functionality, excessive permissions and excessive autonomy. Its worked example is a mailbox assistant built to summarise email, whose plugin can also send. A crafted incoming email tricks it into forwarding sensitive information to the attacker. OWASP's fixes: a mail tool that can only read, a read-only OAuth scope, or a person who reviews and presses send on every draft.

The rule that matters most is where the check lives. OWASP's advice is to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." The reason is in its prompt injection entry: "it is unclear if there are fool-proof methods of prevention for prompt injection." Assume a tricked agent, then decide what it can reach.

We build agent tools to be scoped, rate-limited and policy-gated, with the policy written as code. The cost is build time: several narrow tools take longer than one broad one. Latency is usually a small cost, because a policy check is code, not another model call.

An agent needs a test set before it needs features. We build the eval set before the system, and every release, including a prompt edit, has to pass it. A regression blocks the release. For an agent, each case records the task, the tool calls it should make and the ones it must not. Sometimes the correct answer is handing over to a person, so test for that too.

Pull the cases from real work, as with any golden set; our post on RAG retrieval failures covers building one from support tickets. The same discipline stops a chatbot making up answers. An agent adds the question of what it did, not only what it said.

Then add red-team probes to the same run, as we do on every continuous eval. OWASP recommends regular "penetration testing and breach simulations, treating the model as an untrusted user". Indirect injection arrives in input the model takes "from external sources, such as websites or files". So plant hostile instructions in the tickets, emails and documents the agent will read, not only in the chat box.

The cost is build time before any feature work, and a subject expert's hours to confirm the right action for each case. Every release then waits for the run.

When an agent does something wrong, the first question is why. Without a trace, the answer is a guess. Log each run as a structured trace: the request, the context retrieved, every model call, every tool call with its arguments and result, each policy decision, and the final action.

Replay lets the operations team step through a failed run exactly as it happened, then turn it into a new eval case. We also sample production traces for review, and drift triggers an alert and a re-baseline.

OWASP is precise about what this buys you. Logging, monitoring and rate limiting "will not prevent Excessive Agency, but can limit the level of damage caused." Traces are how you notice, not how you stop.

The cost is storage and build time. Traces hold customer data, so they need the same PII redaction and access control as the system itself. Written asynchronously, they stay out of the user's wait.

We roll out sensitive operations in stages: read, then suggest, then act. In read mode the agent gathers context and answers, and changes nothing. In suggest mode it drafts the action (the refund, the record update) and a person carries it out. In act mode it executes, inside the limits the suggest stage proved.

Promote per action type, not per agent. The same agent can tag tickets on its own while it still only suggests refunds. Every accepted or rejected suggestion is evidence, and it belongs in the eval set.

Track two things per action type during suggest: the share of suggestions a reviewer accepts without edits, and the reviewer time each one takes. The first tells you when an action is ready to promote. The second is what human approval will cost once volume arrives.

Stage by volume as well. Our agent pilots run live on a controlled subset of work, with full instrumentation and human-in-the-loop review. That applies even to agents that only answer: the case-work agent, an illustrative engagement pattern, pilots with a controlled cohort of case officers before a phased rollout to all of them.

The cost is calendar time, plus reviewer attention on every suggestion. That is the price of finding mistakes while they are still drafts.

Human in the loop means a person approves an action before it takes effect. OWASP recommends it "to require a human to approve high-impact actions before they are taken". Use it for actions that are irreversible, leave the company, or cross a value threshold: sending money, contacting a customer, deleting or changing a customer record.

The review console decides whether approval is a safeguard or a rubber stamp. It should show the reviewer what the agent saw, what it proposes and why, with approve, edit and reject side by side. We build these consoles into the agent itself, route failures to a reviewer with the relevant context, and define escalation paths up front.

OWASP puts the approval step in the downstream system or in the tool itself. Not in the prompt: an instruction to "always ask first" is the kind of rule an injected document tells the agent to ignore.

The cost is reviewer hours, which grow with volume, and latency measured in human time, since the action waits for a person. Route every action to approval and the reviewers become the bottleneck. The table above is where to draw the line.

Agents loop. OWASP notes that agent systems typically make "repeated calls to an LLM using output from previous invocations". One confused task can call the model and its tools many times. We build four limits into agent orchestration to keep that bounded:

• Retries with a ceiling, and only on tools that are safe to repeat. A retried payment call that is not safe to repeat can pay twice.
• Fallbacks to another model, a simpler path or a person when the first route fails.
• Cost limits per task and per day, so a runaway loop stops at a known spend.
• Circuit breakers that stop calling a tool when its error rate or spend crosses a threshold, and send the work to people.

Spend is an attack surface too. OWASP's unbounded consumption entry (LLM10:2025) describes denial of wallet: attackers send "a high volume of operations" to exploit "the cost-per-use model of cloud-based AI services". Its mitigations include rate limits, user quotas, timeouts and throttling.

The cost is latency when a fallback fires, and some legitimate long tasks that hit a cap and go to a person. Thresholds need tuning against real traffic, which is one more reason the pilot runs on a controlled subset of work.

A prompt edit changes the agent's behaviour as much as a code change, so ship it like one. Version the prompt, the model, the tool definitions and the policy rules together, and roll back the set rather than one piece. On the systems we build, prompts and models are versioned with rollback in seconds, and the eval gate runs on every release.

Hosted models also change underneath you, and providers deprecate versions. Pin the version you tested. Run the replacement through the same eval set before you switch, because a model that writes better answers can still choose tools differently.

The cost is release discipline rather than runtime. Every prompt change goes through review and the eval run, which feels slow until the first time you need to undo one.

List every tool your agent can call. For each, write down the worst thing it could do if the model were tricked. Then note where its permission check lives: in the downstream system, or only in the prompt.

Anything that can move money, contact a customer or delete data, with no check outside the model, goes back to suggest mode today. Then write the eval cases for those tools, attacks included, before you promote any of them again.