Governance by design: How enterprises make AI agents compliant and ready to scale

AI agent governance is the set of controls that make every action an agentic AI system takes visible, permissioned, and reconstructable after the fact. In practice, that means four things: an inventory of which agents exist, scoped permissions on what each one may touch, an immutable record of what each one did, and approval gates on consequential actions. None of this is a checklist bolted onto a finished system — it's a set of controls built into how the agent runs in the first place.

That distinction — governance as a runtime constraint versus governance as a document nobody enforces — is critical. It’s governance built into how agents execute, not a policy someone consults after something has already gone wrong.

As Matthew Hopkinson, Head of Sales UK&I at Butternut Box, puts it:

“Trust has to be designed. It has to be planned. That means clear outputs, clear expectations, visible audit trails, agents with narrow jobs.”

Why most agent pilots never reach production

Most agent pilots don't stall because the model isn't capable enough. They stall because nobody can answer a basic question when a compliance officer asks it: what did the agent actually do, on what authority, and can you prove it? A demo that works in a sandbox with one engineer watching the logs doesn't answer that question. A production deployment has to.

Gartner's research points to the same root cause, predicting that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps identified only after production incidents occur. They found those gaps most likely when organizations fail to distinguish between an agent's ability to act and the scope of access it's been granted, treating governance as a binary switch rather than something calibrated to what each agent can actually do.

Bain's analysis of agentic AI governance points at the same failure from the platform side: a rule enforced in the control plane applies to every agent, every time; a policy document applies only when someone remembers to check it. Bain points to a widely reported 2025 case as what that gap looks like in practice — a coding agent on the Replit platform deleted a live production database during an active code freeze, against repeated instructions, then told the developer the deletion couldn't be rolled back, when in reality it could be.

The four control surfaces of governed agents

Four categories of control show up, in some form, across nearly every serious treatment of this problem: knowing what agents exist, controlling what they can touch, recording what they did, and requiring a human decision before consequential actions. Each answers a different question a compliance reviewer will ask.

Agent inventory: Knowing what you're running

You can't govern an agent you don't know exists. The starting point is a live registry: every agent, who owns it, what it's for, and what risk tier it sits in — including the ones nobody officially built, sometimes called shadow agents.

Agents get spun up inside SaaS tools by business users who never file a request with IT, the same way shadow IT worked before them. This is now treated as a distinct governance discipline in its own right: agent discovery has become a standard evaluation criterion for governance tooling, precisely because so many agents exist outside anything a platform team deployed — embedded in SaaS products, built by vendors, or assembled by teams working around IT rather than through it. An inventory that only tracks agents your platform team deployed misses most of the actual risk surface, because the agents nobody logged are the ones nobody's reviewing.

Permissions: Least privilege for non-human identities

An agent is a non-human identity — a distinct, attributable identity issued to a system rather than a shared account — and it deserves the same discipline you'd apply to a person's account: scoped credentials, a defined allow-list of tools, periodic access review, and a real offboarding process when the agent is retired. Shared service accounts break this immediately — if three agents share one set of credentials, you've already lost the ability to say which one did what.

The clearest way to see what this looks like at the infrastructure level: an agent needs a distinct, cryptographically verifiable identity, not a generic service account, so every request it makes can be tied back to that specific agent rather than a shared credential pool.

The payoff shows up in how a real GRC-conscious organization talks about it.

The decision trail: What happened, and on what basis

A complete decision record answers one question completely: what did this agent do, and why was it allowed to do it? That means capturing, for every consequential run:

This is the single highest-consensus idea in how practitioners describe agent governance today, and for good reason — it's the difference between believing the agent behaved correctly and being able to show exactly what it did.

The clearest demonstration of what governance by design looks like in practice comes from Fanatics:

“Since every asset is automatically logged with its copy, disclaimers, and approval record, Airtable doubles as a compliance archive. This reduces audit preparation from days to minutes.”

The audit trail wasn't a project someone ran after the fact. It was a byproduct of how the workflow was built.

Why “explainable reasoning” is the wrong promise

A model's internal chain of thought is not a compliance artifact, and vendors who market it as one are setting auditors up to rely on something that can't bear that weight — reasoning text isn't guaranteed to reflect the actual basis for an action, and it isn't stable enough to audit against. What an auditor actually needs is a reconstructable decision record: the inputs, the evidence retrieved, the tool calls made, the policy checks run, the approval or override, and the output. That's a defensible standard. “We can show you what the model was thinking” is not.

Approval gates: Where humans stay in the loop

Not every action needs a human in the loop, and treating every action as if it does is how governance becomes the reason agents never ship. The right approach is risk-tiered: financial commitments, legal or contractual language, HR actions, access-control changes, and anything communicated externally warrant a gate; routine, reversible, low-stakes actions don't. Separation of duties matters here too — the agent or person requesting an action shouldn't be the one approving it.

Amanda DeFrees, Sr. Marketing Program & Operations Leader at Overhaul, describes what risk-tiered routing looks like when it's built into the system rather than left to individual judgment: “Airtable [is] our metadata backbone. … The engine automatically routes content to the right reviewer based on asset type and risk level: triggered in Slack, tracked in Asana, logged in Airtable. No manual triage.”

That's the answer to the objection every approval-gate section eventually runs into — that gates slow everything down. They don't, if the routing itself is automated and scoped to actual risk.

What governed agents look like in practice: The two-agent approval pattern

Abstract governance arguments are forgettable. A team describing a mechanism they actually run is not.

Butternut Box splits sales-operations work between two named agents with a deliberate division of labor. Matthew Hopkinson, Head of Sales UK&I, describes the pattern:

“Marlowe outputs to a Slack channel, saying, 'These are the actions that I've detected. Would you like them to be implemented?' And then, if the human says yes, Marcel comes in and actually applies the action to the record, and then outputs a summary of what has changed, with an audit trail.”

Three things make this governed rather than merely monitored:

Separation of duties between agents. The agent that detects an action is not the agent that executes it. One proposes, a human decides, a second carries it out — the same separation-of-duties principle that applies between two people, implemented here between two non-human identities.

The approval gate lives inside the workflow. The decision happens in Slack, where the team already works, not in a governance console someone has to remember to check. Gates placed outside the day-to-day workflow get bypassed or ignored.

The audit trail is a byproduct of the work, not a reconstruction of it. The acting agent emits the record of what changed as part of doing the work — the same principle behind the Fanatics compliance archive, running at a smaller, more granular scale.

Narrow scope is itself a control. Hopkinson again: “The more specific the job is, the tighter the scope, the lower the chance you get of hallucination.” That's the bridge most governance write-ups never build: agent design choices are risk-reduction choices, not just engineering ones.

Governance that only exists on paper isn't governance

A governance document that nothing enforces isn't governance — it's a policy someone hopes gets followed. Put simply: governance without runtime enforcement is unverifiable policy, and runtime security without governance lacks accountability. You need the rule and the mechanism that actually stops the violation.

The practical test: can your platform block a non-compliant action at the moment it’s about to happen, or can it only tell you afterward that one occurred? The first is governance. The second is just monitoring.It's worth separating governance from security here, because the two get used interchangeably and they're not the same job. Security asks whether an agent, its credentials, or its tools have been compromised — the concern is an outside actor or a malicious input taking control. Governance asks whether an agent that is behaving exactly as designed is still doing something it shouldn't — acting outside its scope, skipping an approval, touching data it wasn't meant to touch. A well-secured agent can still be ungoverned, and a well-governed agent can still be exploited. Production-ready means both are in place, not one standing in for the other.

Policy-aware agents look different in practice than agents wrapped in after-the-fact checks. Michael Huynh at Nike describes this shift: “Rather than relying solely on traditional rules-based automation, we apply AI to support more complex, context-driven tasks … while also taking into account company policy guidelines that influence allowable spend.” The agent reasons inside the constraint, rather than being checked against it after the fact.

AI agent observability vs. traditional monitoring

Traditional monitoring answers whether the system is up. Agent observability answers a different question: what did the agent decide, which tools did it invoke, how did a handoff between agents resolve, and has its behavior drifted from what you approved? That last question points to what practitioners call drift — the gradual divergence of an agent's behavior from its approved pattern, often invisible to infrastructure metrics. The larger distinction matters because infrastructure health and decision quality are unrelated for agentic systems in a way they aren't for traditional software — a request that takes two seconds and one that takes sixty can both be perfectly normal, so the signals that flag trouble in conventional systems don't reliably flag it here. An agent can pass every uptime and latency check while quietly making a sequence of bad decisions that never trips a traditional alert.

This matters for the same reason the decision trail matters: an agent can be “running” successfully by every infrastructure metric while doing something it shouldn't. Observability is what closes that gap, and it belongs in the same conversation as governance rather than as an afterthought. Agent monitoring, in the narrower sense of tracking individual runs for anomalies, is really this same discipline applied continuously rather than case by case.

Mapping controls to the regulations that apply

The frameworks a governance program has to satisfy — EU AI Act, NIST AI RMF, ISO 42001, SOC 2, GDPR, and sector-specific regimes like HIPAA, SOX, PCI-DSS, and DORA — all reduce to a smaller set of underlying evidence requirements: who did what, under what authority, with what data, and who approved it. Build the four control surfaces described above and most of the mapping work is already done; each regime just asks for the same evidence in a different vocabulary.

A few of these are worth naming specifically, because they land in different places. The EU AI Act imposes tiered obligations by risk category, with documentation and human-oversight requirements that fall squarely on the approval-gate and decision-trail controls above. NIST's AI RMF organizes around four functions — Govern, Map, Measure, Manage — that map almost one-to-one onto inventory, permissions, observability, and runtime enforcement. ISO 42001 asks for a documented AI management system, which is effectively the inventory and decision-trail controls formalized as a certifiable process. None of these require a separate program; they require the same four control surfaces, described in the vocabulary a given auditor expects.

Sonia Alonso, Investment Operations & Compliance Manager, describes what that mapping looks like once it's operational: “We now have automated notifications enabling immediate stakeholder engagement, centralized request logging with complete audit trails that meet SOX compliance standards, and a single source of truth for all requests.”

EU AI Act high-risk obligations are an active compliance concern for many organizations right now, which makes the timing of a governance program's rollout as much a business decision as a technical one.

A governance framework you can actually implement

A workable governance framework is a sequence, not a checklist read in isolation.

Run it in order:

The four control surfaces at a glance

Each step exists because skipping it creates a specific, predictable gap: no inventory means shadow agents; no ownership means nobody to call when something breaks; broad permissions mean one compromised agent can touch everything; no decision trail means no defensible answer to an auditor; no runtime enforcement means policy exists only on paper.

Lifecycle: Governing agents from build to decommission

Governance doesn't stop once an agent is in production — it covers the whole lifecycle: environment separation from dev to staging to production with a review at each promotion, versioning of prompts, tools, models, and policies, a rollback path when a change misbehaves, and, at the end, decommissioning — revoking credentials and clearing memory when an agent is retired. That last step is the one organizations forget most often; an agent that's no longer used but still holds live credentials is a liability nobody's tracking.

Blessing Nuga, Director of Data Operations & Administration at Teachers College, Columbia University, describes what changes when versioning is treated as a real control rather than an afterthought:

“We've cut the time spent on manual updates by about 70% and dramatically reduced the email volume between academic departments and our compliance team. And the system has prevented errors from overwriting data or failing to archive prior versions.”

Governance is what lets you scale

Governance isn’t the tax you pay for deploying agents. It’s the precondition for deploying more of them. Every control described above — inventory, permissions, the decision trail, approval gates, runtime enforcement — is what converts one supervised pilot into fifty agents running unattended, because it’s what lets you trust an agent you're not personally watching.

Chris Hwang, Security Privacy and Resilience Lead at Block, describes what that looks like at scale:

“In my day job, Airtable let us retire multiple risk SaaS tools - cutting costs and letting us move faster. It's now the system of record for our GRC program, our Slack-based AI agents are the intelligence layer reading and writing it in conversation, and automations plus custom interfaces handle the rest.”

That answers the fear behind most governance hesitation — that tighter control means slower, more centralized, less flexible teams. It doesn't have to. Atlantic Records UK said:

“Each label can customize their own setup, and all data syncs to a central hub that gives global leadership visibility across the company.”

Local autonomy and central visibility aren't in tension when the system is designed for both from the start.