So, Your Company Wants to Govern AI. Where Do You Start?

Almost every company that starts an AI governance initiative this year opens with some version of the same question: "We need an AI governance framework. Where do we start?"

Here's a scene that's played out in a few hundred companies this year, and it'll probably play out in a few thousand more before the year is over.

Someone (a CISO, a General Counsel, a board member who just read an article about a chatbot lawsuit) sends a message that lands on an AI, product, or transformation leader's desk on a Thursday.

It says something like: "Can you get me a complete picture of the AI we're using across the company? The board's asking."

It sounds like a reasonable ask. It is, in practice, one of the hardest questions in the building to answer honestly.

Because the honest answer usually looks like this:

If you're the person who got that message, you don't have a framework problem. You have a "I genuinely don't know what I'm accountable for" problem.

That's some semblance of the real starting point for AI governance in almost every company right now. It is not a philosophical question about responsible AI principles, but a very concrete, very uncomfortable gap between what leadership assumes is true and what's actually running in production.

Today’s piece is written for the person who gets that message. Not the executive who sends it, the practitioner who has to answer it, build the thing that prevents it from being a crisis next time, and do it with a fraction of the authority and budget the job actually requires.

If you Google AI governance framework, you'll get a hundred variations of the same starting move: adopt a framework (NIST, ISO 42001, pick your acronym), write a set of principles, form a committee.

It's not wrong, exactly. It's just backwards. It's similar to building the roof before anyone's confirmed there's a foundation.

There's that one reframe that actually holds up once you're in the job: governance isn't a document, it's an answer to a question.

Specifically: if an AI system in this company does something wrong (makes something up, discriminates, leaks data, takes an action it shouldn't have), who finds out first, who's accountable, and how fast can we fix it?

If you can't answer that in one sentence, right now, for your company, that's your actual starting point. Not the framework. The inventory, the ownership, and the honest gap between the two.

That's not me being cute about semantics. It's the difference between the companies making real progress on this right now and the ones producing a very polished PDF that nobody's operations are actually built around.

There's a version of this conversation that's been happening in some form since ChatGPT launched in late 2022. So it's worth being precise about what's actually different now, because "AI governance is important" was already true three years ago and didn't require a deep dive to say.

What's different is the gap between how much AI companies are running and how well they're running it has gotten enormous, and it's still widening.

Roughly 88% of organizations use AI in at least one business function now. That's not the interesting number. The interesting number is that more than half of CEOs report zero measurable ROI from all that AI investment over the past year, and independent research keeps landing in the same range: something like a quarter to a third of AI use cases that make it to production actually deliver what they were expected to. That's not a technology failure.

Multiple research firms, looking at this from different angles, keep arriving at the same explanation: the bottleneck isn't whether the model works. It's whether the organization around it (ownership, monitoring, escalation, trust) was built well enough for anyone to actually rely on it.

The piece of that gap that should worry you most isn't the AI you already have; it's the AI that's coming next. Agentic AI (systems that don't just answer a question but take multi-step actions on someone's behalf) is the fastest-growing category of enterprise AI deployment right now, with something like three-quarters of companies planning to roll it out within two years.

Meanwhile, only about one in five companies say they have a mature way of governing agents at all, and more than a third admit that if an agent started doing something it shouldn't, they genuinely aren't sure they could shut it down cleanly.

Think about it: a third of companies building autonomous systems that take real actions don't have a confirmed kill switch. That's the gap you're actually inheriting if you're the one building this function.

By the way, there's no finished rulebook to lean on while you figure it out. This is the part that trips up a lot of practitioners coming from more mature compliance disciplines, where eventually the regulator hands you something close to a checklist. That's not what's happening here.

In the U.S., there's no single federal AI law — instead there's a fast-moving stack of state laws (Colorado, Texas, Illinois, California, New York City, and more joining regularly) that apply based on where your employees and customers physically are, not where your company is headquartered.

Layer on the EU AI Act if you have any European footprint — binding, extraterritorial, with real enforcement starting in mid-2026. Layer on voluntary-but-increasingly-expected frameworks like the NIST AI Risk Management Framework and ISO 42001, which nobody's legally required to follow but which vendors, customers, and auditors are starting to ask about anyway.

Nobody's going to hand you the finished answer. You're going to have to build toward a moving target using judgment, not a checklist someone else wrote for you.

None of this is meant to be alarming for its own sake. It's meant to explain why "just adopt a framework" isn't sufficient advice anymore, if it ever was. You're not implementing a known standard. You're building organizational muscle in real time, in a landscape that's still being written.

If you're building this function, the single most useful thing you can do early is train your eye to recognize the failure pattern before it shows up in your own company. It's the same pattern every time, across totally unrelated industries, which is exactly why it's worth studying closely rather than dismissing as "that could never happen here."

Air Canada
Air Canada's chatbot invented a bereavement discount policy that didn't exist. A customer, grieving, asked the airline's support chatbot about a discounted fare. The bot made something up — confidently, specifically, wrong. When the customer tried to claim it, the airline's actual legal defense was that the chatbot was "a separate legal entity responsible for its own actions." A tribunal rejected that outright and held the airline liable. Read that defense again: a company tried to argue in a legal proceeding that its own product wasn't really it. That's not a hypothetical risk anymore; it's a decided case. Any AI system that talks to your customers is making representations your company will be held to, full stop.

Samsung
Samsung engineers pasted proprietary source code into consumer ChatGPT while debugging. No malice, no policy violation in their minds. They were just trying to get their job done faster, using the best tool they knew about. Samsung found out, banned generative AI company-wide within weeks, and scrambled to build an internal alternative. Notice the sequence: ban first, alternative second. That's backwards, and it's the single most common mistake in this entire discipline, one we'll come back to.

Meta
An internal AI agent at Meta told an engineer to take an action that ended up exposing sensitive data for about two hours before anyone caught it. The employee didn't do anything reckless; they trusted a system that was designed to be trusted, and the system was wrong. This is the one to sit with if you're building anything agentic: an AI's instructions to a human carry the same real-world weight as a manager's instructions. Nobody's built the equivalent of "always double-check what your boss tells you to do" for AI-generated instructions yet, and most companies don't have oversight on what their agents are telling people to do, only on what the agents themselves output.

Dutch Government
The Dutch government used an algorithmic fraud-detection system that disproportionately flagged families with dual nationality, leading to wrongful benefit clawbacks that financially devastated tens of thousands of people, and ultimately helped bring down the government. This is the one that should recalibrate how seriously you take just a scoring model. When AI shapes a consequential decision about a real person's life without adequate testing or a real human check before the harm lands, the ceiling on how bad it gets is a lot higher than most people assume walking into this work.

Four industries — travel, semiconductors, social media, government. One root cause, every time: an AI system was given real authority (to speak for the company, to instruct an employee, to determine a benefit) without the infrastructure to catch it before something went wrong. That's the pattern. Once you can see it, you'll start noticing where it's quietly forming in your own company, usually in a system nobody thought of as AI governance's problem because it wasn't the flagship chatbot. It was the quiet feature buried in a tool everyone already trusted.

If you're new to this role, it's tempting to think the job is mostly about risk assessment, evaluating whether a given AI system is dangerous. That's just a small part of it. The harder, more persistent problem underneath almost every failure above isn't risk assessment; it's fragmentation. Nobody owns the whole picture.

Think about how AI actually enters a company in practice. IT stands up an approved enterprise tool. That's one thread.

A business unit buys a point solution because it solves their specific problem faster than waiting for a centralized process. That's a second thread, usually invisible to anyone outside that team.

A SaaS vendor the company already uses quietly ships an AI feature in a routine update, and it gets turned on by default. A third thread, often invisible to everyone, because nobody thinks of a product update as something that needs AI governance review.

And then there's whatever employees are doing on their own initiative with consumer tools, because the sanctioned option is slower or worse than what they can get for free. A fourth thread that mostly exists on paper as against policy and in practice as everyone's doing it.

Four threads, four different origins, and in most companies, zero people who can see all four at once. That's an org design problem wearing a risk costume. It explains something that trips up a lot of new AI governance leads. You can write an excellent risk framework and still fail, because the framework only works on things you can see, and most of the risk in your company right now is happening in the threads you can't.

There's a second layer of fragmentation worth naming, because it's more political and harder to fix with a document: governance and speed get treated as opposites inside most companies, and everyone loses because of it. The business side assumes legal and risk exist to slow things down.

Legal and risk assume the business will ship something reckless the moment nobody's watching. Both sides are usually right about the other, which is exactly the problem. It's an adversarial relationship dressed up as a process disagreement.

The research on this is actually pretty clear and somewhat counterintuitive: companies where senior leadership is genuinely involved in shaping AI governance (not delegating it entirely to a technical team and staying out of it) see meaningfully better business results from their AI investment than companies that don't.

Governance isn't competing with velocity for the same budget. In the data, it's a leading indicator of whether the velocity turns into anything real.

You don't need to become a lawyer to do this job well, but you do need enough working knowledge to know when to escalate to one. Here's the version that's actually useful day to day, not the exhaustive legal survey.

What you're dealing with instead is a state-by-state patchwork that applies based on where your employees and customers actually are, not your headquarters.

If you have employees in Illinois, you're inside the reach of a state law that makes AI-driven employment discrimination a civil rights violation.

If you have customers in Colorado or Texas, you're inside new state regimes governing algorithmic decision-making and discriminatory AI use.

If you touch New York City hiring, Local Law 144 requires bias audits on automated employment tools.

California has its own transparency and disclosure requirements layered on top.

There's a federal effort to preempt parts of this patchwork, and it's tied up in litigation as of this writing, which means, practically, you should assume the state laws apply to you until a court says otherwise, not the reverse.

General-purpose AI obligations are already in force. Enforcement from the European Commission starts in mid-2026. If any AI system you're using could plausibly be classified "high-risk" (anything touching hiring, credit, safety, or similarly consequential decisions), the obligations are substantial: documentation, human oversight, conformity assessment, post-market monitoring.

Worth knowing even if you have zero EU exposure: several U.S. state laws are explicitly borrowing language and structure from the EU AI Act, so building toward it now isn't chasing a separate track; it's getting ahead of where U.S. law is heading anyway.

NIST AI RMF is voluntary and free, organized into four functions (Govern, Map, Measure, Manage), and it's become the default methodology most companies build around because regulators reference it even without requiring it.

ISO 42001 is the first certifiable AI management standard, and it's increasingly showing up in vendor security reviews and enterprise procurement checklists, meaning it can become a genuine competitive requirement even for companies with no formal legal obligation to have it.

The practical sequencing that works for most companies: build your program's methodology around NIST AI RMF first, because it's free, flexible, and widely understood. Layer ISO 42001 on top once you want external, auditable proof the program is real, usually a few months of work once the NIST foundation is solid.

Then layer in EU AI Act or specific state-law compliance based on your actual jurisdictional footprint, not a blanket assumption either way.

These frameworks share the large majority of their underlying controls (risk assessment, human oversight, documentation, monitoring) so if you build one well, you've already done most of the work for the others. Treating them as three separate projects is the single most common way companies waste budget on this.

You don't need to invent a governance structure from scratch. There's a model that's been battle-tested for decades in other risk disciplines, and it adapts cleanly to AI. It's called the three lines of defense, and if you've worked anywhere near financial services, insurance, or internal audit, you already know it.

This is where the actual risk lives, because this is where the actual decisions get made: which tool to use, how to prompt it, what data to feed it, when to trust its output. The goal here isn't to slow this group down with a separate approval process; it's to build the guardrails directly into the tools they already use.

An approved-tool catalog, automatic logging, a pre-deployment checklist baked into the deployment pipeline. If your only control is a policy document, it will lose to a deadline every single time. If the control lives inside the platform itself, it survives.

This group sets the policy, defines how you tier risk, runs independent review on the higher-stakes use cases, and keeps track of the regulatory landscape described above. This is typically where the AI governance function itself structurally sits, whether that's under a Chief AI Officer, a General Counsel, a CRO, or a dedicated governance lead reporting into one of them.

Independent from both of the above, this group's job is to check whether the first two lines are actually doing what they say they're doing, not just that a policy exists, but that the inventory is genuinely complete and the reviews are genuinely happening.

The reason this structure works better than most companies' first instinct (a single cross-functional committee that tries to do everything) is that it forces a specific person or team to own each part, instead of diffusing responsibility across a room.

Committees are good for coordination. They're bad at accountability, because the committee decided is nobody's fault when something goes wrong.

One structural decision worth making explicitly and early: what kind of governance leader does your company actually need? There are two real archetypes showing up in practice right now, and conflating them is a common early mistake.

A governance-first leader spends most of their time on risk, compliance, bias testing, and regulatory mapping. This is the model that dominates in regulated industries, and it typically reports into risk, legal, or directly to the CEO.

A product-first leader sits closer to engineering, is measured more on adoption and feature velocity than compliance coverage, and typically reports into product or technology leadership.

Neither is wrong, but hiring or building toward the wrong one for your company's actual risk profile (putting a product-first person in charge of what's really a governance-first problem, or vice versa) is a mismatch that tends to surface at the worst possible moment, usually during an incident or an audit.

Skip the year-long framework design project. Yes, it sounds thorough, but in practice, it's how governance programs die. By the time the framework's finished, half the company has moved on to a new set of tools the framework never anticipated. In order to see any meaningful progress, you have to compress the foundational work into roughly a quarter, ship something real, and iterate from there.

Don't run a survey asking business units to self-report their AI usage. It will systematically undercount, because most people don't think of the tool they use as AI. They may think of it as the thing that helps me write emails faster. Instead, triangulate: pull procurement and contract data for anything AI-adjacent, check network and data-loss-prevention logs for traffic to known AI platform domains, use whatever cloud access security tooling you have to classify SaaS usage, and then run an anonymous survey, framed explicitly as intelligence-gathering, not a compliance trap. If people think admitting they use an unsanctioned tool will get them in trouble, they'll lie, and you'll have wasted the one honest data source you had access to.

Build the inventory as a living register from day one, not a spreadsheet you'll update eventually. It needs to capture more than the obvious stuff (the enterprise copilot, the internal chatbot) it needs to capture AI features embedded inside tools you already license, because that's where a lot of the invisible risk actually lives.

And get one uncomfortable question answered explicitly, in writing, by whoever's above you: who's actually accountable if something goes wrong? Not a committee, an actual name of someone in the company you can reach out to if necessary. If you can't get a straight answer to that question in the first month, that's diagnostic information about how seriously the company is actually taking this. It tells you exactly how much political capital you're going to need to secure before you can build anything real.

Take everything in the inventory and sort it by consequence, not by technology type. Don't build separate tracks for AI versus Generative AI versus Regular Software. Build one scale based on what happens if it's wrong: does it touch hiring, pricing, eligibility, safety, or a public-facing commitment your company will be held to (see: Air Canada). That's your risk tier. Everything gets sorted against it, regardless of what's technically running under the hood.

Map your actual jurisdictional exposure honestly: which state laws apply based on where your people and customers really are, whether EU AI Act obligations apply based on your actual market presence. Most companies genuinely can't answer this with confidence at this stage, and that's fine. The point of this exercise is to stop guessing.

For anything you find that's high-risk and already running with zero oversight, don't wait for the full program to patch it. Apply something immediately (a human review checkpoint, a usage restriction, tighter logging) even if it's imperfect. A system running completely ungoverned for six months while the real program gets built is the actual failure pattern that shows up in incident reports later, not a hypothetical.

If your discovery process turned up meaningful shadow AI usage (it almost certainly will), don't respond with a stricter policy memo alone. Go back to the Samsung example: banning without offering a better sanctioned alternative doesn't eliminate the behavior; it just teaches people to hide it more carefully. Get a genuinely usable, sanctioned tool in front of people before you tighten restrictions, not after.

Pick one actual, moderate-risk production use case and run it completely through your new process, start to finish: intake, tiering, review, deployment with monitoring, a named owner. This does two things nothing else can. It tells you honestly whether your process holds up under real conditions instead of theoretical ones, and it gives you a concrete reference point to show skeptical stakeholders what “Governed” actually looks like in practice, instead of asking them to trust a slide.

Wherever you can, build the controls into the actual tools people use rather than a separate approval step bolted on afterward. That's the difference between governance that survives a deadline and governance that gets quietly skipped the first time someone's in a hurry.

Build your monitoring cadence around how the technology actually changes, not your old software release calendar. A vendor's foundation model can shift underneath you on their schedule, not yours. Set concrete thresholds (drift, refusal-rate changes, a spike in something going wrong) that trigger a human review, rather than waiting for an annual check-in that might be eleven months too late.

After 90 days, this becomes a scaling exercise. Expand the governed process to the rest of the inventory by risk tier, formalize how you assess vendor and embedded AI risk (because you own the outcome of a tool you bought exactly as much as one you built), and start layering ISO 42001 or jurisdiction-specific work on top of the foundation you've already proven works.

A lot of companies now claim, when asked, that they have clear AI governance, but a much smaller number actually have the underlying controls to back that claim up. That gap is the single most common failure mode, and it's an easy trap to fall into, because a principles page is genuinely fast to produce and genuinely satisfying to publish. An inventory, a tiering system, and a monitoring process are none of those things. Do the unglamorous work first.

If AI governance sits entirely inside engineering or IT, legal exposure, workforce discrimination risk, and reputational risk get systematically underweighted, not because the technical team is careless, but because that's genuinely not their expertise, and it's unreasonable to expect it to be. The research bears this out directly: companies where senior, cross-functional leadership actually shapes the governance program see meaningfully better outcomes than companies that delegate it entirely downward and step away.

It almost certainly doesn't, and not because anyone did anything wrong. It's because most existing review processes were built for software with a fixed release cycle and predictable behavior. Generative and agentic systems don't behave that way. They change on a vendor's schedule, produce different outputs from identical inputs, and — increasingly — take real actions rather than just generating text. A review process built for the old assumptions will miss the new risks by design, not by accident.

As mentioned above, worth saying again because it's the mistake that recurs most often: restriction without a real alternative doesn't solve the problem; it just makes the problem harder to see.

Federal AI policy in the U.S. is genuinely in flux and could shift again. State enforcement isn't slowing down, and the EU AI Act doesn't care what's happening in U.S. federal politics at all. A program built entirely around today's federal posture will be exposed the moment a state attorney general, a European regulator, or a plaintiffs' attorney takes interest. Based on what's already happened in Colorado, Texas, Illinois, and the EU, several already have.

It's intuitive to scrutinize the AI tool your company built or explicitly purchased, and much easier to overlook the AI feature quietly living inside a platform you've used for years. That instinct is backwards, and the Air Canada case is the clearest evidence why: "the vendor built it" is not a defense that holds up, legally or practically. If your company relies on the outcome, your company owns the outcome, period.

No incident yet doesn't mean no risk. It often just means the risk hasn't surfaced publicly yet. Most of the failures worth worrying about take months or years to become visible, and by the time they do, they're usually already in front of an auditor, a regulator, or a courtroom rather than something you get to fix quietly first.

If you're reading this because you're trying to become the person who runs this function (not just execute someone else's mandate, but actually own it), a few things are worth knowing that most of the standard advice doesn't cover.

The Chief AI Officer role went from a rarity to something like three-quarters of large organizations reporting having one, in roughly a single year. That kind of growth curve means the discipline is still being defined in real time, by the people currently doing the work. Which means there's more room to shape what "Good" looks like than there is in a mature field with fifty years of established practice. You're not late to this. You're early enough that the norms are still forming, and being visibly thoughtful about it right now carries more weight than it will in five years once the playbook's fully written.

It's tempting to position yourself as broadly fluent in AI governance as a category. The fastest way to build real authority is to become the one who can speak fluently to a specific pattern (how this plays out in hiring, in customer-facing AI, in agentic systems specifically) rather than a surface-level familiarity with everything. Pick the pattern you understand best and go deep on it publicly. It's more differentiating than trying to cover the whole landscape evenly.

A framework you can hand someone — an inventory template, a risk-tiering rubric, a set of interview questions for evaluating a vendor's actual governance maturity rather than their marketing claims — does more to establish that you know what you're talking about than an essay explaining why governance matters. Anyone can argue governance matters, but only a few can hand you something you can literally use on Monday.

When something like the Air Canada ruling or the Meta agent incident shows up in the news, most people read it as a story. The people building real expertise here read it as a diagnostic: what was the actual control that was missing, and would my company have caught this?

That habit — translating every public failure into "what would I have needed in place to prevent this" — compounds faster than almost anything else you can do to build genuine judgment in this space, and it's exactly what makes someone credible in a room where the framework everyone already knows isn't the interesting part of the conversation anymore.

Companies have gotten much better at deploying AI than they have at governing it, and the gap is widest exactly where deployment is moving fastest. Agentic systems are where most companies plan to scale up fast, and few have confirmed they could shut one down cleanly if it went wrong.

There's no finished regulatory rulebook to lean on while that gap closes. Instead, there's a genuinely fragmented patchwork of state laws, a binding EU law with real reach, and voluntary frameworks the market is starting to treat as mandatory anyway.

The failure pattern that shows up across every industry that's had a real incident (Air Canada, Samsung, Meta, the Dutch government) traces back to the same root cause every time: AI was given real authority without the infrastructure to catch it before something went wrong.

The instinct to start with a framework or a principles document is understandable, but it's backwards. The evidence is consistent on this point: companies where senior leadership genuinely shapes governance, rather than delegating it entirely and stepping away, see meaningfully better results from their AI investment than the companies that don't. The people making real progress didn't start with a document. They started by finding out what was actually running, naming a real owner for each piece of it, and building the controls into the tools people already use instead of bolting on a separate approval gate nobody follows under deadline pressure. Everything after that (the committee structure, the framework alignment, the jurisdictional mapping) is real work, but it's work that comes after the foundation, not instead of it.