Why AI Regulation Keeps Missing the Actual Risk

Sit through enough AI policy discussion and a pattern emerges. The conversation keeps landing on one of two poles. Either someone is asking whether a sufficiently capable system could eventually escape meaningful human control, or someone is proposing a rule that names one specific use case and restricts it precisely. What almost never comes up, with the same urgency, in the same room, is the question that actually occupies the people who build and ship these systems for a living: where did the data come from, and who is accountable when the system gets something wrong in front of a real person.

We don't think this is a conspiracy or a failure of intelligence on anyone's part. It's a structural pattern, and it's worth understanding on its own terms, because it explains why so much AI regulation ends up either toothless, overbroad, or both — and why the risks most worth legislating are frequently the ones least likely to trend. It also explains why practitioners so often describe policy hearings as being about a different industry than the one they actually work in every day.

The Two Modes Regulation Defaults To

Almost all AI policy conversation gravitates toward one of two modes. The first is existential framing: could a much more capable system pursue goals that diverge from what its operators intended, at a scale that matters. That's a real question worth serious research, and it produces gripping hearings, because the stakes described are about as high as stakes get. But it is, by construction, a question about a hypothetical future system's behavior, which makes it nearly impossible to translate into a specific near-term rule. You can fund research into it, you can ask developers to commit to safety practices that scale with capability, but you can't write a compliance checklist for a scenario nobody has actually observed yet.

The second mode is the opposite: narrow and concrete. Name a specific application in a specific context and restrict it — a particular biometric use, a particular category of synthetic content, a particular sector's reliance on automated decision-making. This is legislatively tractable precisely because it's specific: you can write a bright-line rule, assign an enforcement body, and defend the rule in a single sentence to anyone who asks. The tradeoff is that it only covers exactly what it names, and general-purpose systems, by definition, get used in ways nobody wrote down in advance. The rule goes obsolete, or gets trivially routed around, at roughly the same pace the technology generalizes past the specific case it was written for.

Both modes are attractive to legislate around for the same underlying reason: they're legible. A hypothetical catastrophe and a named use case are both things you can state in one sentence and defend on a stage. The risks in between mostly are not.

The Boring Risk That Doesn't Fit a Press Release

Ask anyone who actually builds these systems what worries them day to day, and data provenance comes up constantly — and it's almost never in the room when policy gets made. The question is simple to state and hard to answer: what went into this system, was it obtained with appropriate rights or consent, and can anyone, the deploying company, a regulator, an affected person, actually trace a given output or behavior back to something in that input in a meaningful way. This isn't a hypothetical concern about a future system doing something dramatic. It's a present, structural fact about every system currently in production, and it quietly determines how much anyone can trust any downstream claim about that system at all.

It doesn't produce a press release because there's no single dramatic moment attached to it: no headline event, no named victim, no clean before-and-after. It's a property of a pipeline assembled over months, often by people several organizational layers away from whoever eventually ships the product. Most people affected by an AI system's output never experience "data provenance" as a category at all; they experience whatever downstream effect it produces, three steps removed from the actual decision that created the exposure. That distance is exactly why it's hard to legislate, and exactly why it matters — the harm and its cause are separated by enough steps that no single moment forces the question onto anyone's desk until something has already gone wrong.

Nobody Wants to Own 'What Happens When It's Wrong'

Here's a structural problem regulation hasn't solved yet, and it's not for lack of trying: when an AI-assisted decision harms someone, most existing frameworks assume you can trace the harm to a single identifiable actor — a manufacturer, a licensed professional, a company that made a specific, verifiable claim. AI systems break that assumption by design. A harmful output might trace back to the base model's training, a fine-tuning step performed by a different company, a prompt or tool built by a third-party integrator, or a human reviewer who was supposed to catch the error and didn't, because the interface made it too easy to rubber-stamp. Every one of those parties can reasonably say the failure wasn't entirely theirs, and in a narrow sense they're all correct, which is exactly the problem. Diffused responsibility isn't a loophole anyone is exploiting; it's a genuine structural feature of how these systems get built and shipped through a supply chain of contributing vendors.

Existing liability frameworks, built for a world with a single identifiable manufacturer or a single licensed professional making a judgment call, don't map cleanly onto that structure. Writing new frameworks that do requires deciding, in advance and in the abstract, how to apportion responsibility across a chain of contributors who may never have directly coordinated with one another. That's a hard, unglamorous drafting problem with no obvious villain to name in a hearing — which is probably a large part of why it lags so far behind the more legible modes of regulation.

Evaluation Standards: The Infrastructure Nobody's Excited to Fund

There's a less obvious risk that's arguably load-bearing for everything else: there's no widely trusted, independent way to verify what a given AI system can and can't reliably do, comparable to how other regulated industries verify claims through audited financials, clinical trials, or safety certifications issued by an independent body. Without that kind of shared evaluation infrastructure, regulators are stuck choosing between two bad options — take a vendor's own claims about their system's capabilities and limitations at face value, or write rules broad and cautious enough to cover the uncertainty, which tends to restrict beneficial uses right alongside genuinely risky ones, simply because the rule has no way to distinguish between them.

Building real evaluation standards, meaning agreed methodology, independent administration, and results that mean the same thing across different systems and different evaluators, is exactly the kind of unglamorous infrastructure project that never gets a ribbon-cutting. Nobody holds a press conference to announce a new testing methodology. But its absence is a big part of why AI regulation keeps oscillating between toothless and overbroad — without a trusted way to verify a narrower, more calibrated claim, lawmakers reasonably default to blunter instruments. Imagine two systems marketed as performing the same task, one rigorously tested against a shared, independent standard, and one tested only by its own maker on data that maker selected. From the outside, in the absence of shared evaluation infrastructure, those two claims are almost impossible to tell apart, and regulation ends up treating them identically by default, which shortchanges the one that actually earned its claim. This is fixable, but it's fixable through years of unglamorous standards work, not through a single dramatic hearing that makes for good coverage.

Why the Mismatch Persists

None of this is a partisan failure or a failure of any particular set of people. It would show up regardless of who is writing the rules, because the underlying incentive shape is the same everywhere. People who write and pass laws are rewarded for responding to salient, easily explained risks — the kind you can describe in a single sentence to someone who wasn't already following the topic. Deep procedural risks like data provenance, diffused accountability, and evaluation infrastructure don't have a natural constituency demanding action, don't have a single bad actor to point to, and don't produce a moment anyone can cite as the reason a law was suddenly needed. The people who understand these risks best, the ones building and operating these systems day to day, are rarely the ones drafting policy — and when their concerns do reach a hearing room, they tend to get compressed into whichever of the two legible modes the room already understands, existential or narrow-use-case, losing the actual structural point somewhere in the translation. This is also why practitioner input tends to arrive too late to shape the framing of a bill, even when it's solicited in good faith: by the time a hearing is scheduled, the two legible modes have usually already set the terms of the debate, and testimony about provenance or evaluation infrastructure has to fight for room inside a frame that wasn't built to hold it.

None of this means the dramatic scenarios or the narrow use-case rules are wrong to pursue; some of them deserve exactly the attention they get. But if the industry and its regulators only ever get animated about risks that are easy to narrate, the boring risks don't go away. They just go unaddressed until they surface as a scandal dramatic enough to finally force legislative attention, usually well after the damage is already done. The fix isn't louder alarm bells. It's making the unglamorous risks legible enough to compete for attention on their actual merits, instead of on their storytelling potential.