The Validation Debt Comes Due

Your procurement team is buying AI tools that wouldn't clear the validation bar applied to a printer cartridge, and the bill for that shortcut is about to land.

The defense writes itself. Vendors point to internal evaluations, model cards, benchmark scores. Procurement points to vendor disclosures. CIOs point to platform-level guardrails. Legal points to terms of service. Everyone's covered. The system works. Sort of.

Ontario's government watchdog reviewed 20 AI medical scribes in clinical use this month. All 20 carried hallucination risk. Not a sample skew — a clean sweep on a variable that wasn't supposed to be universal. These tools write into patient charts. They were procured through pathways that nominally exist for medical software. They still got in.

Same week, OpenAI shipped read access to bank transactions, packaged as user empowerment. No disclosure of what safeguards, if any, sit between a hallucination-prone model and your account history. The empowerment frame isn't an accident; it puts the liability on the user for granting access. Read the announcement and try to find the word "audit." You won't.

This is validation debt. The gap between what got bought and what got tested. Healthcare carries it inside a regulated procurement system. Financial services carries it outside one. The procurement layer is where AI keeps slipping past governance, because the governance assumed procurement would catch obvious things, and procurement assumed the vendor would.

Both assumptions are wrong. The Ontario number is hard to talk past. Twenty out of twenty isn't a Canadian problem. It's a snapshot of what happens when nobody's playing the regulator.

Grant the counter. Twenty tools in one province isn't the world. Extrapolating to systemic failure off one audit is rhetorical force, not statistical proof. Fair. But the prior on "AI tool ships with hallucination behavior" isn't 50 percent. It's closer to 100, and a clean sweep confirms what we already know. The audit isn't novel evidence. It's documented evidence — which is the part that makes lawyers move.

The uncertainty I'll own: I don't know which sector's enforcement action lands first. Could be a state AG with a scribe complaint. Could be a consumer-protection regulator on an AI-mediated financial advice incident. Could be a tort case after a model summarizes a transaction history wrong and someone makes a decision off it.

What I'm sure about: within 18 months, a regulator names AI procurement, not AI safety, as the category of governance failure. The enforcement target will be a buyer, not a vendor. The first executive testimony will include the sentence "we relied on the vendor's representations." It will not save them.

The validation debt has been accruing on every quarterly feature adoption that skipped the part where someone asked what would happen if the output was wrong. The interest compounds in the form of patient harm, account errors, and discovery requests. It gets paid eventually. The only variable is who signs the check.

Buy accordingly. Or don't, and find out.