Actually getting value out of AI

Anyone using AI – specifically, LLMs – eventually has to deal with the same question. How do you trust the output of a tool that’s capable, useful, and confidently wrong some of the time? It’s a real question, and a hard one, and it’s the question that determines whether AI in underwriting, credit, claims, pricing, hiring, procurement, or anywhere else with non-trivial judgement at scale, actually pays for itself.

It turns out software engineers have already worked through a version of this problem, and I think the solution is genuinely useful for everyone. This gets a little bit ‘techie’, but I think it’s worth your while… because the parallel is unusually clean and the lesson maps surprisingly well onto the kind of decision-making most large organisations are trying to put AI into.

The compiler nobody reviews

Hugo Venturini, an engineer at SkipLabs, wrote a piece recently that names this problem cleanly in the engineering context. His subject is AI-generated code, but the underlying point is very good.

His starting point is a tool called a compiler. When a programmer writes code, a compiler converts it into the binary instructions a machine actually runs. Nobody reviews the compiler’s output. Nobody opens the binary, reads through the assembly, and signs off. The reason is not that everyone naively trusts compilers. Compilers have bugs, and have had famously catastrophic ones. The trust isn’t blind, it’s earned.

What engineers built, instead of reviewing compiler output, is an apparatus around it. ‘Type systems’ constrain what the compiler can produce. ‘Test suites’ verify behaviour against specifications. ‘Production monitoring’ catches problems quickly and rolls back faster.

The insight here is subtle - quality control moved upstream and downstream. The manual middle, where a human eyeballs every artifact, was eliminated.

Venturini’s argument is that AI-generated code feels uncomfortable to engineers for the same reason corporate AI deployment feels uncomfortable to risk officers and operating committees: the apparatus around it doesn’t exist yet. The discomfort isn’t irrational. It’s the right response to using a non-deterministic tool in a workflow that was designed around determinism. The fix isn’t to keep reviewing every output. The fix is to build the equivalent apparatus.

This really matters, because the same category error is happening across corporate decision-making, and almost nobody is naming it.

The same mistake, in a different room

Walk into a large insurer, a hospital group, a bank, a logistics company, a retailer with a meaningful pricing function, anywhere with serious decisions being made at scale, and you’ll find one of two patterns in AI usage today.

The first is the conscientious one. Someone senior has insisted that every AI-generated output in a regulated or high-stakes function be reviewed by a person before it has effect. Every underwriting recommendation, every credit memo, every claims summary, every shortlist of candidates. This feels prudent. It is also a productivity ceiling dressed up as governance. Once the cost of the model and the analyst checking the model are added together, the firm has bought itself a moderately faster way to produce drafts. The transformation everyone was promised in the board paper is nowhere to be seen.

I think it’s also a dishonest approach. It’s either arrogant – a human doesn’t make mistakes, the AI does; or just a way of doing CYA (cover your rear-end) – you now have a human to blame if a mistake is made (which it inevitably will – humans also have an error rate). It’s a way of showing regulators or customers “we didn’t trust AI, we just had a human failure” rather than actually designing a way to do better.

The second pattern is the reckless one. The tool gets deployed into a workflow built around deterministic systems and human-paced volume, with no equivalent apparatus around it. Then something embarrassing happens. A chatbot promises a refund the company has to honour. A legal brief cites cases that don’t exist. A pricing model drifts and underwrites a portfolio nobody intended. The headlines follow, the project gets paused, and the cycle resets a year or two later when memories have faded.

Both patterns share a single root cause. The firm is treating an LLM as if it were the last generation of corporate technology. Workflow software, robotic process automation, traditional credit scoring models: these were deterministic, predictable, and produced output at a volume a human could plausibly check. The integration model that grew up around them was reasonable for tools of that kind. Define the rule, run it through the system, have a person sign off at defined gates.

LLMs are not tools of that kind. They are by design non-deterministic, which means the same input can produce different outputs. They generate plausible-sounding text at a volume that makes line-by-line review impossible at any honest scale. And the failure modes are subtle in a way that traditional model errors are not. A credit memo can be ninety-five percent reasonable and five percent confidently wrong, with the wrong part being precisely the part that matters.

The category error is to use these tools in the old integration model. The discomfort that risk officers feel about AI in decision-making is, like the engineers’ discomfort with AI-generated code, a correct response to the wrong setup. The answer is not to push through the discomfort by adding more reviewers, and it isn’t to ignore it and deploy anyway. It’s to recognise that the integration model itself has to change, in roughly the same direction the engineering world has already been moving for a long time.

What it looks like to build for the tool

If you take Venturini’s three-layer apparatus and translate it into corporate decision-making, you end up with something that looks meaningfully different from what most firms are doing today.

Upstream, the equivalent of a type system is a machine-readable specification of the decision itself. What is the firm’s actual underwriting appetite, expressed in a form a model can be tested against? What are the policy constraints on a credit decision, beyond the regulatory minimum? What does an acceptable hire actually look like, in terms specific enough to evaluate against? Most firms have these things written down somewhere, in committee minutes and policy PDFs, but they don’t exist in a form that can constrain or verify a model’s output. Building that specification layer is unglamorous work, and it’s where most of the value sits. It also forces a useful discipline. A firm that can’t articulate its risk appetite in machine-readable terms probably doesn’t have as clear an appetite as it thinks.

The equivalent of a test suite is adversarial AI verification. Use a second model, with a different prompt and ideally a different architecture, to check the first model’s output against the specification before it has effect. This is the layer developing fastest among sophisticated deployers, and it’s the layer that lets you actually move past line-by-line human review without going blind. AI checking AI sounds paradoxical, but it isn’t. It’s the only thing that scales, and the engineering world has been moving toward it for several years.

Downstream, the equivalent of production monitoring is outcome instrumentation with fast rollback. What did the model recommend, what was the actual outcome, where is drift appearing, and how quickly can the firm pull the model back if something starts going wrong? Most firms have some version of this for their traditional models, with quarterly performance reviews and annual recalibrations. The cadence required for LLMs is closer to weekly (maybe even daily?) than quarterly, and the rollback has to be a button rather than a project.

These three layers aren’t a checklist. They’re a pattern. Together they describe a coherent way of integrating a non-deterministic tool into a serious decision function, one that doesn’t depend on the manual middle holding up under volumes it was never designed for. The firms working on AI deployment seriously are converging on something like this shape, even if they don’t describe it in quite these terms. The firms still trying to retrofit AI into their existing review processes are the ones disappointed with their pilots.

A pattern, not a working apparatus

The compiler took decades to acquire the apparatus that makes it trustworthy, and the corporate equivalent for LLMs is being worked out in real time. Anyone claiming to have the definitive playbook is selling something. The direction is clear, even when the destination isn’t.

Every previous wave of corporate technology eventually forced firms to rethink their integration model. Mainframes did. Client-server did. The internet did. Cloud did. In each case, the firms that recalibrated quickly to the actual nature of the new tool, rather than wedging it into the previous model, captured a disproportionate share of the gains. The firms that wedged got the costs without the benefits.

LLMs are at that inflection point now, and the wedge is currently winning. Most boards and operating committees are still authorising AI deployments using a mental model that assumes the tool is deterministic and the volume is human-scale. The deployments that result are either bottlenecked by human review or quietly accumulating the kind of failure modes that eventually surface as a headline.

I am curious to see who is able to rethink their governance and control models fastest. I suspect it’ll be the companies with leadership who have their own coding experience, but I am happy to be proven wrong….