AI Eats Software.Trading Determinism for Probability

In late 2024, on the BG2 podcast, Satya Nadella said the quiet part loud. Most business applications, he argued, are just CRUD databases (create, read, update, delete) wrapped in a layer of business logic. That logic is about to migrate into an AI agent layer, and when it does, the applications sitting on top will collapse. The internet compressed it to three words: SaaS is dead.

It's no longer a lone hot take. Developers now claim they can rebuild the core of Booking.com in an afternoon or stand up a Salesforce competitor in a couple of weeks with an agent doing the heavy lifting. Analysts are pricing the shift in: IDC expects seat-based software pricing to be largely obsolete by 2028 as vendors refactor around consumption and outcomes rather than logins. And the startup world has distilled it into a validation heuristic: does your idea survive being handed to a sufficiently capable model? If a smart enough AI could just do the thing from a plain description, why build the product at all? Describe the workflow, invoke the model, collect the outcome. No dashboards, no forms, no UI. In this picture AI doesn't compete with software. It eats it.

Is that actually feasible? Not "will it sell" or "is it a good idea", but does the engineering hold up? That's the question this piece sets out to answer, and it's worth taking seriously rather than dismissing or cheerleading. What follows is the analysis. Worth saying up front: part of the premise survives it intact, and part of it dissolves the moment you look at where the reliability actually comes from.

The question everyone is skipping

The debate collapses, every time, onto capability. Will the model be good enough? Good enough to plan, to reason, to recover from its own mistakes, to replace the thing you would have built. And the honest answer is: probably yes, for most things, most of the time. So grant it. Assume the capability arrives on schedule. That assumption doesn't settle the interesting question; it exposes it.

Here's the part the "SaaS is dead" framing walks straight past. A model is a probabilistic system. It samples its output from a distribution; that isn't a defect waiting to be patched, it's what the thing is. And "usually right" is a triumph for a search box, a translator, a first draft, and a catastrophe for an anti-lock braking system, an infusion pump, a settlement engine, a flight controller. In those systems the most capable decision-maker isn't what you want. You want the most predictable one: the one whose failure modes are mapped, whose behavior reproduces, whose wrong answer can be traced afterward and ruled out forever.

So the real question was never "can AI do it?" It's this: for the software where a single wrong call is expensive or fatal, is determinism worth trading away for probability, just because the probabilistic thing is more capable?

That's a trade, not an upgrade. It buys enormous flexibility and it costs you the guarantee. Sometimes it's obviously worth it. Sometimes it obviously isn't. And here's what I find genuinely worrying: the trade is being made by default (because writing the prompt is easier than architecting the system) instead of on purpose, domain by domain, where the stakes actually differ.

Reliability is a property of systems, not components

The reframe that clears the fog: "can the model become reliable enough to trust on its own" is a malformed question, because reliability almost never comes from a flawless component. It comes from architecture that corrects the errors of unreliable ones.

This is true nearly everywhere that matters. TCP delivers your data intact across physical links that drop, corrupt, and reorder packets constantly, through checksums, acknowledgements, and retransmission, not a perfect wire. Storage and memory hold bits on noisy physical media and reach vanishingly small error rates using error-correcting codes that repair corruption in flight. DNA replication miscopies bases as it goes, then proofreading and mismatch-repair machinery drive the effective error rate down to roughly one in a billion: the polymerase isn't perfect, the system is. A cockpit pairs two nondeterministic pilots with checklists, cross-checks, and air traffic control, and the assembled system is among the safest things humanity has ever built.

None of those components is 99.999% reliable. The systems around them are. "Make the model individually perfect" was always the wrong target. The right question is whether a nondeterministic model can be wrapped in an architecture that corrects it, and, crucially, when it can and can't.

The real line: verifiability, not capability

The answer turns on one thing, and it isn't intelligence. It's whether you can cheaply check the output.

Where verification is cheap, nondeterminism stops mattering. Does the code compile? Does the test pass? Is the number inside the allowed bounds? Does the proof check? When you have a fast, truthful way to ask "was that right?", you wrap the flaky model in a generate-check-retry loop and pull effective reliability as high as you want. You're no longer trusting the model. You're trusting the check. The model can be wrong nine times as long as the tenth passes a real test.

Where verification is expensive or impossible, no amount of capability saves you. Is this the right architecture for a system that doesn't exist yet? Is this the correct call in a situation nobody anticipated? There's no unit test for that. And here's the quiet cruelty of it: the domains where checking is hard are disproportionately the high-stakes ones: open-ended judgment, novel situations with no ground truth to check against. There's a trap in the cheap case too: where verification is easy, you can often just solve the problem directly and skip the model. The expensive-to-verify case is exactly where you most want the model to rescue you, and exactly where it can't.

That's not an engineering gap a bigger model closes next year. It's structural.

Figure 1. What actually gets eaten. Green where verification is cheap or stakes are low; red bottom-right where determinism still earns its keep.

Why being smarter doesn't make it deterministic

Smarter and more deterministic are different axes. Scale moves the first and leaves the second essentially untouched. A bigger, better-trained model samples from a better distribution, but it's still sampling. You can drop the temperature to zero and take the single likeliest token every time, but that isn't robustness; it's one fixed path through the same uncertainty, and floating-point non-associativity across hardware can make even that non-reproducible. Reproducible isn't the same as correct. You can get the identical wrong answer every time.

And the errors don't stay put: they compound. Suppose each step in a chain is 99% reliable and errors go uncorrected. Over a hundred steps, overall success is 0.99¹⁰⁰ ≈ 37%. Worse than a coin flip, not because any single step is bad, but because independent 99%s multiply.

Figure 2. Uncorrected per-step reliability compounds. At 99% per step, success after 100 steps is ≈37%, a coin flip.

To be clear about the model: that curve assumes independent per-step errors, which real chains violate (errors correlate, and a good system catches some). It's the mechanism, not a measured rate. But the mechanism shows up plainly in the best empirical work on agents to date. METR measures the length of task (in human time) that frontier agents can complete autonomously, and the headline is genuinely striking: that length has been doubling roughly every seven months, faster on recent models. Then look at what happens when you raise the reliability bar. The task length an agent can handle at 50% success is far longer than the length it can handle at 80%: in their data, a model good for around an hour of work at coin-flip odds drops to roughly fifteen minutes once you demand it actually succeed most of the time. Current systems are near-perfect on tasks that take a human a few minutes and fall below ten percent on tasks that take more than a few hours. METR is explicit that real work often needs not 50% but 80%, 99%, or 99.9999% reliability, and the achievable horizon collapses as you climb toward those numbers. That collapse is the compounding curve, measured in the wild.

The fix isn't a better step. It's error correction between steps. Check and repair at each checkpoint and the multiplication breaks: each stage restarts from a verified state, so failures stop compounding.

Figure 3. Same model, different scaffolding. Without checkpoints, reliability decays; with a real check at each step, it holds.

That's the whole case for observable, checkpointed systems over one long opaque generation. It's arithmetic, and it's benchmarks, not nostalgia for the old way of building.

But won't infinite context fix it?

The proponents have an answer ready, usually some version of this: the real unlock is a model that holds everything (your whole codebase, every document, the full history) in one enormous window, plus near-infinite memory. Then it won't lose the thread, and reliability follows. This promise is doing a lot of load-bearing work in the optimistic case, so it's worth taking apart.

The first problem is cost. Standard attention is O(n²) in sequence length: double the context and you quadruple the compute. You can't simply scale this to a billion tokens; you hit economic and eventually physical walls. There's real, promising work on sub-quadratic architectures (linear attention, state-space models like Mamba, sparse and sliding-window schemes), and it genuinely softens the wall. But these typically buy efficiency by giving up precise long-range recall: pulling the exact right needle out of a very large haystack. So this isn't a solved problem you're waiting on. It's a live tradeoff between how much you can hold and how reliably you can retrieve from it, and worth watching honestly, because if anything moves the ceiling, it's here.

The second problem is deeper: even if context were free, it wouldn't buy reliability. More context often makes things worse. The best-known result is "Lost in the Middle" (Liu et al.): models use information best when it sits at the very start or end of a long input, and measurably worse when the relevant fact is buried in the middle: a U-shaped accuracy curve that appears even in models built explicitly for long context. Later benchmarks like RULER and ∞Bench reproduced and extended it: long contexts degrade not just retrieval but reasoning and aggregation. And the functional version of "unlimited memory" is arriving as retrieval and hierarchical memory anyway, not one giant window, which is how human memory works: you don't hold everything in mind at once, you retrieve. The punchline: context was never the reliability lever. You can have perfect memory and still reason wrong.

"But the future isn't LLMs: it's agents"

This is the strongest objection, and it deserves a real answer, because it sounds like it defeats everything above. Sure, a single generation is unreliable. But an agent isn't a single generation. It plans, acts, observes the result, corrects itself, retries, calls tools. The loop fixes the unreliability.

Look closely at what was just conceded. An agent is a model in a loop that observes outcomes and corrects. That's error-correction architecture: the exact thing that produces reliability in every system above. The agentic counterargument isn't a rebuttal; it's the thesis wearing a costume. It agrees that raw capability isn't enough, that you need a correction loop around the model. Good. So the only question that matters is the one from three sections ago: what is the loop checking against?

A correction loop is exactly as good as its error signal, and verifiability decides everything all over again.

Where the feedback is cheap and real (a compiler error, a failing test, a stack trace, an API returning 400, a type checker), agents genuinely shine. The agent isn't magic; it's getting a truthful oracle each turn and iterating against it. This is precisely why coding agents are the flagship success story of the whole agentic wave, and why METR's fastest-improving numbers are on software tasks: code is the richest, cheapest, most immediate verification environment humans have ever built. That's not a coincidence. It's proof of the thesis: agents are strongest exactly where checking is cheapest.

Where the feedback is absent or fake, the loop degrades, or actively hurts. If the agent can't cheaply verify whether an action was correct, it grades its own homework, self-correcting against its own probabilistic judgment of success: the same unreliable signal it started with. That doesn't converge to truth. It can converge, confidently, to a wrong answer, or wander off entirely. The loop amplifies whatever the error signal is: good signal, it moves toward correct; no signal, it amplifies noise.

And the compounding math gets worse for agents, not better (the opposite of what people assume). An agent takes far more steps than a single generation: plan, act, read, re-plan, act again. Every extra step is another factor in the product. If those steps aren't each verified against something real, "agentic" doesn't buy safety; it buys higher variance and more chances to compound. It's why METR's reliability curves fall off as tasks lengthen. The loop only breaks the multiplication when each checkpoint has a genuine verifier behind it. (Recall Figure 3: the green line holds only if the check is real.)

In fairness: agents do expand what's cheaply verifiable. A model that can run code, query a database, or hit a real API can sometimes manufacture a ground-truth check where a bare model only had a guess. That's real, and it's why agents are a genuine step forward. But it relocates the line; it doesn't erase it. The domains where no available tool can cheaply produce a truthful signal are exactly the domains where agents also stall, for the same structural reason.

The compressed version: a feedback loop with no cheap way to check its work is just nondeterminism with extra steps.

What actually gets eaten

So here's the honest map. AI (models and agents both) will eat the verifiable and the low-stakes with real appetite. Glue code, one-off automations, internal tools, the connective tissue where "usually right" is genuinely good enough and a mistake costs a shrug. Those don't need architecture; they need a model in a loop, and they're better for it. Nadella isn't wrong that a great deal of enterprise software is thin logic over a database, and much of that thin logic will, in fact, dissolve into an agent layer. That's most software by volume.

The bifurcation is real and mostly fine: prompt-first for the low-stakes and the checkable; deliberate, checkpointed, observable systems for the high-stakes and the unverifiable. The danger isn't that AI is unreliable; it's that "ship fast, usually works" is seductive enough to get reached for in domains where "usually" is measured in consequences. Nobody forces that line. It gets crossed by drift, one easy prompt at a time, because writing the description was less work than building the system that could be trusted.

The verdict

Can systems around nondeterministic models hit 99.999% reliability? Yes: the how is already known, and in verifiable domains the work is well underway. But the reliability lives in the scaffolding, not in a perfect model. It comes from cheap verification and error correction, the same way it always has, everywhere else in engineering.

Will there ever be a model (or an agent) so capable it needs none of that scaffolding? I'd bet against it. Not in ten years, not in twenty. Because the barrier was never capability. It's that probability isn't determinism, that unverifiable judgment has no oracle to check against, and that independent errors multiply. Those are structural facts, and a smarter model doesn't repeal them.

So AI will eat software, a lot of it, faster than most people expect. But "SaaS is dead" answers the easy question and skips the hard one. Stop asking how good the machine will get; it will get very good. Start asking the question underneath: what can be cheaply verified, and what exactly gets traded away when it can't? The only thing worth deciding, deliberately and not by drift, is which guarantees get handed over for the power, and which ones stay.

Note on the compounding chart: it assumes independent per-step errors and is illustrative of the mechanism, not a measured failure rate. Real-world step errors are correlated, and error-correcting systems catch a fraction of them, which is precisely the point.