Start with a word that doesn't get enough attention in the AI safety conversation: drift. The failures we argue about — jailbreaks, hallucinations, the model that starts insisting it's something it isn't — get treated as separate problems, each with its own benchmark and its own patch. But they’re surface readings of the same underlying motion. A model's internal, mathematical representation of what it's doing wanders, step by step, and what we catalog as distinct failure modes is largely just where that wandering breaks the surface. We do name the drift — we've just learned to market it.
The same wandering that produces a hallucination gets sold, in a friendlier register, as adaptability, creativity, a model that meets you where you are. Call it a feature and you never have to fix the bug.
Think of a ship navigating by dead reckoning. Each position is estimated from the last one plus a guess at speed and heading. Every estimate carries a small error, and because each new fix builds on the previous one, those errors compound. An hour later the ship can be miles from where its instruments insist it is, and there was never a wrong turn — only accumulation. A large language model holds an internal sense of context the same way, updated token by token, and it can drift just as silently.
The field studies what happens at the end of that process. We catalog the bad outputs, benchmark them, and bolt on a fix. I argue this is the wrong altitude — that stability is a property of the structure underneath the behavior, not of the behavior itself. What I didn’t expect was to find that argument being made, independently and from an entirely different direction, by people who study silicon.
The machine is not the math
The property we work hard to get from these systems and assume we have, hasn’t been captured: determinism. Feed a model the same input, get the same output. Infrastructure exists to defend that property, because we lean on it to debug what went wrong and to certify that a system will behave. The hardware underneath breaks determinism outright, and the reason is almost comically mundane.
Floating-point addition isn’t associative. In plain arithmetic, (a + b) + c and a + (b + c) are equal. In a computer, where every number is rounded to a fixed number of digits, the order you add them in changes the last few digits of the answer. It’s the same reason a group splitting a restaurant bill can get slightly different totals depending on who throws in their share first and where the rounding lands. The total is “the same” — until you count the pennies.
A GPU computing a model is summing millions of these numbers in parallel, adding them in whatever order each thread happens to finish. So the same model, on the same input, doesn’t produce a bit-identical answer twice. A recent paper by Tadisetty Sai Yashwanth, On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication, set out to measure exactly how much this should destabilize a model’s output — and found something more interesting than the answer it expected.
A photo finish that never changes
When a model picks its next word, it scores every candidate and takes the highest. The gap between the top score and the runner-up is the logit margin — think of it as the gap between first and second place in a race. If hardware noise jiggles those scores, you’d expect that in a tight race, the noise would occasionally flip the winner.
The math predicted it should, a bit over one percent of the time. Across ten thousand trials, the actual number of flips was zero. None. The noise was real and measurable, but it never changed the outcome — because it wasn’t random static hitting each score independently. Nearly half of it moved the scores together, in a coordinated shift. Picture the photo finish where every runner’s recorded time is nudged by the same gust of wind: the numbers all move, the finishing order doesn’t. The variation was wide; the thing that mattered stayed invariant.
That’s noteworthy, because it’s the same lesson relativity teaches: variability isn’t instability. A system can churn underneath and stay perfectly coherent where it counts — if there’s structure holding the coherence in place. Here, the structure was an accident of how the hardware sums its numbers. Nobody designed it. It just happened to protect the result.
Where the floor gives way
That free protection has a limit, and the second paper finds it. To see where, you need one fact about how a computer stores a number. It keeps the digits separately from the scale — the mantissa holds the digits, the exponent says where the decimal point sits. Corrupt a digit and the number is slightly off. Move the decimal point and you’ve multiplied by a thousand.
In Exploring Silent Data Corruption as a Reliability Challenge in LLM Training, Altenbernd, Wiesner, and Kao deliberately flipped individual bits inside a model’s calculations — the kind of fault a cosmic ray or a degrading chip can cause for real — and watched what survives. The result is sharp. Flips in the mantissa wash out, absorbed like the harmless noise from the first paper. Flips in the exponent are a cliff: they trigger runaway values, corrupted updates, and lasting divergence. The same structure that protects the model has an edge, and on the far side of that edge a single bit ends the coherence.
So there’s an invariant living in the number format itself — a basin where perturbations stay coherent and a boundary where they turn catastrophic. And here’s the part that should sound familiar to anyone who has watched the governance conversation: the field’s response is to detect the corruption after it happens and recompute the step. External monitoring, catching failures downstream. The same instinct, one layer lower.
The wall both papers hit
There’s a brilliant idea hiding in the second paper, and it’s significant. Instead of watching for damage after the fact, you can bake a mathematical invariant directly into the computation — a quantity that must hold true if the math ran correctly, checked continuously as it runs. It’s an elegant approach and also runs into a wall, stated plainly in the paper: it works for the linear parts of a network, and nobody knows how to define such an invariant across the non-linear parts.
That wall is the whole problem. A neural network is layers of linear operations interleaved with non-linear ones, and the non-linear steps are where a clean, coherent shift can curdle into something destructive. The first paper measured its beautiful zero-flip result on a single linear operation and stopped at the threshold of the non-linear stack. The second admitted it can’t carry its invariant through that stack. Two papers, two directions, same dead end: we do not know how to preserve a stability property through non-linear composition.
The hypothesis
I have spent the last year building a system, ATLAS (AI Tensor Lattice Active Stabilization), whose entire job is to watch a model’s representation drift and correct it in real time — operating on the geometry as it forms, across the layers, non-linearities included, in training and in inference alike. It was built to address drift as such, wherever it arises: the long slow wander of a lengthy conversation, the pull of conflicting objectives, the divergence that creeps into a training run.
It wasn’t built with cosmic rays in mind.
But the substrate literature shows that hardware noise is also drift — the same wandering of the same representation, arriving from a different cause and at a different layer of the machine. A drift controller that operates on the representation doesn’t need to know what pushed it; it watches the thing that’s moving, not the hand that moved it. Which raises a hypothesis I want to state precisely, and no more strongly than the evidence allows: if ATLAS stabilizes representational drift, it should absorb substrate-induced drift as a side effect of what it already does — including across the non-linearities where the existing invariant methods give out.
I want to be exact about what that sentence is. It’s a hypothesis, not a result. ATLAS has been built and validated against drift as it ordinarily arises; it hasn’t been tested against bit-level hardware faults, and I’m not claiming otherwise. The claim is falsifiable and the test is concrete: take a model under live ATLAS control, inject mantissa- and exponent-region faults the way the second paper does, and measure whether it holds its representational geometry where an unprotected model drifts or cascades.
I expect the smooth case — mantissa-region drift — to fall directly within what ATLAS already handles. The honest open edge is the exponent cliff. ATLAS corrects drift before it accumulates; an exponent flip is not a gradual wander but a discrete, order-of-magnitude jump that is catastrophic the instant it lands. Whether a real-time controller can catch a perturbation that’s already over the cliff at the moment of arrival, rather than one that creeps toward it, is exactly the question the experiment would answer. I don’t know yet but that’s the point of running it.
What I do know is that two groups working from the silicon up arrived at the precise problem I have been working from the representation down — preserving coherence through a non-linear stack — and named the same wall. When inquiries from opposite ends of a system converge on the same obstacle, it’s usually because the obstacle is real and central. Good hypotheses begin exactly there: not as conclusions, but as a convergence worth putting to the test.
- Taggar
- AI Safety Trustworthy AI AI Governance
- Logga in för att kommentera