A model wrote the word null ninety times. We wrote it down as ninety lies.
That single mistake is the whole project in miniature. We had asked Claude Haiku to extract arguments from a user's prompt through a tool contract, and the schema typed every field as string. So when the prompt didn't mention a value, the model could not emit an actual null — the contract had no room for one. It did the only thing available: it wrote the four characters n-u-l-l into a string field. Ninety times across 131 calls. Eight more times it wrote an empty string. Once, memorably, <UNKNOWN>.
Our scorer counted every one of those as a fabricated value, and we concluded that structured output makes models less trustworthy. It doesn't. We had taken away the ability to abstain, and then graded the model on abstaining.
Change one thing — retype the field as anyOf so that "nothing" is expressible — and the same model, on the same prompt, with not one word of instruction changed, writes 179 real nulls. Restraint jumps 69.7 percentage points. 129 cases recovered, none lost.
Why we care what a model doesn't say
When you type "email bob about the launch" into Veya, something has to decide which action fires and what its arguments are. The action has a schema: recipient, subject, body, maybe a filter or a date. Some of those you named. Most you didn't.
A model handed that form fills it in. Not maliciously — it is doing exactly what it was trained to do, which is produce a plausible continuation. But plausible and stated are different things, and the gap between them is where an assistant stops being useful. A volunteered since: "today" silently filters mail you wanted to see. A volunteered limit: 10 quietly truncates a list. A volunteered recipient sends an email to someone you never named.
None of this looks like a hallucination. There is no confident nonsense, no invented citation. The output is well-formed, schema-valid, and wrong in a way nothing surfaces. That is precisely why nobody counts it.
So we started counting it. We call the two halves recall — fill what the prompt licenses — and restraint — leave alone what it doesn't. The field has a better name for the second one, abstention, which we found out embarrassingly late and will come back to.
First, a ruler
The plan was to fine-tune a local 3B model to be restrained, so the work could happen on your machine instead of in someone's data centre. Before spending a night of GPU time we checked what our existing benchmark could actually prove.
It measured restraint over 50 slots. At that sample size the 95% confidence interval is ±8.3 points — wide enough that a model measuring 90% could truly be 82%, and one measuring 85% could truly be 93%. Our bench could not distinguish the bar from a clear failure to reach it. Every restraint number we had ever published, including the one we were treating as the target, was uncertifiable.
So we rebuilt it: 378 labelled slots across two splits, every prompt traceable to something a real user or a real test actually wrote, with a machine check that fails the build if a prompt's claimed provenance doesn't match what the harvester recorded. 29 slots where the correct answer was genuinely arguable were excluded rather than guessed, and the exclusion count published.
That took the interval to ±4.2 points. It also produced an uncomfortable fact we've kept in every document since: to certify a claim of ≥90% at that sample size you must measure 94.3%. A model that truly sits at 92% would need 885 slots to certify, and one at 90.5% would need 14,017. No realistic corpus certifies anything between 90 and 94 — including the frontier model we were trying to match.
Three dead ends, and the arithmetic that closed one before it cost anything
Don't show it the optional fields. The obvious fix: only ask the model for fields the action actually requires. Before running it we computed the ceiling — 50.4% of the licensed values in our corpus sit on optional fields, because filters and formatting options are what these actions are. So the intervention caps recall at 49.6% by construction. Measured, it cost 45.3 points of recall to buy 47 points of restraint, and it achieved that by not asking the question: 70 of 131 cases never reached the model at all.
That one is worth dwelling on, because the ceiling calculation took about a minute and closed a direction that would otherwise have eaten a day.
And a finding that came free with it: a hidden field is not a forbidden field. Measured recall came in above the computed ceiling and restraint below the computed floor — one behaviour, both directions. The models were answering about fields that were never in the payload at all. The 1.5B emitted body fifteen times and subject six, having simply learned the shape of an email. This refutes every "just don't show it the field" proposal, not only ours.
Fine-tuning. We built a 250-case corpus with an enforced invariant — every field must appear at least twice licensed and twice declined, so the rule the model learns is necessarily conditional on the text. Twelve arms across two base models. All twelve rejected on recall. The best of them bought +42 points of restraint for six recall slots, and every single one of the twelve paired comparisons recovered exactly zero slots — the losses were entirely one-directional.
Then the dial turned out not to be the dial. A LoRA adapter is applied with a scale factor that the tooling reads from a config file at fuse time. Rewriting that one number gives you a model an arbitrary fraction of the way from base to trained — about 70 seconds of CPU per point, against 20 minutes of GPU per training run. The entire frontier fell out of a checkpoint that already existed, and it was sharply convex: the first half of the adapter delivered 30.8 of the 43 available restraint points for one recall slot. Everything after cost four times as much for a third of the benefit.
The measurements that broke our own findings
This is the part we'd want another team to read.
We published a number that didn't reproduce. We reported that the tuned model improved fabricated dates from 6 to 2. A twin trained on the same recipe, same data, same dose — differing only in batch order — scores 7. Recall and restraint reproduced to the decimal; that column didn't. It turned out to carry no information at all: the base model fails it too, on the same slots.
The noise floor wasn't noise. We had been quoting a ±3.7-point seed-to-seed variance and designing gates around it. The cause was ours: the timestamp we inject into every prompt rendered the minute, so two runs an hour apart were two different system prompts. Pin the clock and three consecutive runs are byte-identical on all 131 outputs. We had been measuring our own instrument and calling it model variance.
And the control that killed our favourite theory. We became convinced that our routing failures were caused by an ambiguous action registry that had accreted duplicates over two years — the same canvas reachable by two names, descriptions that read as synonyms. The evidence looked strong: 89% of the errors landed on an action carrying some defect.
Then we computed the base rate. 94–96% of the cases the models got right also landed on an action carrying a defect. Lift: 1.0. In a 272-action registry essentially everything violates something, so "the error landed on ambiguity" carries no information whatsoever. Four of our headline defect classes turned out to have lift at or below 1.0 — correct answers sat on them slightly more often than errors did.
We then tested the single fix we were most confident in, rather than estimating it: rewrite the description we had identified as the top defect, re-run all 186 cases. Four decisions changed. Precision went down 1.1 points.
The corrected attribution: 11% of errors are registry-attributable, 8% are non-English prompts, and 82% is the model.
A prompt is code, and it rots
One defect was real, and its history is worth telling because it will happen to you.
The system prompt that teaches our extractor carries four worked examples. Two of them demonstrated a field called createNew. In May that was correct and load-bearing: two actions declared it, and only a model could decide whether "new doc" meant create or open.
Later, someone split those actions in two — also correct, and cheaper, since a regex could now decide it. Nobody deleted the examples. For two months every extraction call carried two worked examples demonstrating a field that no action in the registry has.
And it got worse than obsolete. Look at what the example then teaches:
prompt "open github.com", schema {url, createNew}, partial {}
→ {"url":"https://github.com","createNew":false}The user said nothing about new-versus-existing. The example fills the field anyway. Once createNew became unreachable, that line stopped teaching a key and started teaching a habit: given a form, fill the form. We were demonstrating the disease in the prompt while paying GPU time to train it out.
Deleting it bought +7.5 points of strict restraint at bit-identical recall. Free. The second worked example, incidentally, was a held-out test case sitting in the shipped prompt with its answer beside it — a slot reserved for once-only certification, handed to the model on every request.
A prompt is a second reader of your schema, linked to it by nothing but someone's memory. Everything else that reads that registry has a type or a test. The prompt was exempt because it's a string.
Restraint needs somewhere to put "nothing"
Three independent experiments converged on the same shape.
The clock. We inject the current date into every extraction call so relative dates resolve. On prompts containing no date at all, both Haiku and the local 3B fabricated one — the injected date, verbatim, on all five test cases. We rewrote the block from an instruction into a polite reference, matching the deferral rule the system prompt already states. It changed nothing: 5 of 5, both models, identical values. It is not the block's mood that fills the field. It is its presence. A date in the payload makes a date inferable, and the model's deferral rule is conditioned on inferability.
The examples. Our instruction said "if a field can't be inferred, leave it null." All four worked examples omitted the field instead. Functionally identical after filtering — but we ran three arms to find out which channel the model listens to. Making the examples show explicit nulls gained 8.6 points on the local model and 1.6 on Haiku, losing nothing. Making the instruction match the examples instead — the other way to resolve the contradiction — lost 14 points. Denied the word null, the model doesn't omit the field. It fills it, or writes an empty string. It emitted 29 of those, every one on a slot it should have left alone.
And the tool contract, which is where this piece opened.
Three different mechanisms, one lesson: give the model a way to say nothing and it invents less; take it away and it invents more. Restraint is not something you exhort a model into. It is something you leave room for.
What actually worked was not the model
The local 3B fills 102 of 185 forbidden slots. We enumerated every one, and they have exactly two shapes. Constant defaults — since:"today" eighteen times, unreadOnly:true seventeen, limit:10 thirteen, always the same value regardless of the prompt. And prompt echo — free text restating what the user typed.
The model isn't reasoning about those fields. It's emitting learned defaults. And every one of those values is type-valid and vocabulary-valid, which is why the guards we already shipped caught none of them: they were built to reject malformed input, and the model never emits malformed input.
So we wrote rules that check something else — not whether a value is well-formed, but whether the prompt licenses it. You cannot extract a number the user never said. You cannot resolve a date from a prompt containing no date. You cannot choose an enum member whose word does not appear.
That took harm-adjusted restraint from 44.9% to 79.5% at bit-identical recall — not "statistically unchanged", zero slots moved — and the zero held across ten different model arms including Haiku. No training, no new weights, one reversible flag.
The honest framing matters here, so we'll state it the way our own review insisted: a guard raises the number that reaches the user and leaves the model's own judgement untouched. It is not "the model got better." It is "less reaches the user, deterministically, at no cost to recall." Our best trained checkpoint reaches 93.5% — but 13 points of that are borrowed from guards, where the frontier model borrows nothing. Those are not the same achievement.
And the guards have a real weakness, which our adversarial review found and we'd rather publish than bury: the rules are English word lists. "Show me twenty-five emails" dropped limit: 25, because the lexicon had no compound number words. That's fixed. "Anything I haven't gotten to" still silently drops the unread filter, and Spanish still loses its dates.
The gap was never where we thought
The premise of the whole project was that a local model is worse at this than a frontier model, and training would close the distance. So we finally measured the frontier model on the same 185-slot bench — something nobody had done, because its published numbers came from the 50-slot bench we'd already thrown out.
On recall, a paired test across the same slots cannot separate them: 96.6% versus 97.4%, two discordant slots against three, p = 1.000. On restraint, Haiku wins by 49.7 points and takes 92 slots while losing zero.
We had been treating this as a two-axis problem and defending a wall that wasn't there. The local model's recall is already frontier-grade. The entire gap is abstention — one axis, and the one where deterministic rules turn out to do most of the work.
What we'd tell someone starting today
Build the ruler first, and compute what it can prove. Not "is the number good" but "can this sample size distinguish the bar from failure." Ours couldn't, and we'd have spent a week optimising against noise.
Put a confidence interval next to every percentage, and make the tool refuse to certify what the sample can't carry. Ours does now, which is why it declines to certify results we'd have been happy to publish.
Compute the base rate before believing a cause. The most persuasive finding we had — a messy registry explaining our failures — evaporated the moment we asked how often the successes looked the same.
Test the fix rather than estimating it. Ours made things worse, on the one we were most sure of.
Treat prompts as code. Version them, test them against the registry they describe, and assume any example you leave behind is teaching something after it stops being true.
And search the literature before you train. We reinvented a named problem — abstention — that has a benchmark, a better training method than the supervised imitation we used, and a released 3B model. We found that out after the GPU time, not before.
What's still open
The non-English failures are a genuine capability gap and no amount of cleanup touches them: our descriptions and examples are entirely English, so retrieval cannot place メール開いて near anything. The 82% of routing errors that are model failure remain model failure. Our best local result still borrows 13 points from deterministic rules. And one rule carries two thirds of that borrowing on a single hand-written regex validated against five examples — a number we only discovered because a reviewer refused to accept the version measured on the arm nobody would ship.
We haven't shipped a restrained local model. What we have is a bench that can tell us when we do, a set of rules that get most of the way there for free, and a much clearer idea of which half of the problem is actually ours.
Veya is an AI workspace that runs on your machine. The routing kernel described here is local-first by design; a previous post covered what happens when the deterministic half gets it wrong.