Every serious document tool eventually faces the same question: against ECMA-376 — the six-thousand-page spec behind .docx, .xlsx and .pptx — how much do you actually support? Most products answer with a feature page. We tried to answer with a number, and the first thing we learned is that the number you produce by reading your own code is worthless. We know because we tried: our importer's file header had claimed since version one that page breaks were "listed in warnings." No code had ever added one. The claim sat there for months, true-sounding and false.
So the rule of the system we built — we call it the completeness lattice — is one sentence: a claim counts only when a test the scorer watched pass says so, in a child process the scorer spawned, whose output file the scorer deleted first. Every format module carries a ledger: an enumeration of spec features, each marked supported, partial, or unsupported, and every "supported" must name the exact tests that prove it — verbatim titles, checked twice, statically on every build and dynamically on every scoring run. A claim whose test can't be found doesn't lower your score. It fails the build.
The honest number was 41%
Writing seven ledgers — 253 enumerated features across Word, Excel, PowerPoint import and export plus Markdown — took two days with AI builders doing the enumeration against the spec chapters. The BEFORE table, committed to the repo so nobody can quietly move the goalposts:
| module | verified | score |
|---|---|---|
| docx import | 17/33 | 51.5% |
| docx export | 10/37 | 27% |
| xlsx import | 19/48 | 39.6% |
| xlsx export | 9/27 | 33.3% |
| pptx import | 15/42 | 35.7% |
| pptx export | 18/34 | 52.9% |
| markdown | 16/32 | 50% |
| total | 104/253 | 41.1% |
Enumerating honestly is itself a bug-finding technique. On day one, writing the hyperlink ledger entry forced a test that Word's actual output — namespaced r:id attributes — didn't pass; the importer had been reading the unnamespaced form. Seven more defects came straight out of ledger-writing before any new feature work started: a spreadsheet cell holding the text "00701" silently became the number 701 on export; column widths were parsed and never fed to the renderer; the monospace font our design tokens promise was declared in CSS and never loaded, so code blocks paint differently on every machine. None of these were found by a user or a reviewer. They were found by the act of writing down, precisely, what we claimed.
The ratchet, and the first time it caught me
A score you can recompute is a score you can game, so the lattice adds a dual ratchet: against the committed baseline, neither a module's percentage nor its absolute verified count may ever decrease. It runs on every pull request, unconditionally — our CI has a contract test that literally rejects workflow steps carrying an if:, because a step that can decide not to run reads as green when it's deleted. I know that contract works because it rejected my first attempt to add the lattice step conditionally.
The ratchet's first catch was also me. I corrected a ledger — added a feature the enumeration had missed — and the percentage dropped from 53.1 to 51.5, because honesty grew the denominator. The gate went red on my own machine. That's the designed behavior: growing the enumeration is how the number gets more honest, and it goes through a deliberate, reviewed baseline update, not a quiet recompute.
The machine kept catching its makers
This is the section I'd want to read if someone else wrote this post, because it's the evidence that the doctrine isn't decorative. In eight weeks of building the lattice itself, the lattice's own review gates — run by Claude, adversarially, with mutation testing — caught the builders (also Claude, and me) five separate times:
The proxy inside the anti-proxy machinery. Our "battle corpus" runs real files from Apache POI's test suite through the importers. Its spreadsheet test asserted that the imported document's JSON was longer than fifty characters. A mutant importer that imports nothing — empty grid, default sheet — stringifies to ninety-one characters and passed. The test guarding against hollow imports was itself hollow. It now asserts a real populated cell, and a mutation run proves the empty-import mutant dies.
Twelve of fourteen fonts, dead in a comment. The render lane self-hosts pinned fonts so screenshots are deterministic. A malformed comment — --> where CSS wanted */ — swallowed twelve of the fourteen @font-face rules. Every rendering test stayed green, because reference tests compare two renderings that both fell back to the same system font. And the guard that should have noticed? FontFaceSet.check() returns true when no face matches. A guard that passes by matching nothing. It was found only because a mutation test renamed a font file and nothing went red — the survivor was the finding.
The boundary that ten tests missed. Our pixel comparator implements two-sided tolerance bands: a diff below the low bound also fails, forcing stale annotations to tighten. Ten tests covered the bands. A mutant that widened one boundary by exactly one — < lo to < lo − 1 — survived all ten. Band edges need edge-flip mutants, not just band-present mutants.
Word's real encoding, one wrapper deeper. A feature round added text-box recovery, with tests for both the DrawingML and legacy VML encodings. The review gate asked a colder question: what does modern Word actually write? Answer: every text box wrapped in mc:AlternateContent — an element the new code never entered. The exact silent drop the feature existed to fix, one namespace deeper, caught before merge by probing the claim instead of the code.
The build profile nobody had. While embedding a PDF engine we noticed cargo's warning that our release profile — link-time optimization, symbol stripping, abort-on-panic — sits in a non-root workspace manifest, where cargo silently ignores it. Every binary we had ever shipped was built with settings nobody selected, and every published size number described a build that didn't exist.
Pixels are claims too
Parsing a file correctly and painting it correctly are different promises, so the lattice has a render lane: the scorer spawns a real browser, mounts each surface in a stripped harness page, and compares renderings. The default instrument is the reference test — the imported document versus a hand-authored equivalent, painted by the same engine, so fonts and anti-aliasing cancel out — and every reference pair ships with a mismatch companion, a pair that must differ, because a renderer that paints nothing passes every equality test ever written.
Flaky tests get no mercy and no deletion either: a test can be quarantined as intermittent only if a twenty-run detector observed it both pass and fail in one window — a deterministic failure can never be classified away — and the quarantine is capped, auto-draining, and audited nightly. A hand-planted quarantine entry for a healthy test turns the nightly red until it's removed. Allowlists that never drain are how test suites rot; this one drains by construction.
One PDF engine instead of three
The audit also found we had three separate PDF export paths — a JavaScript layout library, a rasterizing screenshot pipeline, and an external converter — each with its own bugs and none covering spreadsheets or slides. They're being replaced by one: the Typst engine, embedded in the app as a Rust crate behind a closed sandbox — no filesystem, no network, no packages; document content enters as pure data, and the review gate spent its round attacking exactly that boundary with path traversals, code-in-content, and hostile SVGs. The PDF lane scores like every other module: claims, named tests, parse-back oracles, a ratchet.
Then we let the agents run
All of that machinery exists for one reason: so a coding agent can be handed a module and told close the gap, without anyone lying awake wondering what it broke. We ran that experiment. One agent, one session, the Word importer: six ledger entries closed in about thirty minutes of wall clock — column widths, internal hyperlinks, text boxes, named styles, tables of contents, page-break warnings — with regressions exactly zero, refereed by the ratchet, the full 15,000-test suite, and mutation runs on every new guard.
The centerpiece wasn't speed. The agent's first column-width implementation froze every fluid table — it had treated Word's internal grid hints as authored sizes. Its own tests were green. What went red was the exporter's round-trip test, in a different module's ledger, which only the lattice's cross-module scoring runs against importer changes. A cross-module lattice caught a cross-module bug that the author's chosen tests structurally could not see. That is the whole thesis in one incident: the agent didn't need to be smarter; the verification needed to be wider than the agent's attention.
The pilot also surfaced where the incentives still point wrong: four of the six landings were honest partials — the feature works, with named losses — and partials currently move the headline number by zero. A system that pays full credit only for "supported" is quietly training its agents to over-claim. We filed that against ourselves; the fix lands after this measurement period so the before/after stays comparable.
Where the number stands
Eight days after the BEFORE table was committed, the program's build phase closed. The honest AFTER, generated by the same scorer on final main:
| lane | BEFORE | AFTER |
|---|---|---|
| engine (7 format modules) | 104/253 · 41.1% | 105/253 · 41.5%, +4 honest partials; docx-import 51.5% → 54.5%, tested claims 17 → 22 |
| PDF export | unmeasurable — three engines, no tests | one engine · 11/24 · 45.8%, every claim parse-back-proven |
| render (doc / sheet / slides) | unmeasurable — zero pixel evidence | 8/9 · 1/1 · 1/1 on macOS, reftests + excused goldens |
Read that honestly: the engine headline moved one point, because the flywheel ran for exactly one pilot session and four of its six landings were partials the metric scores at zero — a bias we filed against ourselves rather than papered over. What actually changed is the denominator of what can be known: two whole lanes went from unmeasurable to scored, twenty-plus real defects surfaced, the test suite grew by two hundred and fifty tests and the lattice's own guard count by two hundred, and the measured cost of an honestly-verified feature fell to about five minutes of agent time — mutation runs, full suite, and the ratchet included. The remaining reachable tail across all seven modules is, at the pilot's measured rate, weeks of fenced agent sessions. Before the lattice, we would have called it years, and we would have been guessing.
The honest framing of "years into days" is not that models type fast. It's that the expensive half of format work was never the writing — it was knowing, with evidence, that a change to clause 17.4.65 didn't break clause 17.4.16. The lattice makes that knowledge cheap and automatic, which means the long tail of a six-thousand-page spec stops being a decade of careful human attrition and becomes something you can hand to agents in parallel, every one of them fenced by the same referee. The machine doesn't trust us either. That's the feature.
The lattice was designed, built, adversarially reviewed, and driven by Claude (Fable) — working as Theron, our principal agent — with the review gates run as separate adversarial sessions seeded to attack the builders' claims. Every number in this post is generated by the scorer, not retyped; the repo's rule is that prose drifts from tables, so tables are code.