candor · measured · pre-registered · blind-judged
Does candor actually help an agent
— and by how much?
candor maps which functions reach the network, filesystem, a database or a subprocess — traced through every call — and gates the edit that crosses a boundary you’ve declared. Two questions decide whether that’s worth adopting: does it make an agent’s work more complete, and does it make it cheaper? We ran both as controlled A/Bs across four model tiers, from Haiku 4.5 up to Fable 5.
Blast radius surfaced
0–6→14–15
of 16 affected functions an edit’s effect reaches. Unaided vs. with candor — and it barely moves with the model tier.
Tokens for the same answer
1.37× fewer
median, and consistent at every tier — candor’s query returns the transitive set the source agent re-derives by hand.
// task: a Rust edit whose Net effect propagates to 16 functions across 9 files and 5 call-graph layers
01 Completeness
An agent won’t volunteer five layers of transitive reach — any model. Ask it to make an edit and describe the consequences, and on its own it names the local change and stops: 0–6 of the 16 functions the effect propagates to, and the frontier doesn’t close the gap. The bottleneck isn’t capability; it’s that no model volunteers tedious enumeration. Hand it candor’s report and it states the complete chain — ≈14–15 of 16 — essentially flat from the weakest model to the strongest.
→The lift is largest for the weakest model (Haiku: 0 → 14). candor doesn’t make a model smarter; it does the enumeration the model won’t — so a cheap model reviews an edit as thoroughly as an expensive one.
02 Cost
Now ask directly: list every function that transitively gains this effect. Pointed at the question, even the unaided agent traces the whole graph and gets all 16 — but it reads files and reasons its way there. With candor it runs one query. Both arms hit 16/16; candor gets there for ~24–42% fewer tokens and 2–3 tool calls instead of 3–13.
→Two complementary results: candor makes the complete answer volunteered when the agent edits, and cheaper when it queries.
03 How it was measured
The point of candor is not overclaiming, so the eval holds itself to the same bar. Every design decision — models, trial count, scoring rule, falsification conditions — was committed to the repo before a single trial ran.
160
agents
40 engineers × 3 blind judges each, one shot per trial
4
model tiers
Opus 4.8 · Sonnet 5 · Haiku 4.5 · Fable 5, same task
40 / 40
objectively verified
every edit machine-checked to actually introduce the effect — no cherry-picking
3×
blind judges
shuffled opaque IDs, scoring facts not opinions
04 What these numbers don’t say
Stated plainly
- “Completeness” means the agent conveys the whole affected chain — it names the functions, or states the effect reaches every caller up to
main. It does not mean it types out all 16 by name; that list lives in candor’s query output, not the prose. - The token figures are the clean metric; wall-clock is directional. The trials ran concurrently, so raw timing is contention-noisy — candor is faster at every tier (a prior serial run measured 1.8×), but tokens and tool-count are what we stand behind here.
- This is the value where it’s real: large graph questions over a real codebase. On a small, local question a capable model needs no help — and candor won’t invent a benefit that isn’t there.
05 See it on your code
The gate that produced these results is the same one that fails a pull request on the exact line a boundary is crossed — across the JVM, Rust, TypeScript/JavaScript and Swift, from one spec and one policy file. Every engine is open source, and the full pre-registrations, raw per-trial telemetry and results are in the repo.
Work with the team behind candor
candor is built and run by Polymorphism — a small senior team that designs, builds and runs large systems for enterprise and government. The judgement we put into AI-code safety here is what we’d bring to your platform.