How much of an agent can you own at $0? Two percent, today.
The delegation architecture says a frontier model should propose goals and read anomalies while a world model that costs nothing handles the steps it has already learned. Nobody had measured how many steps the free end could take. So we did, on games the learners had never seen, scored after the fact against what happened.
| method | commits | commit right | delegable |
|---|---|---|---|
| last-effect one-line heuristic | 22.7% | 8.6% | 2.0% |
| nyaya-templates | 38.1% | 2.5% | 0.9% |
| dsl-synthesis-rel | 59.0% | 0.7% | 0.4% |
| last-effect, complete gate | 1.2% | 50.0% | 0.6% |
| nyaya-templates, complete gate | 1.0% | 32.4% | 0.3% |
| dsl-synthesis-rel, complete gate | 0.6% | 21.1% | 0.1% |
| union of the three gates | 2.7% | 38.5% | 1.1% |
| memorise exact recurrence | 1.5% | 90.0% | 1.4% |
| union of the gates and memorise | 4.2% | 56.7% | 2.4% |
An action is delegable when the model committed to a prediction and the whole frame was right. The learners over-commit: synthesis claims 59% of actions and is right on under one percent of them. Gating on per-rule evidence changes nothing. Gating on completeness, committing only when the current theory reproduces the action's last frames exactly, lifts every learner to 20 to 50% right at about one percent coverage. The commit criterion is solved and cheap. The theories are what is incomplete. That is the research.
First measurement. Delegable 0.4% (synthesis), 2.0% (last-effect). Commit precision 0.7% and 8.6%.
Per-rule evidence gates: no change. Whole-effect gate: precision 8.6% → 51.4%, coverage 1.1%.
General completeness gate on every learner: 20 to 50% right, about 1% coverage. Requiring three repeats zeroes the synthesiser.
A model in the loop, the way EvoSkill does it, on the same games with an open 8B model: 107 programs written, none survived the verifier, 0 commits, F1 0.000, 156k tokens; with the evaluator's verdict fed back once, 137 programs, still none, 180k tokens. The $0 heuristic scores 0.181 on the same slice. The loop's value is in the model, and the model is what costs $119 a game.
The gated learners cover different actions. Their union is 1.1% delegable at 38.5% right, against 0.6% for the best single gate. The first point that moved coverage, not precision.
Add exact recurrence. 2.4% delegable at 56.7% right: above the ungated heuristic's 2.0% for the first time, with seven times its precision. 1.4 of the 2.4 points are recall, not generalisation, and are labelled so.
The question moved, so we moved with it
In July 2026 two systems put frontier language models inside a world model's learning loop and largely closed the benchmark. In their own tables, program synthesis without a model in the loop clears zero levels. So capability is answered, and not in our favour. Price is not answered by anyone.
| system | model in loop | result | cost / game | code |
|---|---|---|---|---|
| Tycho | Opus 5 | 100.00 RHAE, 183/183 | ~$119 | Apache-2.0 |
| Tycho | GPT-5.6 Sol | 100.00 RHAE, 183/183 | ~$179 | Apache-2.0 |
| OPINE-World | Opus 4.8 | 20/25 games, 160/183 | not reported | none |
| WorldCoder (run by OPINE) | — | 0 levels | — | public |
| nyaya | none | 2–4 early levels | $0 | MIT |
Every number is the authors' own, sourced in LANDSCAPE.md.
What exists today, not what is promised
The substrate
50
episodes across two corpora, 24,499+ transitions, packed 331× so they ship inside a git repo. A causal-replay protocol: predict, be scored, then observe. Cost reported next to every score.
bench/README.md →The engine
1.00
precision, over 90 examples, for a program the search found on its own. Programs are searched for, not parameters fitted — on CPU, no model called, ever.
The artefact
47
beliefs a scam filter learned from 5,574 real messages in 2 seconds — each a sentence with the evidence that earned it. Switch one off and the verdict changes.
Try the belief ledger →The results, with the intervals we were tempted to leave out
Five methods, one protocol, two corpora. The second corpus was never developed against. Every score carries a 95% bootstrap interval over episodes, and where intervals overlap this page says not shown to differ rather than quoting the larger number.
| method | dev F1 | reached | held-out F1 | reached | ms / step |
|---|---|---|---|---|---|
| copy-forward null | 0.000 | 0/25 | 0.000 | 0/25 | 0.00 |
| memorise lookup table | 0.048 | 3/25 | 0.024 | 0/25 | 0.01 |
| last-effect one-line heuristic | 0.228 .136–.316 | 11/25 | 0.151 .065–.289 | 6/25 | 0.09 |
| nyaya-templates | 0.185 .043–.355 | 6/25 | 0.253 .039–.490 | 5/25 | 3.8 |
| dsl-synthesis-rel | 0.204 .119–.286 | 6/25 | 0.237 .134–.348 | 7/25 | 7.9 |
No method dominates and every interval overlaps. Two things are
still visible. memorise
scores near zero, so the corpus is genuinely novel — the benchmark measures
generalisation, and that is now a measured fact rather than a hope. And the
synthesis engine is the only method never worst on either corpus, with an
interval a third the width of the template learner it replaces.
What we got wrong, in the order we found out
A funder is not buying a result. They are buying what a person does when the evidence goes against them. Here is the record, because it is the strongest thing on this page.
7 September 2026 · a reviewer would have found this in five minutes
“Precision 100%, recall 78%, F1 0.875 on real messages.”
True, and beside the point: naive Bayes gets 0.965 on the identical split in a thirtieth of the time. Published in the README above our own number. The honest claim became readability costs nine points of F1, which nobody had priced before.
7 September 2026 · the novelty check our own document kept deferring
“Nobody has beaten this benchmark.”
False since July. Tycho hit 100.00 RHAE. The proposal was re-scoped in public from a better learner to the price of a world model, and the retraction has its own file.
7 September 2026 · our own benchmark, the week it was built
“The template learner reaches F1 0.253 held out.”
Adding a one-line baseline — replay whatever this action did last time — showed it beats the template learner outright on development data. Nobody made us run that comparison. The registry did, the moment it existed.
8 September 2026 · the central claim, tried twice in one night
“Programs that survive replay become primitives, and the library transfers.”
A library of concrete programs halved performance (0.204 → 0.105): colour literals carry confident wrong vocabulary between games. Making programs abstract made them transferable and too general to be selective (→ 0.087). Both reverted, both published. Abstraction is necessary for transfer and insufficient for selectivity. That is what the funded work is for.
9 September 2026 · the delegation number
“The free end of the curve can carry an agent's routine steps.”
Two percent of them, today, and only with a completeness gate does any learner get its commits right more than half the time. Published the day it was measured.
Run all of it
No API key, no GPU, no dependencies beyond the Python standard library. Every number on this page regenerates from one of these, and GitHub Actions regenerates the held-out table on every push.
The ask, and what you keep if I stop
- One deliverable: the cost–capability curve. Every method with public code — including Tycho at the expensive end — under one protocol, cost measured rather than quoted; and delegation re-measured with completeness-based commitment.
- The largest line is $10,000 of compute spent running our competitor's method, not ours. A curve with an unmeasured expensive end is an opinion.
- The first milestone was delivered before this ask, unpaid: the substrate is public and runnable today.
- If the work stops at any point, everything published to that point is already MIT. There is no version of this where the money buys nothing.
- The remaining programme is $68,000 across three further tranches, an option you hold, released on delivery — including on a failed prediction, because paying only for wins is how funders buy quiet negative results.
Full build-up, tranche schedule and the risks named plainly: ASK.md. Who this is for and who pays, eventually: PITCH.md.
Who is asking
A student in India, working solo. No lab, no cofounder, no credential worth listing — so the argument is the artefact. The runtime, the substrate, the engine, 249 tests, and five retractions of our own claims, all in ten days of a public repository. The reason nobody has measured the cost of these systems is that everyone positioned to is a funded lab with API credits. If your compute is free at the margin, price is not a variable you notice. Mine was never free, so the runtime was instrumented for tokens and milliseconds before I knew that was the interesting part.