Agent benchmark — question bank v0
We built two oracles: one that respects as_of, and one that ignores it entirely and answers with figures that were not public on the day. Scored on accuracy, both are 100% — the one that ignores the time gate is the best model on a conventional scoreboard. The only thing that separates them is leak rate: 0% against 100%. That is what this bank exists to measure.
Every figure on this page is transcribed from the 2026-08-21 run, whose report is generated by report_benchmark_bank.py reading the run artifacts rather than anyone typing them. Nothing here is rounded or restated.
Four checks, run against real data. All four passed, and the fourth is the one worth reading twice: accuracy alone rates a model that answers with information it could not have had as highly as one that plays fair.
These are oracles, built to prove the gate works. Neither is an agent, and this is not a leaderboard — there is nothing here to come first.
| Control | Accuracy | Leak rate |
|---|---|---|
| oracle The control that respects as_of. | 100% | 0% |
| leaky-oracle Answered 40 questions with figures that were not public on the day — exactly the behaviour accuracy alone rewards. | 100% | 100% |
148 questions: 40 embargoed (27%) and 108 answerable, with 37 of each of the four question types. The embargoed ones are the point — they are the questions whose answer was not public on the date being asked about.
Of the 148 questions in the bank, 20 were sampled and their proof references resolved: 6 verified, 14 not_in_snapshot, 0 connection failures, 0 key errors.
The 14 not_in_snapshot are not key errors: the row keys came back unchanged and the snapshot_version was older than the row's date, meaning those rows are newer than the last published checkpoint. That is a checkpoint cadence gap, not a broken proof pipeline. This section will be re-run once checkpoint period stamping lands.
How a run is defined — what binds an as_of to a question, how leak rate is computed, and what a run must emit to be citable — is written out in the benchmark specification.
The time gate this benchmark measures is the same as_of parameter documented in the quant research runbook.
Not investment advice.
