Leak-resistant benchmark specification
Conventional scoring rewards a correct answer, and a model that ignores the time gate gives the most correct answers. This specification defines leakage as a measured quantity so the two come apart: same bank, same accuracy, and leak rate separates them 0% against 100%.
twmd-leak-benchmark/v0-draft · updated 2026-08-23
A score is citable only alongside these three fields. Recompute the hash over the bank you were given and compare.
Rules a run has to satisfy, not a description of our runner — the point of publishing them is that somebody else's runner can satisfy the same rules.
A question has the form "on date D, what was field F for instrument S". D is the as_of: the system under test may answer only from what was public on D. The specification does not dictate how it obtains data — the TWMD API, its own store, or nothing but model parameters are all permitted — but the answer is judged against D.
An embargoed item is one whose answer was not public on D — monthly revenue before its release date, institutional flow before the close is disclosed. A system that respects the time gate must REFUSE these, not guess. The bank currently holds 40 embargoed items of 148 (27%).
Leak rate is the share of embargoed items answered substantively. The denominator is the embargoed count, not the bank size — using the bank size lets leak rate be diluted by adding questions, which makes it a metric you can improve by writing more items. A refusal is not scored as an error: under this specification, answering "unknown" to something unknowable on the day is the correct behaviour.
A run must include both oracles: one that respects as_of and one built to ignore it. If their leak rates do not separate, the run does not count — that is not a pass for the system under test, it means the measurement itself failed. This is the specification's only check on whether the measurement is working, which is exactly why it belongs in the specification rather than in someone's runner.
If an item's date falls before the training cutoff of the model under test, a correct answer may come from the model's parameters rather than from the data layer — and a low leak rate over such items does not demonstrate that the time gate works. The specification requires each data window to be flagged for overlap with published model cutoffs, and requires a report to state overlapping and non-overlapping scores separately.
Per item: an item identifier, the as_of, whether it is embargoed, the system's response, and a judgement (correct / incorrect / refused / leaked). Per run: accuracy, leak rate over the embargoed denominator, both oracle leak rates, and the question-bank version identifier. A score missing that last field cannot be cited — nobody can tell which bank it was run against. The 4 time-gate checks are the thresholds that decide whether a run is valid.
A specification exists so somebody else can run the thing. Today you cannot, and these are the reasons — each with the path you can call to confirm it for yourself.
Without a version hash, two people saying "I ran v0" have not established that they ran the same items. This specification therefore cannot yet be used to cite a score — only to understand how scoring works. /v2/benchmark/manifest is where it will live; probed 2026-08-23, it answers 404.
A runner exists but is not open, and there is no keyless entry point. An outside team cannot run its own agent against this bank today. /v2/benchmark/run is where it will live; probed 2026-08-23, it answers 404.
The clause above requires windows to be flagged against model cutoffs. The clause is written; the data is not published. Current scores therefore cannot separate a correct answer that came from the data layer from one that came from the model's memory.
The bank composition and the one run measured against it — 148 questions, and what the two controls scored — are on the benchmark results page.
Not investment advice.
