Two sets, one leaderboard
Open questions · sealed answers
A benchmark is only worth something if the answers aren't already on the internet — the moment a question and its ground truth go public, the next model trains on them. So the bench runs in two halves: in one the questions are open and anyone can enter; in the other the questions stay sealed and we run them for you. In both, the answers stay with us.
The public set
Open to anyone with an account. You download the questions — every prompt, every unit — run them through your own model, and upload the answers. The ground truth isn't in the download: it stays on the server, so the score is computed for you, not by you.
You explore the questions on the Questions page and watch the rankings on the Leaderboard. Because the questions are in the open, treat a public score as a practice court: a real measure of estimation, but one whose questions a model could in principle have seen.
The held-out set
Never enters the public repository — not the prompts, not the ground truth. It lives in a private store and is versioned as named cohorts (holdout-v1, holdout-v2, …) so it can rotate as models improve.
On the site it appears in exactly one place: a Holdout filter on the Leaderboard. You'll see how each model ranks on the held-out questions, but the questions themselves — and their answers — are withheld. There are no held-out entries on the Questions or Models pages, because there's nothing we can show there without leaking them.
That's the point: a held-out score is the contamination-free number. No one — no lab, no model — has seen these questions, so the ranking reflects estimation skill, not recall.
How to compete
Two tracks, matching the two sets. Pick the one that fits what you want to claim.
Download the questions, run your model, upload to qualify
The public track is open to anyone with an account. You download the questions, run them through your own model wherever it lives, and upload the answers — the bench scores them against ground truth it keeps to itself and places qualifying runs on the leaderboard.
The questions are in the open; their answers are not. That's the trade for a public board that hasn't simply been trained on. For the cohort where even the questions stay sealed, there's the held-out track.
Create an account
Entering the board takes a free account — it's how a run gets attributed to you, and how uploads stay rate-limited and accountable. You can browse the Questions and Leaderboard without one; sign in when you're ready to compete.
Download the questions
Pull the current public set from the app as a single file: each question's prompt, its unit, and the output format — but not its ground truth. The answers stay on the server, which is what lets the board mean something even though the questions are in the open.
Every entrant gets the same set, so two models are always compared on identical questions.
// questions.json — pulled from the app
[
{ "id": "kettle",
"prompt": "How long does a kettle take to boil a pot of tea?",
"unit": "minutes",
"format": "squiggle" }
// …the full public set. note: no answers.
]Run your own model
Run the questions through whatever model you want to enter — any provider, any harness, your own code. Nothing about your setup touches our servers at this stage.
Each answer is a single fenced squiggle program: a chain of 90% confidence intervals that evaluates to a distribution, ending in a {p5, p50, p95} record. That's the one format the scorer reads.
```squiggle
population = to(5e6, 12e6) // population of Chicago
piano_fraction = to(0.005, 0.05) // households with a piano
minutes_per_tuning = to(30, 180) // per tuning, incl. travel
working_minutes = 48 * 5 * 6 * 60 // weeks * days * hours * 60
tuners = population * piano_fraction * minutes_per_tuning / working_minutes
{ p5: quantile(tuners, 0.05),
p50: quantile(tuners, 0.50),
p95: quantile(tuners, 0.95) }
```Upload to qualify
Upload your answers and the bench scores each one against the sealed ground truth, by the same rule set out in the Methodology: Mean Cramér — shape-sensitive, log-space, lower is better. Runs that clear the bar are attributed to your account and placed on the public Leaderboard.
Because the ground truth never leaves the server, you can't tune an answer toward a number you can't see — and you can't re-derive your own score either. The board is the result, not your local run.
That's the trade the public track makes: open questions and open enrollment, in exchange for a score only the server can compute. The held-out track goes one step further and seals the questions themselves — you never see them, and the maintainers run your model for you.
Want to strengthen the bench instead of just competing on it? Contribute a question. Good submissions are auto-graded and reviewed, and the best of them seed future held-out cohorts — which is also why a contributed question may never appear in the public set.