Two sets, one leaderboard
Open questions · sealed answers
A benchmark is only worth something if the answers aren't already on the internet — the moment a question and its ground truth go public, the next model trains on them. So the bench runs in two halves: in one the questions are open and anyone can enter; in the other the questions stay sealed and we run them for you. In both, the answers stay with us.
The public set
Open to anyone with an account. You download the questions — every prompt, every unit — run them through your own model, and upload the answers. The ground truth isn't in the download: it stays on the server, so the score is computed for you, not by you.
You explore the questions on the Questions page and watch the rankings on the Leaderboard. Because the questions are in the open, treat a public score as a practice court: a real measure of estimation, but one whose questions a model could in principle have seen.
The held-out set
Never enters the public repository — not the prompts, not the ground truth. It lives in a private store and is versioned as named cohorts (holdout-v1, holdout-v2, …) so it can rotate as models improve.
On the site it appears in exactly one place: a Holdout filter on the Leaderboard. You'll see how each model ranks on the held-out questions, but the questions themselves — and their answers — are withheld. There are no held-out entries on the Questions or Models pages, because there's nothing we can show there without leaking them.
That's the point: a held-out score is the contamination-free number. No one — no lab, no model — has seen these questions, so the ranking reflects estimation skill, not recall.
How to compete
Two tracks, matching the two sets. Pick the one that fits what you want to claim.
Sealed questions, maintainer-run, a number that can't have leaked
The held-out track is the contamination-free half of the bench. You can't self-score a set you can't see — so the questions stay sealed, the maintainers run your model against them, and only the aggregate score is published.
It's the eval-server model used by serious contamination-safe benchmarks: the prompts and ground truth never leave a trusted environment, and no lab or model gets to train on them. For the half where the questions are open and anyone can enter, there's the public track.
The questions are sealed
Held-out questions never enter the public repository — not the prompts, not the ground truth. They live in a private store, versioned as named cohorts (holdout-v1, holdout-v2, …) that rotate as models improve.
They never appear anywhere you can reach: not in the question set you can download, and not on the Questions page. There's nothing we could show there without leaking them.
// the downloadable public set lists public questions only. // held-out questions never appear in it — they live in a private // store as named cohorts, rotated as models improve: // holdout-v1, holdout-v2, …
You hand over a way to call your model
Because you can't see the questions, you can't produce the answers — so instead of run records, you give the maintainers a way to call your model: an Anthropic or OpenAI model id we can reach, or an OpenAI-compatible endpoint.
To enter, reach out to the maintainers to arrange access. Nothing about the cohort comes back in return — not the questions, not the count, not the answers.
# What you provide to enter: provider: openai-compatible # or anthropic / openai endpoint: https://api.your-host.com/v1 model: your-model-name # Open an issue on the repo to arrange access. No cohort details come back.
The maintainers run and score it
Your model is run against the current cohort inside a trusted environment, through the same scorer that grades the public set. Same output contract — a fenced squiggle program per question — and the same headline metric, Mean Cramér, so a held-out score sits on the same axis as a public one.
The prompts and ground truth never leave that environment. Only the scores come out.
# inside the trusted environment, against the sealed cohort: # your model answers each question with a fenced squiggle block, # scored by the same rule as the public set — Mean Cramér. # prompts + ground truth never leave this environment.
Only the aggregate number ships
What enters the committed web bundle is scores and domain tags — never a prompt, a ground truth, or a raw response. A guardrail in the build fails if any held-out content tries to slip into the public payload.
On the site it surfaces in exactly one place: the Holdout filter on the Leaderboard. That ranking is the contamination-free number — no one has seen these questions, so it reflects estimation skill, not recall.
# runs.public.json — the committed web payload:
{ "held_out": true, "model": "your-model",
"score": { "cramer_log": 0.21 } }
# prompt, ground truth, raw response: stripped.
# surfaces under the Holdout filter on the Leaderboard.That's the point of sealing them. A held-out score is one no amount of training on public data can inflate — the price is that you trust the maintainers to run it, rather than running it yourself. The public track is the other half: open questions, open enrollment.
Want to strengthen the bench instead of just competing on it? Contribute a question. Good submissions are auto-graded and reviewed, and the best of them seed future held-out cohorts — which is also why a contributed question may never appear in the public set.