← The Book
FormalQualBench / runs / leaderboard
Benchmark Dashboard
FormalQualBench Leaderboard
23 research-level Lean theorems · 11 main runs · solve rates, wall time, and token usage compared side by side.
Auto-generated 2026-06-02T05:05Z · harness/aggregate.py
Best clean rate
35%8/23
claude-opus-4-8 · claude-code-opus-4.8-lean-tools
Highest clean rate on 8-subset
38%3/8
deepseek-v4-pro[1m]
Main runs
11
8 on full 23, 3 on subsets
Problems
23
research-level Lean theorems
01Config map
Runs are grouped by execution path and gateway/tooling choices.
02Leaderboard
Split by run scope so partial-set runs aren't ranked against the full suite. Bars scale within each run's own problem set.
Tooling
lean-tools (lean-lsp MCP)
vanilla (no MCP)
03Problem × Run matrix
Click a column header to sort by that run's verdicts. Hover any cell for full data.
04Per-run breakdown
Solved problems list, total wall time, dominant model.
05Cost profile
Tokens summed across all sessions per run; only tokens on solved sessions vs total burn shown side by side.
Total tokens — all sessions
Tokens on solved sessions only