← The Book FormalQualBench / runs / leaderboard
Benchmark Dashboard

FormalQualBench Leaderboard

23 research-level Lean theorems · 11 main runs · solve rates, wall time, and token usage compared side by side.

Auto-generated 2026-06-02T05:05Z · harness/aggregate.py
Highest clean rate on 8-subset
38%3/8
deepseek-v4-pro[1m]
Main runs
11
8 on full 23, 3 on subsets
Problems
23
research-level Lean theorems

01Config map

Runs are grouped by execution path and gateway/tooling choices.

02Leaderboard

Split by run scope so partial-set runs aren't ranked against the full suite. Bars scale within each run's own problem set.
Tooling lean-tools (lean-lsp MCP) vanilla (no MCP)

03Problem × Run matrix

Click a column header to sort by that run's verdicts. Hover any cell for full data.
Cell shows
Filter
✓ clean (kernel-accepted) s use_sorry Mstatement_mismatch Ddep_mismatch Aillegal_axiom F/T build_fail / timeout · not in run / theorem missing

04Per-run breakdown

Solved problems list, total wall time, dominant model.

05Cost profile

Tokens summed across all sessions per run; only tokens on solved sessions vs total burn shown side by side.

Total tokens — all sessions

Tokens on solved sessions only