← The Book Challenge-AI4M2026 / track1 / case study

FormalRx Case Study Analysis

Eight hand-diagnosed misalignments between informal mathematical statements and their Lean formalizations — plus one aligned control — drawn from a stratified sample of FormalRx_Test.jsonl.

Sample
case_study_samples.jsonl · 20 stratified from 7,030 · seed=42
Strata
8 short (<120c) · 8 medium · 4 long (≥250c)
Analyzed
8 representative · 7 error patterns + 1 aligned control
Disclaimer on category names. The FormalRx paper is not yet on arXiv, so the official 28-class SCI taxonomy strings are unknown. Category labels below are plausible guesses based on common autoformalization error types. When the paper drops, names must be replaced with the official taxonomy strings — exact-match is required for scoring.

Difficulty map across the 8 cases

Easy
Cases 1 · 2 · 4
Single-point substitution
90%+
Medium
Cases 3 · 6
Numeric / typeclass omission
60–80% CoT
Hard
Cases 5 · 7 · 8
Quantifier inversion · coercion · paraphrase
30–70%
INFORMAL: Given a finite set s, the sum of f i + g i over i in s equals
          the sum of f i over i in s plus the sum of g i over i in s.

FORMAL:   theorem finprod_mem_mul_distrib :
            ∏ i in s, f i * g i = (∏ i in s, f i) * ∏ i in s, g i
          -- aggregator sum→prod, inner op +→*
verdict
Misaligned
category
Operator Substitution (Semantic)
segment
the whole RHS expression with prod/*
corrected
theorem ... : ∑ i in s, (f i + g i) = (∑ i in s, f i) + ∑ i in s, g i := sorry
Insight. Structure intact but both the aggregator (sum→product) and the inner operator (+→*) are swapped. A frontier LLM should one-shot this. Typical "easy" sample.
INFORMAL: The character of isomorphic representations is the same.

FORMAL:   theorem char_iso {V W : FDRep k G} (i : V ≅ W) :
            ∃ g : G, V.character g = W.character g
verdict
Misaligned
category
Quantifier Weakening (Semantic)
segment
∃ g : G,
corrected
theorem char_iso ... : ∀ g : G, V.character g = W.character g := sorry
Insight. Textbook ∀→∃ weakening. LLMs catch this easily; the risk is naming the category wrong (Weakening vs Strengthening direction).
INFORMAL: ...the martingale defined by E[g | ℱ n] converges in L¹ to g.

FORMAL:   Tendsto (fun n => eLpNorm (μ[g|ℱ n] - g) 2 μ) atTop (𝓝 0)
                                                       ^ should be 1
verdict
Misaligned
category
Numeric Constant Substitution / Implementation Index Error
segment
eLpNorm (μ[g|ℱ n] - g) 2 μ
corrected
Tendsto (fun n => eLpNorm (μ[g|ℱ n] - g) 1 μ) atTop (𝓝 0)
Insight. One-character error (2 instead of 1). Hardest for human reviewers, medium for LLMs — solvable if the prompt explicitly asks for token-level constant checking.
INFORMAL: Let s be a discrete subset of a topological space. Then the
          preimage of s by a continuous INJECTIVE map is also discrete.

FORMAL:   theorem DiscreteTopology.preimage_of_continuous_injective ...
            (hc : Continuous f) (hsurj : Function.SURJECTIVE f) : ...
verdict
Misaligned
category
Predicate Substitution (Semantic)
segment
Function.Surjective f
corrected
replace (hsurj : Function.Surjective f) with (hinj : Function.Injective f)
Insight. The theorem name itself says ..._injective, but the hypothesis is Surjective — an internal inconsistency. Strong LLMs should weigh the lemma name as a cross-check signal during diagnosis.
INFORMAL: ...Is it NECESSARY that sum_digits(ab) = sum_digits(a) * sum_digits(b)?
          Prove that the answer is: No

FORMAL:   theorem omni_theorem_4388 {a b : ℕ} (ha : isPedestrian a)
          (hb : isPedestrian b) (hc : isPedestrian (a * b)) :
            (digits 10 (a*b)).sum ≠ (digits 10 a).sum * (digits 10 b).sum
verdict
Misaligned
category
Conclusion Quantifier Inversion (Semantic, severe)
segment
the universal conclusion (asserting ≠ for all pedestrian a, b)
corrected
∃ a b : ℕ, isPedestrian a ∧ isPedestrian b ∧ isPedestrian (a*b) ∧ (digits 10 (a*b)).sum ≠ (digits 10 a).sum * (digits 10 b).sum
Insight. "Answer is No" to "is it necessary?" means a counterexample exists, not "the equation fails for all inputs". The formal as written is actually a false proposition (take a=b=1: both sides equal 1). Pattern — "Find / Prove that the answer is X" — is a high-frequency trap throughout the dataset.
INFORMAL: If R is a ring WITH A CONTINUOUS MULTIPLICATION, then negation
          is continuous ... since it is just multiplication with -1.

FORMAL:   theorem IsTopologicalSemiring.continuousNeg_of_mul
            [TopologicalSpace R] [NonAssocRing R] /* missing [ContinuousMul R] */ : ContinuousNeg R
verdict
Misaligned
category
Missing Premise / Hypothesis Omission (Constraint)
segment
the [TopologicalSpace R] [NonAssocRing R] instance bracket block
corrected
add [ContinuousMul R] to the instance bracket
Insight. Typeclass omissions are Lean-specific. Informal explicitly says "with continuous multiplication" but formal drops [ContinuousMul R]. LLMs often miss bracket-style instance args because they look syntactically different from (h : ...) hypotheses.
INFORMAL: ...0 < a - b < sqrt(5 + 4·sqrt(4m + 1))    (real-valued sqrt)

FORMAL:   Set.ncard {x : ℕ × ℕ | ... x.1 - x.2 < sqrt (5 + 4 * sqrt (4*m + 1))}
                                                     ^^^^ resolves to Nat.sqrt (integer floor)
verdict
Misaligned
category
Implicit Coercion / Type-driven Operator Selection (Implementation)
segment
sqrt (5 + 4 * sqrt (4 * m + 1))
corrected
coerce both sides to ℝ and use Real.sqrt. Also: x.1 - x.2 uses truncating Nat subtraction; the comparison should be over integers.
Insight. Argument types make Lean elaborate sqrt as Nat.sqrt (integer floor) rather than Real.sqrt. The values differ enormously (e.g. Nat.sqrt 12 = 3 vs Real.sqrt 12 ≈ 3.46). You cannot see this from informal text alone — you must mentally elaborate the formal statement. This is where strong LLMs commonly fail.
INFORMAL: The Jordan decomposition of a signed measure is unique.

FORMAL:   theorem toSignedMeasure_injective :
            Injective <| @JordanDecomposition.toSignedMeasure α _
verdict
Aligned
category
null
segment
null
corrected
null
Insight. "X is unique" ⇔ "the map X → ... is injective" is a Mathlib idiom. LLMs commonly misclassify these as misaligned because the formal lacks the literal word "unique". The 1,791 aligned samples (25.5% of the test set) likely contain many such paraphrase-style restatements — this is the dominant Verdict F1 failure mode.

Practical implications for prompt design

01

Always include the header field

Case 7 cannot be diagnosed without knowing the inferred types from the imports/variables block — informal + formal alone is not enough.

02

Add a paraphrase-aware aligned check

Prompt the model to enumerate Mathlib idioms ("X is unique" ⇔ "map injective", "X is the same" ⇔ "∀ point equal"). Without this, aligned samples are systematically rejected.

03

Handle answer-type questions separately

"Prove that the answer is …" / "Find the value …" carry an implicit existential quantifier (counterexample for "No", witness for "Yes"). Naive ∀ translations are wrong.

04

Bidirectional self-check at verdict stage

Have the model argue both aligned and misaligned, then pick the better-supported side. Cuts false positives on paraphrased aligned cases.

05

Constant- and predicate-level token diff

Many errors (Cases 3, 4) are one-token substitutions. A pass that enumerates predicates, constants, quantifiers, operators and verifies each pairwise is cheap and high-yield.

Open questions for development