How We Benchmarked AI Models for SAT Practice Content

Generating a plausible-looking math question is easy. Generating one that is mathematically correct, appropriately difficult, unambiguous, well-formatted, and safe to put in front of a student is much harder.

We wanted evidence before choosing a production model for our study-content platform. So we built a repeatable benchmark around a simple principle:

A question is not good because a model says it is good. It is good when independent checks show that it satisfies the specification.

What we tested

The benchmark used 12 fixed SAT-style specifications with independently known answers:

• Algebra: linear equations, systems, and line intersections
• Advanced Math: quadratics and exponential equations
• Problem-Solving and Data Analysis: percentages, means, and conditional probability
• Geometry and Trigonometry: circles, similarity, and right-triangle trigonometry

Each specification defined the skill, difficulty, exact numbers, requested quantity, and expected answer. Models had to return a title, question stem, KaTeX expression, four answer choices, distractor rationales, the correct option, a solution, and an estimated solving time.

We also created 16 labeled audit cases. They included clean questions and deliberately injected defects:

• wrong answers and solution mismatches
• invalid LaTeX
• duplicated mathematical content
• duplicated answer choices
• missing visuals
• difficulty mismatches

This let us test both sides of the workflow: creating content and catching content that should not be published.

Evaluation pipeline

Every generated question passed through four layers.

1. Local deterministic checks

These checks did not ask another model for an opinion. They verified:

• the declared answer matched the known answer
• the referenced option contained the correct numeric value
• option IDs and numeric values were unique
• KaTeX compiled without errors

This layer is intentionally boring. A model should not receive credit for a polished explanation if the answer is wrong or the equation cannot render.

2. Defect detection

Each model audited the 16 clean and defective cases. We measured precision, recall, and F1 across the seven defect categories. A false positive matters here: flagging every question as broken would make a review workflow unusable.

3. Blind adjudication

An independent judge reviewed each generated question against the original specification and known answer. The judge scored mathematical correctness, ambiguity, difficulty alignment, distractor quality, solution quality, and SAT alignment, then recommended accept, revise, or reject.

4. Cross-judging

We ran a second independent judge over the same generated questions. The purpose was not to manufacture a perfect score. It was to expose disagreement, especially around subjective dimensions such as difficulty and distractor quality.

OpenAI/Codex results

We ran 12 isolated Codex processes: four model families at low, medium, and high reasoning effort. The full run generated 144 questions, audited 192 defect flags, and used independent adjudication and cross-judging.

All 12 configurations achieved 100% local mathematical and LaTeX cleanliness on this sample. That is encouraging, but it does not mean the models are universally reliable; it means the fixed benchmark did not catch a hard mathematical failure in this run.

The most important finding was that effort level was not a reliable proxy for quality. Higher effort generally increased latency and token use, but did not consistently improve acceptance, difficulty alignment, or distractor quality.

The Codex run used approximately 1.18 million tokens. Because Codex is an agent runtime rather than a raw completion endpoint, that number includes substantial process and reasoning overhead. It is useful for evaluating an agent workflow, but it is not a direct price comparison with a single API completion.

Anthropic results

We ran the same benchmark through the Anthropic Messages API using six currently available models:

The clearest result was Claude Opus 4.7: it combined strong defect detection with the highest agreement between independent judges. Haiku 4.5 was unexpectedly effective as a low-cost audit model. The newer Claude 5 models were mathematically clean, but their acceptance rates were less stable across the two judges in this small sample.

The Anthropic runs consumed approximately $2.25 in API credits. Current model pricing varies substantially by model and output tokens; the official pricing table is available in Anthropic’s documentation.

What we learned

Correctness and quality are different metrics

Every configuration produced mathematically clean questions in this benchmark, yet adjudicators still requested revisions. The remaining issues were usually difficulty calibration, weak distractor rationales, or questions that technically matched the numbers but did not feel sufficiently SAT-like.

Cheap models can be excellent reviewers

Haiku 4.5 matched the best critic F1 score in this run. That suggests a practical architecture: use a stronger model for difficult generation, then use a cheaper model for broad structural and defect screening.

Independent review is not optional

The judges disagreed most often about difficulty and distractor quality. That is exactly why a production workflow should preserve review states rather than silently publishing model output. A question can be mathematically valid and still need a coach or content editor’s judgment.

Batch generation is useful, but it changes the experiment

To make the Codex comparison affordable and operationally realistic, the fixed cases were batched into isolated one-shot processes. This preserves the task specifications and scoring rubric, but it is not identical to running one model request per question. Future experiments should test both batch and per-item generation.

Our current production recommendation

We would not use one model for every stage.

• Generation: a high-quality reasoning model such as Claude Opus 4.7 or an OpenAI premium model, with strict schemas and known-answer checks.
• High-volume review: Haiku 4.5 or a comparable inexpensive model for LaTeX, duplicate-content, option, and solution-mismatch detection.
• Final adjudication: a separate model from the generator, plus human review for difficulty, pedagogy, and originality.
• Publishing gate: no item enters the company content pool until deterministic validation passes and the review status is explicitly accepted.

The benchmark does not prove that one vendor is universally better. It gives us a safer operating model: generate, validate, audit, adjudicate, and only then promote content into a student-facing library.

Limitations and next steps

This was a small, controlled benchmark rather than a statistically powered study. It used 12 specifications, one primary run per configuration, and synthetic defect cases. It did not measure long-term student outcomes, originality against every public question bank, visual diagram quality, or performance on every SAT domain.

The next iteration should add hundreds of specifications, repeated trials, image-based geometry questions, plagiarism and near-duplicate detection, and human coach ratings. We also want to measure whether generated distractors diagnose real misconceptions instead of merely looking plausible.

The goal is not to find a model that never makes a mistake. The goal is to build a content system where mistakes are cheap to detect, impossible to publish silently, and easy for a coach to correct and promote.