Medical AI Evaluation · Quantitative Reasoning
MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
Nikhil Khandekar*, Qiao Jin*, Guangzhi Xiong*, Soren Dunn, Serina S. Applebaum, …, Qingyu Chen, Zhiyong Lu
* equal contribution
NeurIPS D&B 2024
These slides were generated with the help of AI and may contain errors.
01Motivation
Medical AI is tested on knowledge, not calculation
What clinicians actually do
- Doctors rely on clinical calculators — equations and rule-based scores — for evidence-based decisions.
- These need quantitative, step-by-step computation, not descriptive recall.
The evaluation gap
- Existing medical-LLM benchmarks test knowledge QA and descriptive reasoning.
- None measured whether models can actually compute clinical values.
02Method
A benchmark of 55 medical calculation tasks
- Each instance: a patient note, a question asking for a value, a ground-truth answer & a step-by-step solution.
- Two families — rule-based scores (19) and equation-based calculators (36).
- Patient notes build on our PMC-Patients collection.
1,047
expert-reviewed eval instances
55
calculation tasks
10k
training instances
03Result
Even GPT-4 reaches only 50.9%
- Across proprietary, open-source & biomedical LLMs, medical calculation is hard.
- The best model — GPT-4 with one-shot chain-of-thought — scores just 50.9%.
- Error analysis: models lack calculator knowledge and make computation mistakes.
50.9%
best accuracy — GPT-4, one-shot CoT
04Impact
A standard benchmark for medical calculation
- Frontier labs benchmark on it — Anthropic evaluates Claude on MedCalc-Bench with code execution.
- A Clinical Decision Support task in Stanford’s MedHELM benchmark suite.
- Widely used to evaluate quantitative reasoning — an o1-in-medicine study, MEDIC & MedAgentGym.
- Extended by others — CMedCalc-Bench (EMNLP’25), MedCalc-Eval & MedCalcBenchV2.
05Summary
MedCalc-Bench at a glance
| Background | Clinicians rely on equation- and rule-based calculators for decisions. |
| Problem | Yet medical-LLM benchmarks tested knowledge QA, never quantitative calculation. |
| Approach | 1,047 expert-reviewed instances across 55 clinical calculators (19 rule-based, 36 equation-based). |
| Results | Medical calculation is hard — even GPT-4 (one-shot CoT) reaches only 50.9%. |
| Impact | Benchmarked by Anthropic and in Stanford’s MedHELM; extended by CMedCalc-Bench & more. |