Med-V1
AI for Evidence Utilization · Verify

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

Qiao Jin, Yin Fang, Lauren He, Yifan Yang, Guangzhi Xiong, Zhizheng Wang, Nicholas Wan, Joey Chan, Donald C. Comeau, Robert Leaman, Charalampos S. Floudas, Aidong Zhang, Michael F. Chiang, Yifan Peng, Zhiyong Lu
arXiv 2026
PDF arXiv
These slides were generated with the help of AI and may contain errors.
01Motivation

Citation verification must be cheap enough to run at scale

The scale
  • ~150M medical conversations happen on ChatGPT every day.
  • Each answer cites sources — but citing a paper doesn’t mean it supports the claim.
The cost
  • Verifying those citations with GPT-5 would cost over $500k/day.
  • … but only ~$5k/day with a 3B model.
The requirement
  • Feasible verification needs small models that are as good as frontier ones.
02Method

Med-V1 is trained on MedFact-Synth by supervised fine-tuning and GRPO

  • Two 3B verifiers — Med-V1-L3B (from Llama-3.2-3B) and Med-V1-Q3B (from Qwen-3B).
  • Trained by supervised fine-tuning, then reinforcement learning (GRPO) on 1.5M synthetic pairs.
  • One model emits a verdict plus explanation for QA, claim verification, and citation attribution.
Overview: MedFact-Synth trains a 3B Med-V1 that emits a verdict + explanation.
Overview: MedFact-Synth trains a 3B Med-V1 that emits a verdict + explanation. Fig. 1.
03Method

MedFact-Synth: large-scale synthetic data

  • From a source paper, frontier LLMs write one supporting and one refuting claim.
  • Our MedCPT retrieves candidate articles.
  • A frontier-LLM panel scores agreement on a five-point scale; 1.5M high-consensus pairs are kept.
Generation and evaluation of MedFact-Synth.
Generation and evaluation of MedFact-Synth. Fig. 2.
04Result

Med-V1 closes the small–frontier performance gap

  • MedFact-Bench aggregates five verification sets — SciFact, HealthVer, MedAESQA, and versions of our PubMedQA & BioASQ.
  • Untrained 3B backbones verify poorly (~51% avg); frontier LLMs saturate at ~72–74%.
  • Trained on MedFact-Synth, Med-V1 gains +42% to match GPT-5 — consistently across all five datasets.
MedFact-Bench average: both Med-V1 variants (~0.73) match the frontier band, +42% over their 3B back
MedFact-Bench average: both Med-V1 variants (~0.73) match the frontier band, +42% over their 3B backbones. Fig. 3.
05Application 1

Med-V1 flags citations that do not support the claims they accompany

  • Med-V1 checks each generated citation-statement.
  • Flags claims whose cited paper does not actually support them.
Verifying LLM-generated citations at scale.
Verifying LLM-generated citations at scale. Fig. 4a.
06Application 2

Med-V1 identifies high-stakes misattributions

  • Applied to 57k clinical-guideline statement–article pairs.
  • Surfaces contradictions — e.g. a cited “32% risk reduction” where the source reported a 4%→2.5% drop.
Auditing clinical-guideline citations.
Auditing clinical-guideline citations. Fig. 5a.
07Summary

Med-V1 at a glance

BackgroundMedical AI now emits ~1B citation-backed statements a day.
ProblemVerifying each one with a frontier LLM is far too expensive at that scale.
ApproachA 3B verifier trained on synthetic MedFact-Synth claim–article pairs.
ResultsMatches frontier LLMs (GPT-5) on verification, at ~$5k vs >$500k/day.
ConclusionA small specialized verifier makes citation-checking deployable — in case studies with NCI, NEI & Cornell.