PubMedQA
Biomedical QA · Reasoning Benchmark

PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, Xinghua Lu
EMNLP 2019
PDF ProceedingsHugging FaceLeaderboard
These slides were generated with the help of AI and may contain errors.
01Motivation

Can models reason over biomedical research text?

The gap
  • QA is the standard probe for reasoning, but biomedical QA was small and factoid (BioASQ has <3k).
  • Larger auto-built sets ask questions whose answers can be copied from the text, needing little reasoning.
Our aim
  • A biomedical QA dataset with expert labels that requires reasoning over the quantitative results in abstracts.
02Task

Answer research questions yes / no / maybe

  • Each question is a research-article title; the context is its abstract without the conclusion.
  • The model must answer yes / no / maybe by reasoning over the abstract’s quantitative results.
An instance: the answer needs the results — here 16% vs 33% postoperative AF (p=0.005) →
An instance: the answer needs the results — here 16% vs 33% postoperative AF (p=0.005) → yes. Fig. 1.
03Dataset

Three subsets: expert, unlabeled & artificial

  • Questions are article titles; the context is the abstract with its conclusion removed.
  • Labels come from the conclusion — hand-annotated for PQA-L, auto-derived (yes/no) for PQA-A.
1.0k
expert-labeled · PQA-L
61.2k
unlabeled · PQA-U
211.3k
artificial · PQA-A
04Result

Reasoning over quantitative results is hard

  • The best model — multi-phase fine-tuned BioBERT with long-answer statistics — reaches 68.1%.
  • A single human scores 78.0% and the majority baseline 55.2% — a wide reasoning gap remains.
Majority: 55.2%55.2%MajorityBest model: 68.1%68.1%Best modelSingle human: 78.0%78.0%Single humanAccuracy on PubMedQA
Accuracy on the PQA-L test set (reasoning-required setting). Source: Table 5 & abstract.
05Impact

A foundational benchmark for medical LLMs

06Summary

PubMedQA at a glance

BackgroundQA is the standard probe for machine reasoning, but biomedical QA was small and factoid.
ProblemNo benchmark required reasoning over the quantitative results in research abstracts.
Approach1k expert + 61.2k unlabeled + 211.3k artificial yes/no/maybe QA from PubMed abstracts.
ResultsBest model 68.1% vs. 78.0% single-human — a wide reasoning gap.
ImpactThe default benchmark for open medical LLMs.