MedReview
AI for Evidence Utilization · Summarize

Closing the Gap Between Open-Source and Commercial LLMs for Medical Evidence Summarization

Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas R. Campion Jr., Zhiyong Lu, Chunhua Weng, Yifan Peng
npj Digit. Med. 2024
PDF Journal
These slides were generated with the help of AI and may contain errors.
01Motivation

Open models trail commercial ones at summarizing evidence

A high-value EBM step
  • Summarizing many studies into a conclusion is one of the most valuable steps in evidence-based medicine.
The open-model gap
  • Most work relies on proprietary models — raising transparency and vendor-lock-in concerns.
  • Open models have lagged in quality.
02Method

A benchmark of 8,161 review–summary pairs

  • MedReview — 8,161 systematic-review & summary pairs from the Cochrane Library (37 topics).
  • Three open models fine-tuned with LoRA: PRIMERA, LongT5, Llama-2.
The dataset’s topic distribution (top) and the fine-tuned models (bottom).
The dataset’s topic distribution (top) and the fine-tuned models (bottom). Fig. 1.
03Result

Fine-tuning closes the gap

Fine-tunedZero-shotPRIMERA · Fine-tuned: 20.520.5PRIMERA · Zero-shot: 18.918.9PRIMERALongT5 · Fine-tuned: 24.624.6LongT5 · Zero-shot: 14.714.7LongT5Llama-2 · Fine-tuned: 20.020.0Llama-2 · Zero-shot: 17.017.0Llama-2ROUGE-L (median)
ROUGE-L medians, zero-shot vs. fine-tuned (PRIMERA & LongT5 exact from text; Fig. 2).
04Result

A small fine-tuned model beats a larger zero-shot one

LongT5-base (fine-tuned)LongT5-xl (zero-shot)ROUGE-L · LongT5-base (fine-tuned): 2424ROUGE-L · LongT5-xl (zero-shot): 1515ROUGE-LMETEOR · LongT5-base (fine-tuned): 2626METEOR · LongT5-xl (zero-shot): 1919METEORCHRF · LongT5-base (fine-tuned): 4242CHRF · LongT5-xl (zero-shot): 3030CHRFSummary-quality scores
A smaller fine-tuned model beats a larger zero-shot one. Digitized from Fig. 3.
05Result

Confirmed by experts and GPT-4 evaluation

Fine-tuned betterZero-shot betterCompreh. · Fine-tuned better: 3030Compreh. · Zero-shot better: 88Compreh.Consistency · Fine-tuned better: 2828Consistency · Zero-shot better: 1010ConsistencyReadability · Fine-tuned better: 2626Readability · Zero-shot better: 1212ReadabilitySpecificity · Fine-tuned better: 2424Specificity · Zero-shot better: 1414SpecificitySummaries judged better
Number of summaries judged better, by dimension (fine-tuned vs. zero-shot Llama-2). Digitized from Fig. 5.
06Summary

MedReview at a glance

BackgroundSummarizing the evidence in a systematic review is slow, expert work.
ProblemOpen LLMs trail commercial ones at medical evidence summarization.
ApproachFine-tune open models on 8,161 review–summary pairs.
ResultsFine-tuned LongT5 ≈ GPT-3.5; a small fine-tuned model beats a large zero-shot one.
ConclusionFully open, reproducible medical summarization — no proprietary model needed.