MedRAG
AI for Evidence Utilization · Summarize

Benchmarking Retrieval-Augmented Generation for Medicine

Guangzhi Xiong*, Qiao Jin*, Zhiyong Lu, Aidong Zhang
* equal contribution
ACL Findings 2024
PDF Proceedings
These slides were generated with the help of AI and may contain errors.
01Motivation

Medical RAG had no standard way to build or evaluate it

  • Grounding an LLM on retrieved sources cuts hallucination.
  • But a medical RAG system has many moving parts — which corpus, which retriever, which backbone.
  • No standard way to know which choices actually matter.
02Method

The MIRAGE benchmark for medical RAG

The MIRAGE benchmark: 7,663 questions across five datasets, multiple-choice and yes/no.
The MIRAGE benchmark: 7,663 questions across five datasets, multiple-choice and yes/no. Fig. 1.
03Method

A toolkit of interchangeable parts

  • Corpora — PubMed, StatPearls, textbooks, Wikipedia → combined MedCorp.
  • Retrievers — BM25, Contriever, SPECTER, and our MedCPT.
  • Backbones — six LLMs; 41 combinations benchmarked.
The MedRAG toolkit’s interchangeable parts.
The MedRAG toolkit’s interchangeable parts. Fig. 2.
04Result

MedRAG lifts every backbone LLM

  • Grounding on retrieved evidence beats chain-of-thought for every model.
  • GPT-3.5 + MedRAG (72%) nearly matches GPT-4 without RAG (73%) — prompt engineering, not a bigger model.
Backbone LLMChain-of-thought+ MedRAGΔ
GPT-473.480.0+6.5
GPT-3.560.771.6+10.9
Mixtral 8×7B61.469.5+8.1
Llama-2 70B50.253.4+3.1
MEDITRON 70B57.060.2+3.1
PMC-LLaMA 13B52.452.9+0.5
Average accuracy (%) across the five MIRAGE datasets: chain-of-thought vs. MedRAG. Source: Table 6.
05Result

The corpus matters more than the retriever

  • PubMed is the best single corpus; combining all sources (MedCorp) wins.
  • Retriever choice matters far less — our MedCPT leads but barely beats BM25.
GPT-3.5 accuracy across every corpus × retriever combination: rows (corpora) vary far more tha
GPT-3.5 accuracy across every corpus × retriever combination: rows (corpora) vary far more than columns (retrievers). Table 7.
06Result

Log-linear gains with more retrieved snippets

  • More retrieved snippets → higher accuracy, log-linear — echoing pretraining’s compute–performance scaling.
  • This favors long-context LLMs, which can pack in more evidence.
Accuracy vs. number of retrieved snippets across five QA datasets — log-linear gains on most (
Accuracy vs. number of retrieved snippets across five QA datasets — log-linear gains on most (retrieval in blue, no-retrieval baseline red dashed). Source: Fig. 3.
07Impact

A de-facto standard for medical RAG

08Summary

MedRAG at a glance

BackgroundRetrieval promises to ground medical LLMs in real evidence.
ProblemBut medical RAG had no standard way to build or evaluate it.
ApproachThe MedRAG toolkit — corpora, retrievers & LLMs — with the MIRAGE benchmark (7,663 Qs).
ResultsRAG lifts every backbone LLM — up to +18%; the corpus choice matters most.
ImpactA de-facto standard for medical RAG; extended by our i-MedRAG & RAG-Gym.