PubTator 3.0 official Nucleic Acids Research paper
AI for Evidence Utilization · Retrieve

PubTator 3.0: an AI-powered literature resource for unlocking biomedical knowledge

Chih-Hsuan Wei†, Alexis Allot†, Po-Ting Lai, Robert Leaman, Shubo Tian, Ling Luo, Qiao Jin, Zhizheng Wang, Qingyu Chen, Zhiyong Lu
† Joint first authors
Nucleic Acids Research 2024
PDFJournal
These slides were generated with the help of AI and may contain errors.
01Motivation

Keyword matching can miss relevant biomedical evidence

  • Different names for the same biomedical concept can reduce recall in keyword searches.
  • Articles can mention both query entities without evidence for the requested relationship.
EntitiesMap synonymous names to a shared biomedical identifier.
RelationsSpecify a relationship (e.g., a chemical reducing gene expression).
Source: Introduction, pp. W540–W541.
03Method

PubTator annotates new biomedical articles weekly

  • AIONER identifies 6 entity types; normalization maps entity mentions to database identifiers.
  • BioREx extracts 12 relation types for 8 entity-type pairs.
Official Figure 2A: PubMed and PMC literature passes through entity recognition, entity normalization, relation extraction, and storage for PubTator search.
Annotations are stored in MongoDB and indexed with Solr. Open the complete figure for details. Fig. 2A.
04Resource

PubTator provides large-scale biomedical annotations

  • PubTator searches PubMed abstracts and full-text articles in the PMC Open Access Subset.
  • The web interface enables interactive search; APIs and bulk downloads enable programmatic use.
~36M
PubMed abstracts
6M+
PMC-OA full texts
1.6B+
entity annotations
33M
relation annotations
Counts in the 2024 paper. Abstract and full-text collections overlap; annotation counts include repeated mentions and relations. Source: System overview, p. W541.
05Result

PubTator achieves 90.0% pooled precision in 12 retrieval case studies

  • 2 curators evaluated the top results for evidence of a relationship between the query entities.
  • PubTator achieves higher pooled precision than PubMed and Google Scholar in this evaluation.
Search systemRelevant / judgedPooled precision
PubTator 3.0216 / 24090.0%
PubMed84 / 10381.6%
Google Scholar98 / 20248.5%
Up to 20 results per query; denominators vary by system. Searches used 12 selected entity pairs on 19 May 2023. Curators knew the search system. Source: Case study I, pp. W543–W544.
06Case Study

PubTator improves GPT-4 citation accuracy on 8 questions

  • GPT-4 uses 3 PubTator APIs to retrieve evidence for biomedical relationships.
  • In a qualitative evaluation of 8 questions, PubTator-augmented GPT-4 achieves higher citation accuracy than PubMed-augmented and standard GPT-4.
Find the entityResolve a concept (e.g., breast cancer) to a standardized identifier.
Find relationsIdentify entities with the requested relation (e.g., drugs that treat breast cancer).
Find evidenceRetrieve PubMed article identifiers for evidence of the requested relationships.
Citation accuracy measures if cited articles provide evidence for the stated relationships. Source: Case study II, p. W544.
07Limitations

The 2024 PubTator system extracts relations from abstracts

  • In the 2024 system, relation extraction uses abstracts only; entity annotation also includes full texts.
  • Relation extraction includes 12 types, and automated annotations can contain errors.
  • The case studies evaluate 12 entity-pair queries and 8 questions; broader retrieval and clinical performance require further evaluation.
Source: Discussion, p. W545; case studies, pp. W543–W544.
08Summary

PubTator 3.0 enables entity and relation retrieval

BackgroundBiomedical questions often concern relationships between entities.
ProblemKeyword matching can miss synonyms and specific relationships.
ApproachPubTator uses normalized entities and extracted relations for search.
Results90.0% pooled precision (216/240) in 12 retrieval case studies.
ConclusionEntity and relation annotations enable biomedical evidence retrieval.