RAG-Gym
AI for Evidence Utilization · Search Agents

Supervising the search process produces reliable and generalizable information-seeking agents

Guangzhi Xiong*, Qiao Jin*, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, Zhiyong Lu, Aidong Zhang
* equal contribution
arXiv 2025
PDF arXiv
These slides were generated with the help of AI and may contain errors.
01Motivation

Outcome-only rewards produce unreliable information-seeking agents

  • LLMs now act as autonomous agents that iteratively search external knowledge.
  • High-quality intermediate search steps are hard to generate and harder to supervise.
  • Rewarding only the final answer invites reward hacking and over-reliance on parametric memory.
  • A correct answer can hide a flawed search path — and that path generalizes poorly out of domain.
02Method

RAG-Gym formulates agentic search as a Markov decision process

  • Each search query is an action; the accumulated question, queries and documents form the state.
  • That framing makes every intermediate step addressable by a reward, not just the final answer.
Knowledge-intensive question answering formulated as an MDP: the agent alternates actions and environment stat
Knowledge-intensive question answering formulated as an MDP: the agent alternates actions and environment states until it answers. Fig. 1a.
03Method

RAG-Gym supervises the search process rather than the final answer alone

  • A wrong intermediate query can still yield the right answer, so outcome reward marks it correct.
  • RAG-Gym optimizes three axes: architecture design, parameter tuning, and action evaluation.
Left: why outcome supervision is insufficient. Right: the three components RAG-Gym optimizes. Fig. 1b–c.
Left: why outcome supervision is insufficient. Right: the three components RAG-Gym optimizes. Fig. 1b–c.
04Result

Reasoning reflection is the capability search agents most often lack

  • Sweeping architectures and tuning methods isolates which agent capabilities actually matter.
  • Re2Search is the only design carrying the full capability stack, and it leads on every axis of the radar.
Benchmark coverage by agent architecture under zero-shot, SFT, DPO and PPO training. Fig. 2d.
Benchmark coverage by agent architecture under zero-shot, SFT, DPO and PPO training. Fig. 2d.
05Result

Process supervision increases the number of search queries agents issue

  • Agents trained with process rewards issue more search queries before committing to an answer.
  • The shift is consistent across HotpotQA, 2WikiMultihopQA, Bamboogle and MedQA.
Average number of search queries per question, by training method. Fig. 3c.
Average number of search queries per question, by training method. Fig. 3c.
06Result

A process-reward critic improves accuracy across every agent architecture

  • Scoring candidate actions with a process-reward critic improves accuracy without retraining the actor.
  • The gain holds for Direct, CoT and RAG agents, and for ReAct, Search-o1 and Re2Search alike.
F1 with and without critic-guided action selection, across agents and benchmarks. Fig. 4b–c.
F1 with and without critic-guided action selection, across agents and benchmarks. Fig. 4b–c.
07Result

The critic transfers across model backbones, including proprietary actors

  • A Llama-3.1-8B critic improves a proprietary GPT-4o-mini actor — 2WikiMultihopQA rises from 50.4 to 58.0.
  • Bamboogle gains most, from 56.6 to 66.4 F1 on the tuned backbone.
Critic-guided selection on DPO-tuned Llama-3.1-8B and on GPT-4o-mini, against the untuned baseline. Fig. 4d.
Critic-guided selection on DPO-tuned Llama-3.1-8B and on GPT-4o-mini, against the untuned baseline. Fig. 4d.
08Result

Re2Search++ generalises to out-of-domain benchmarks

  • Re2Search++ leads every baseline on both Llama-3.1-8B and Qwen-2.5-7B backbones.
  • The margin is widest on Bamboogle, the out-of-domain benchmark neither model was tuned for.
F1 by agent and backbone; hatched bars are in-domain, solid bars out-of-domain. Fig. 5c.
F1 by agent and backbone; hatched bars are in-domain, solid bars out-of-domain. Fig. 5c.
09Result

Performance scales with both training samples and sampled actions

  • Accuracy rises steadily with the number of training samples, well past the zero-shot baseline.
  • It also rises with the number of sampled actions at inference — more candidates for the critic to choose from.
Scaling with training samples (left) and with sampled actions per step (right). The dashed line is the zero-sh
Scaling with training samples (left) and with sampled actions per step (right). The dashed line is the zero-shot baseline. Fig. 6e–f.
10Summary

RAG-Gym at a glance

BackgroundAgentic RAG interleaves search and reasoning to answer hard questions.
ProblemOutcome-only rewards cause reward hacking and poor out-of-domain search.
ApproachProcess supervision over an MDP — a critic scores every search step.
ResultsRe2Search++ beats outcome-supervised baselines; critics transfer to proprietary LLMs.
ConclusionA principled blueprint for training reliable, generalizable search agents.