DecomposeR: Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward

Deep research tasks require LLMs to plan what to investigate, retrieve evidence, and synthesize long-form answers across multiple branches of inquiry. Existing training paradigms either rely on short-form verifiable QA as a proxy or optimize monolithic long trajectories, which makes planning and execution difficult to disentangle and yields weak credit assignment for the planning process.

We propose DecomposeR, a planner-centric deep research framework that represents research plans as typed directed acyclic graphs (DAGs), allowing planning to be made explicit, structured, and rewardable. We train a Qwen3-8B model in two stages: planner RL first learns graph structure and query decomposition to improve research planning, and answerer RL then learns branch-level execution and final synthesis conditioned on the learned plan.

By assigning rewards to explicit planner tokens and structured components rather than to a flat trajectory, DecomposeR enables finer-grained optimization of planning while reducing the ambiguity of end-to-end training. Experiments show that DecomposeR-8B improves over strong comparable open baselines by 5.1–8.0 points on popular long-form benchmarks due to improved planning and answering capabilities.

The core motivation behind DecomposeR is that current RL-trained deep research systems suffer from two limitations: ambiguous credit assignment — reasoning, search, evidence selection, and synthesis are interleaved in a flat ReAct-style trajectory, so a single scalar reward propagated across every action cannot identify which failure point to correct — and sparse rewards — intermediate planning and search decisions receive no direct supervision.

DecomposeR addresses both by materializing the research plan as an explicit typed DAG in which search nodes issue web queries, aggregate nodes synthesize branch-level conclusions, and a terminal answer node integrates branches into the final report. Because every plan component is individually addressable, reward can be assigned to the specific part of the research process responsible for each decision.

Rollout Structure

The DecomposeR rollout proceeds in five steps at both training and inference time:

1. Initial Planning: Given a query, the planner emits an initial plan G₀ as a typed DAG.
2. Initial Search: The environment executes every search node in G₀ and returns retrieved observations.
3. Plan Revision: The planner consumes the search results and emits a revised plan G₁ together with a small fetch set of high-value URLs.
4. Conditional Search: The environment runs only newly added or modified search nodes in G₁, reusing observations from unchanged nodes.
5. Answer Generation: The answerer fills aggregate nodes of G₁ in topological waves and produces the final citation-grounded report at the answer node.

The research plan G is a typed DAG with three node types. Search nodes carry a query string and a list of key points specifying what evidence the query should return. Aggregate nodes carry a synthesis brief — a need statement and key points to be covered in output. The single answer node v_ans carries the final synthesis brief and has no outgoing edges. A directed edge u→v declares that the output of u is consumed by v.

A validator enforces that G is a connected DAG with valid JSON syntax, valid node IDs, allowed node types, no cycles, and exactly one v_ans. Invalid graphs are rejected at a validity gate before reward computation — this ensures protocol correctness is established before quality signals take effect.

Given a valid G₁, the answerer executes it wave by wave. A wave contains all aggregate nodes whose parents have already been completed; all nodes in a wave are generated in a single model turn, exploiting conditional independence of same-wave aggregates given their parents. This topological-wave execution preserves dependency order while minimizing model turns.

Why a Typed DAG?

A typed DAG with search-, aggregate-, and answer-type nodes makes cross-source evidence reuse and hierarchical synthesis explicit at the structural level — both of which linear lists and tree-structured plans cannot natively express. The structure exposes well-defined properties (branch breadth, cross-branch integration, query distinctness) as addressable units that planner reward can target directly.

The planner and answerer are trained sequentially atop a shared Qwen3-8B backbone with role-conditioned prompts and role-specific LoRA adapters. Training proceeds in three phases.

Why Decouple Planner and Answerer?

Collapsing planning and answering into one policy under a single trajectory-level reward conflates their failure modes. By training the two roles sequentially, Planner RL optimizes rewards that depend only on the revised plan G₁ and never on answerer output, so plan quality is isolated from execution noise. Answerer RL then trains against a fixed planner, so answerer credit is not confounded by drifting plan quality. Staging also reduces the non-stationarity inherent to jointly training two policies.

DecomposeR-8B (SFT + Planner RL + Answerer RL) is evaluated on three long-form deep research benchmarks: DeepResearchBench (DRBench), HealthBench, and ResearchQA-Mini. All benchmarks evaluate long-form research responses requiring multi-step retrieval, evidence synthesis, and citation grounding.

We conduct six ablation groups to isolate the contribution of each design decision in DecomposeR. Full numerical results for each ablation are reported in the paper; the descriptions below summarize the key findings.

B1 — Staged vs. Joint RL

We compare training both the planner and answerer jointly under a single end-to-end reward against the proposed two-stage approach of first running Planner RL with the answerer held fixed, then running Answerer RL with the planner held fixed. Joint RL produces weaker planning graphs and less stable answerer convergence; staged training consistently achieves higher scores on all three benchmarks, confirming that decoupling the two roles prevents non-stationarity from drifting plan quality confounding answerer credit.

B2 — Planner Quality with a Fixed External Answerer

To assess planner quality independently of the trained answerer, we evaluate several planner variants (SFT-only, after Planner RL) by pairing each with a single fixed strong external answerer. This isolates the quality of the plan structure from execution skill. Planner RL plans consistently produce better downstream answers even when executed by the same answerer, confirming that reward gains in Phase 2 reflect genuine plan quality improvements rather than answerer over-fitting to planner quirks.

C1 — Planner Reward Component Ablations

The planner reward combines three components: Rubric Coverage (R_rub — plan nodes semantically anticipate query-specific rubric criteria), Search Quality (R_search — search nodes target the right aspects with low redundancy), and Graph Expressiveness (R_graph — aggregate branching, cross-branch integration, and search breadth). We ablate each component by removing it while keeping the other two. Removing any single component degrades downstream benchmark performance, with Rubric Coverage contributing the largest individual gain, indicating that anchoring plans to query-specific criteria is the most critical structural signal.

C2 — Answerer Reward Component Ablations

The answerer reward combines: Execution Reward (R_exec — whether each aggregate node covers its declared key points), LLM Judge Reward (R_jrub — overall synthesis quality as judged by a frontier LLM), and Branch Utilization (U_branch — whether the final answer integrates evidence from all branches). Ablating each in turn shows that execution fidelity is the most critical for alignment with the plan, while the LLM judge reward drives overall answer quality. Removing branch utilization leads to answers that skip branches, reducing coverage on multi-faceted queries.

D1 — Plan Structure: Typed DAG vs. Tree vs. Linear

We compare three plan representation types: a Typed DAG (the proposed structure, which allows cross-branch evidence reuse via multiple parents), a Tree (each node has exactly one parent, preventing cross-branch edges), and a Linear list (a flat sequence of search and synthesis steps with no structure). The Typed DAG outperforms both alternatives, with the Tree outperforming Linear. The margin between DAG and Tree is most pronounced on queries requiring integration of evidence from multiple independent search branches, confirming that the ability to express cross-source dependencies is load-bearing.

D2 — Single-Turn vs. Two-Turn Planning

We compare a single-turn planning variant — where the planner emits the final graph G₁ directly from the query without seeing any search results — against the proposed two-turn design (Turn 1: initial graph G₀; environment executes search nodes; Turn 2: revised graph G₁ + fetch set). Two-turn planning consistently yields higher quality plans as measured by downstream benchmark scores and graph validity rate. The plan revision step allows the planner to ground its graph in actual retrieved evidence, leading to more targeted aggregate nodes and better fetch set selection.