RPTune: Learned Context Curation for LLM Catalog Search
RPTune addresses product search for small merchants whose catalogs fit inside long-context LLMs. Rather than relying on retrieval pipelines, the authors show that feeding the full catalog works only when items are learned to be pruned, ranked, and positioned with downstream LLM feedback, after which the LLM itself is post-trained on these curated contexts using purely synthetic supervision. The result is a fully automated pipeline that achieves large accuracy gains and latency reductions across 7 merchants and 9 model families.
The paper presents a conceptually clean pipeline for in-context catalog search and supports it with extensive experiments across proprietary and open-weight LLMs. The authors report that "RPTune consistently improves search accuracy across both proprietary and open-weight LLMs, with context curation yielding gains of up to 31.4 percentage points and post-training adding a further 10.3 points on average." However, the evaluation relies entirely on synthetic queries and LLM-generated relevance labels, which creates a closed loop between training data and ground truth, and the cross-backbone curation gains are not ablated to isolate the LLM-specific reorganizer from the encoder. The motivation for curation is nonetheless well-founded, aligning with Liu et al.’s observation that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."
The evidence that RPTune meaningfully outperforms fixed-budget context selectors is strong and internally consistent. The authors note that "substituting Jina Reranker 3.5 or Gemini Embedding 2 at the same retention budget degrades accuracy on 5 and 3 of the 7 merchants respectively, whereas RPTune improves all 7," and the ablation showing that a supervised reorganizer trained without LLM feedback "reaches 16.4% average EM, no better than the encoder alone and 4.3 points below RPTune’s reorganizer" confirms that the LLM-in-the-loop signal is actually doing work. The monotonic improvement across the staged pipeline—raw backbone to encoder-only to full curator to post-training—also holds up as a design validation.
The primary risk is a tightly closed evaluation loop built entirely on synthetic data. The authors concede that "Because gemini-3.1-pro assigns the reference labels, they may favor Gemini-family backbones," and while human audits reject 9.6% of cases and independent LLM judges are brought in, the 700-query evaluation set is still small and anchored to the synthetic labeler’s consensus. A second concern is limited ablation for cross-backbone transfer: the reorganizer is trained using group-relative advantages from a frozen gemma-4-E4B-it, yet when the curated context is evaluated across eight other LLMs the paper reports only the full curator without separating encoder-only and reorganizer-only contributions, so the gains on non-gemma backbones may largely reflect the encoder rather than genuine LLM-aware curation.
The evidence strongly supports the claim that full-catalog prompting dominates retrieval-based baselines for SMB-scale catalogs: the authors state that "directly prompting the LLM with the full catalog outperforms all baselines by at least 9.0 EM points." The retention-budget comparisons are fair because off-the-shelf selectors are forced to the exact same 25% retention rate. However, the baseline choices of RAG-Fusion, TourRank, and LongLLMLingua are somewhat mismatched because they target large-scale retrieval and compression rather than the high-recall, compact-context regime studied here; the paper acknowledges this distinction but still frames them as the primary comparators. The ordering ablations and pruning sweeps are well-executed, showing that RPTune is robust to layout variations while consistently improving over raw-context baselines.
Independent reproduction would be severely hampered by missing artifacts and proprietary dependencies. The authors provide detailed infrastructure notes—e.g., "gemma-4-E4B-it is trained on two 80GB H100 GPUs using Unsloth and evaluated on a single 80GB H100 GPU"—but they do not mention releasing code, model weights, merchant catalogs, or the synthetic query dataset. The pipeline depends on API calls to multiple proprietary services (Gemini, Claude, Grok, Moonshot) for both training data synthesis and baseline evaluation, and the LLM-in-the-loop reorganizer training requires running $G=16$ rollouts per query against a frozen gemma-4-E4B-it checkpoint. Without released data and code, verifying the reported gains on new merchants would require rebuilding the entire expensive synthetic pipeline from scratch.
For small merchant businesses (SMBs) whose catalogs fit within a long-context LLM, full-catalog prompting offers a compelling alternative to multi-stage retrieval designed primarily for large marketplaces with millions of items. However, fitting the full catalog into the context window does not ensure that the model can use it effectively, since LLMs do not exploit long contexts uniformly. We therefore study in-context catalog search through two complementary questions: (1) how to curate and present catalogs to the LLM, and (2) how to adapt the LLM for product selection on curated contexts. We propose RPTune, an end-to-end framework that couples learned catalog curation with LLM post-training using automatically generated, catalog-grounded supervision. An encoder-reorganizer curator orders and prunes products guided by downstream LLM feedback, while the resulting curated catalogs in turn improve the effectiveness of LLM post-training with a context-relative reward. We evaluate RPTune on 7 real merchants spanning distinct retail verticals, using 100 complex conversational queries per merchant. RPTune consistently improves search accuracy across both proprietary and open-weight LLMs, with context curation yielding gains of up to 31.4 percentage points and post-training adding a further 10.3 points on average.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.