$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions
Existing agent benchmarks generally assume cooperative, emotionally stable users, overlooking a major source of real-world failure. This paper introduces τ-Emotion-Bench, a ten-domain benchmark that evaluates tool-calling agents under controlled emotion-conditioned user interactions, together with Tracer, a pipeline that collects training trajectories through an interaction-aware harness where dissatisfied users can terminate inefficient exchanges. It matters because it reveals large reliability gaps in frontier models and shows that small models can be trained to substantially improve interaction consistency with far less data than prior methods.
The paper makes a solid, practically relevant contribution by identifying interaction fragility as a distinct failure mode and demonstrating that an emotion-conditioned harness yields both a harder benchmark and a useful training signal. The empirical gains from Tracer are consistent across scales, but the scope is limited to supervised fine-tuning of Qwen3 models, and the broader validity of LLM-simulated emotions as a proxy for real human affect remains unproven.
The benchmark construction is principled and scalable: tasks are built on manually verified tool dependency graphs, executable reference trajectories, and an emotion taxonomy grounded in Ekman’s Atlas of Emotions that yields 594 distinct conditions. The $ ext{Pass}^k$ metric is well motivated and exposes severe inconsistency—e.g., GLM-5.3 achieves 78.6% Avg@3 but only 66.5% $ ext{Pass}^3$ under one simulator—while Tracer’s sample efficiency is striking, outperforming methods trained on 9K–91K trajectories using only 1,498 out-of-domain trajectories.
Several limitations temper the strong empirical results. First, the central behavioral claim that Tracer improves the balance between tool use and follow-up is explicitly acknowledged by the authors as correlational. Second, the training experiments are restricted to the Qwen3 family, leaving open whether the supervision transfers to different architectures or RL pipelines.
Third, evaluation conditions are not fully controlled: some baselines use different prompting schemes—for instance, AReaL and Simia-Tau are evaluated with $\tau^2$-Bench wrapped prompts—and decoding configurations vary across models from greedy to temperature 1.0, which could independently affect interaction style.
The paper supports its core claims with detailed per-domain breakdowns, two distinct user simulators, and cross-benchmark transfer to $\tau$-Bench and $\tau$-Trait. However, the transfer benchmarks share underlying environments and APIs with $\tau$-Emotion-Bench, so the generalization may be narrower than claimed. The user hallucination analysis is reassuringly rigorous—only 1.1% of failed trials show simulator error—but this does not validate that LLM-simulated emotions alter agent behavior in the same way real human emotions would, which is the harder generalization.
Reproducibility is generally strong: the authors release code, models, and datasets on Hugging Face and GitHub, and report detailed fine-tuning hyperparameters in Table 8 including learning rate $5\times 10^{-5}$, cosine schedule, and batch size 32. That said, full reproduction requires access to proprietary API models such as GLM-5.2 and Qwen3-235B for user simulation, and the vLLM serving stack on specific hardware. The inconsistent decoding configurations across evaluated models—ranging from temperature 0 to 1.0—are another variable that could be more tightly controlled.
In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce $\tau$-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to $\tau$-Bench and $\tau$-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, $\tau$-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.