Case study · NeurIPS 2026 Workshop (UserSim) · arXiv 2610.19560

τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Rishitosh Kumar Singh*Amir Saeidi*†Shamanthak Hegde*Shrinidhi KumbharChitta Baral

* equal contribution · † corresponding author · Arizona State University

Agent benchmarks assume a calm, cooperative user. Real customers are angry, anxious or in a hurry. τ-Emotion-Bench keeps the task fixed and changes only the user's emotion, and lets the user walk away if the agent handles the conversation badly.

PaperLeaderboardDatasetModelsCodeProject site

  • 580verified tasks
  • 10domains
  • 173tools
  • 22emotions
  • 594emotion conditions
  • 1,958training trajectories

Headline result

With emotional users, the best frontier agent reaches only 55.9% Avg@3, and no model passes more than 32.2% of tasks in all three trials.

Even the best frontier agent, DeepSeek-V4.1-Flash, solves only 55.9% of tasks on average when the user is emotional.

  1. DeepSeek-V4.1-Flash 55.9DeepSeek-V4.1-FlashDeepSeek · non-thinkingAvg@3 55.9% · Pass^3 32.2%Loses 23.7 pts to inconsistency
  2. GLM-5.3 51.9GLM-5.3Z.ai · thinkingAvg@3 51.9% · Pass^3 31.7%Loses 20.2 pts to inconsistency
  3. Qwen3.8-27B 50.0Qwen3.8-27BQwen · thinkingAvg@3 50.0% · Pass^3 27.3%Loses 22.7 pts to inconsistency
  4. Kimi-K2.7-Code 49.3Kimi-K2.7-CodeMoonshot AI · thinkingAvg@3 49.3% · Pass^3 26.1%Loses 23.2 pts to inconsistency
  5. Gemma-4-31B-it 44.8Gemma-4-31B-itGoogle · thinkingAvg@3 44.8% · Pass^3 24.7%Loses 20.1 pts to inconsistency
  6. MiniMax-M2.7 38.6MiniMax-M2.7MiniMax · thinkingAvg@3 38.6% · Pass^3 16.8%Loses 21.8 pts to inconsistency
  7. Tracer-High-8B ours38.1Tracer-High-8Bτ-Emotion-Bench · thinkingAvg@3 38.1% · Pass^3 19.8%Loses 18.3 pts to inconsistency
  8. MiniMax-M3 37.0MiniMax-M3MiniMax · thinkingAvg@3 37.0% · Pass^3 17.6%Loses 19.4 pts to inconsistency
  9. Qwen3-235B-A22B (thinking) 36.0Qwen3-235B-A22B-Thinking-2507Qwen · thinkingAvg@3 36.0% · Pass^3 16.5%Loses 19.5 pts to inconsistency
  10. Muse-Glimmer-30B 35.1Muse-Glimmer-30BMeta · thinkingAvg@3 35.1% · Pass^3 20.5%Loses 14.6 pts to inconsistency
  11. Qwen3-235B-A22B (no thinking) 32.6Qwen3-235B-A22B-Instruct-2507Qwen · non-thinkingAvg@3 32.6% · Pass^3 13.8%Loses 18.8 pts to inconsistency
  12. Qwen3-32B (thinking) 28.5Qwen3-32BQwen · thinkingAvg@3 28.5% · Pass^3 8.6%Loses 19.9 pts to inconsistency
  13. Qwen3-32B (no thinking) 23.9Qwen3-32BQwen · non-thinkingAvg@3 23.9% · Pass^3 9.4%Loses 14.5 pts to inconsistency

Trained on our Tracer dataOther modelsAvg@3: mean success rate over three trials. Snapshot 2026-10-19. Full leaderboard ↗

How the benchmark is built

Tracer is an automated pipeline that builds policy-valid, verified tasks, then reuses the same harness to collect training trajectories.

  1. 1Dependency graphsValid tool transitions per domain, manually verified.
  2. 2Trace samplingExecutable multi-step workflows, including multi-intent tasks.
  3. 3Reverse refinementEach trace becomes a natural user instruction with goals and preferences.
  4. 4Emotion mixingA task-specific feeling is mapped to one of 594 emotion conditions.
  5. 5VerificationExecutability and policy checks, plus manual review of all 580 tasks.
Overview of τ-Emotion-Bench and the Tracer data-generation pipeline
Overview of τ-Emotion-Bench and the Tracer pipeline (Figure 2 of the paper).

Emotions follow Ekman's Atlas of Emotions, crossed with politeness, urgency and trust: AngerDisgustEnjoymentFearSadness

Ten domains, 173 tools

  • Airline8 + 670 tasks
  • Retail9 + 770 tasks
  • Telecom12 + 670 tasks
  • Telehealth18 + 570 tasks
  • School9 + 450 tasks
  • Basketball17 + 350 tasks
  • Restaurant9 + 650 tasks
  • Travel9 + 750 tasks
  • Movie13 + 350 tasks
  • Bank13 + 950 tasks

read toolswrite tools

Consistency is the hard part

Share of tasks solved in all three trials (Pass^3, %), by domain. Every model swings widely: strong in School or Retail, far weaker in Travel and Movie.

AgentAirlineRetailTelecomTelehealthSchoolBasketballRestaurantTravelMovieBankAvg.
DeepSeek-V4.1-Flash3344173142382630223832.2
GLM-5.3394632048622814302831.7
Qwen3.8-27B3339113048283016102827.3
Kimi-K2.7-Code304410144434121285226.1
Gemma-4-31B-it303713235626186122624.7
Tracer-High-8B27311631381612222219.8
Qwen3-8B4141323012420128.4

Qwen3-235B-A22B-Instruct as the simulated user. Tracer-High-8B is our fine-tune of Qwen3-8B.

Training with Tracer

  • Narrows the gap. Tracer-High-8B reaches 38.1% Avg@3, above MiniMax-M3 (37.0%) and Qwen3-235B-A22B-Thinking (36.0%).
  • Generalizes. Tracer-Low, trained on four domains only, has the best out-of-domain Pass^3 at every scale, against methods trained on up to 91K trajectories.
  • Transfers. Gains carry over to τ-Bench and τ-Trait, which use different tasks and user simulators.
Grouped bars of pass^1, pass^2 and pass^3 for Qwen3 base, Tracer-Low and Tracer-High at 1.7B, 4B and 8B on τ-Bench Airline, τ-Bench Retail, τ-Trait Telecom and τ-Trait Telehealth
Qwen3 base vs. Tracer-Low and Tracer-High on τ-Bench and τ-Trait (Pass^1 to Pass^3).

Key findings

Policy violations persist

Violations drop with capability and Tracer tuning, but reach about 67% in Telehealth and 91% in Telecom even for stronger models.

Dot plots of policy violation rate per emotion family in each domain, for Qwen3, Tracer-Low, Tracer-High, stronger models and weaker models

Balance tool use and follow-up

Higher Avg@3 goes with a moderate tool-call to follow-up ratio; more aggressive tool use does not help.

Five scatter panels, one per emotion family, of Avg@3 against the tool-call to follow-up ratio

Empathy adds follow-up

Empathetic replies come with more follow-up questions and often lower success, especially under Enjoyment.

Heatmap of change in success with empathy by emotion and model group, and share of follow-up questions with and without empathy

The user simulator is reliable

Of 6,961 failed trajectories, only 124 user turns are confirmed user errors (1.1% of failed trials), at most 1.4% per emotion family.

Heatmap of user hallucination rates by sub-emotion and domain, mostly empty

Why Tracer generalizes

Tool calls outnumber follow-up actions in 76–79% of Tracer trajectories, versus 56% for AReaL and 22% for Simia.

Scatter plots of tool calls against follow-up actions in AReaL, Simia, Tracer-Low and Tracer-High training trajectories