XYBench: Helping with the problem behind the question
Researchers introduced XYBench, a new benchmark of over 8,000 real‑world questions designed to test AI assistants’ ability to detect and correct the classic “XY problem,” where users ask about a mistaken approach rather than their true goal. Evaluations show leading models like GPT‑5 and Claude Sonnet often fail to redirect users toward the underlying solution, highlighting a key gap in current LLM troubleshooting capabilities.