Arena.ai unveils 'Agent Arena'; GPT-5.5 tops real-task leaderboard
Arena.ai (formerly LMSYS Chatbot Arena / LMArena) on June 4, 2026 announced 'Agent Arena,' a new platform that scales real-world agentic evaluation. Across millions of live user sessions, it measures how models complete actual complex tasks using tools such as web search, filesystem, and terminal.
According to the announcement, Agent Arena gives models web search, filesystem, and terminal (bash) tools to carry out real workflows including code writing, slide creation, web research, app building, and document analysis. Rather than a static benchmark closer to question-and-answer, it collects data from "live sessions where real users accomplish real tasks" and updates continuously. The platform is live at arena.ai, alongside a dedicated leaderboard and a technical blog.
In the snapshot at launch, OpenAI's GPT-5.5 (High) ranked first, Anthropic's Claude-Opus-4.7 (Thinking) second, Z.ai's GLM-5.1 third, Google DeepMind's Gemini-3.1-Pro fourth, and Moonshot's Kimi-K2.6 fifth (source).
The effort responds to an evaluation gap in the agent era. The Chatbot Arena that anchored the LMSYS era was a static, QA-centric human-preference benchmark, criticized as insufficient for today's workloads involving tool use, long-horizon tasks, user iteration, and error recovery (source). Arena aims to close that gap with dynamic data drawn from real users' production workloads. Related efforts include the older Gorilla-based agent-arena.com, Microsoft Windows Agent Arena, and xlang-ai's Computer Agent Arena, but Arena positions "live signals at the scale of millions of sessions" as its differentiator.
The scale was detailed concretely. At launch, more than 300K tasks, over 2M tool calls, and 40M lines of code generated by agents were recorded, with analysis examples drawn from over 160K real user tasks. Over a seven-day window, bash was called 936K times, write_file 550K times, and web_search 276K times, while generated code reached 8.5M lines of .py and 7.8M lines of .md (source). Rankings combine signals such as task success, steerability, error recovery, user praise versus complaints, and tool hallucination via causal inference, covering long-horizon workflows spanning 50+ turns and hundreds of tool calls. Anyone can try frontier models such as GPT-5.5 and Claude Opus 4.7 with tools via Agent Mode, with cost-performance Pareto analysis also provided.
Reaction on X was broadly positive, with many calling it "the most important real-world eval" and "more reliable than static benchmarks" (source). There were congratulations to the OpenAI team for the top spot and remarks that, compared with text leaderboards, it had "finally become an accurate leaderboard," praising real use cases like full-stack app building, financial modeling, real estate data analysis, and research document creation, as well as multi-day, hundreds-of-turns long-horizon tasks (source). Some were skeptical, citing weakness on edge cases and infrequent tasks and arguing that "live sessions alone are not enough." Arena says it plans further analysis of harness effects (a Claude-Code-like execution environment), and the debate continues.