Researchers from Inclusion AI, which is affiliated with Alibaba’s Ant Group, proposed Inclusion Arena, a new model leaderboard and benchmark that focuses more on a model’s performance in real-life scenarios. They argue that LLMs need a leaderboard that takes into account how people use them and how much people prefer their answers compared to the static knowledge capabilities models have. Inclusion Arena stands out among other model leaderboards, due to its real-life aspect and its unique method of ranking models. Inclusion Arena works by integrating the benchmark into AI applications to gather datasets and conduct human evaluations. Currently, there are two apps available on Inclusion Arena: the character chat app Joyland and the education communication app T-Box. When people use the apps, the prompts are sent to multiple LLMs behind the scenes for responses. The users then choose which answer they like best, though they don’t know which model generated the response. The framework considers user preferences to generate pairs of models for comparison. The Bradley-Terry algorithm is then used to calculate a score for each model, which then leads to the final leaderboard. According to the initial experiments with Inclusion Arena, the most performant model is Anthropic’s Claude 3.7 Sonnet, DeepSeek v3-0324, Claude 3.5 Sonnet, DeepSeek v3 and Qwen Max-0125. The Inclusion AI researchers argue that their new leaderboard “ensures evaluations reflect practical usage scenarios,” so enterprises have better information around models they plan to choose.