Wei Sun of Purdue University : From LLM Evaluation to Ranking: Low-Rank Inference for Reliable LLM Leaderboards

Posted by Ping-Shou Zhong , part of the Statistics and Data Science Seminar.

At

Oct. 7, 2026, 4:30 p.m.

In

636 SEO

Abstract

Modern LLM evaluation increasingly relies on sparse and heterogeneous pairwise human preferences, making reliable estimation and uncertainty quantification challenging. In this talk, I will present a low-rank statistical framework that shares information across related tasks, models, and evaluation settings while preserving task-specific heterogeneity. I will first formulate LLM evaluation as a structured tensor-completion problem under pairwise comparisons and develop semiparametrically efficient inference for model abilities, score gaps, and other smooth functionals. A key technical ingredient is a score-whitening approach that handles the anisotropic Fisher information induced by non-uniform comparisons. I will then turn from score estimation to task-specific ranking, where ranks and top-K membership are nonsmooth functions of many dependent score gaps. We develop entrywise-accurate low-rank estimation, efficient debiased inference, and multiplier-bootstrap procedures that provide simultaneous rank intervals and uncertainty-aware top-K certificates. Together, these results provide statistical foundations for moving from sparse human feedback to reliable, task-specific LLM evaluation and ranking.