ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models — Elangovan et al. (2024)

1 More Paper · Oct 4, 2026 · 1h 13m · transcript

In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertak- ing that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the ex- perimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cog- nitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such…