Agreement Is Not Accuracy
A revision sweep through the cs.AI batch surfaced five independent papers pushing on the same thing: the instruments we use to decide whether a model is right are themselves unreliable — self-consistency, calibration error, count-based F1, and the model judges grading all of it.
Kaihua Ding audits self-consistency across 265,000 samples — agreement predicts correctness weakly, and worst on the most consistent frontier model.
Jon-Paul Cacioli separates accuracy from metacognitive sensitivity with Signal Detection Theory — and his v3 note retracts his own headline after 1,830 human…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.