
Building AI Evaluations You Can Actually Trust - Tejas Kumar
Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.
CHAPTERS
00:00 Podcast intro
00:08 Tejas's path to AI engineering
12:24 Why evals go beyond a manual check
17:37 Sourcing evaluation cases and calibrating a judge
28:33 Refreshing datasets and pinning severe cases
33:10 Judge agreement versus agent performance
50:01 Retraining judges and keeping data fresh
55:59 Human preferences, sample sizes and audience questions
59:46 Evaluating final answers versus the agent's…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.