skinny.

Latest / DataTalks.Club

Building AI Evaluations You Can Actually Trust - Tejas Kumar

Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.

CHAPTERS

00:00 Podcast intro

00:08 Tejas's path to AI engineering

12:24 Why evals go beyond a manual check

17:37 Sourcing evaluation cases and calibrating a judge

28:33 Refreshing datasets and pinning severe cases

33:10 Judge agreement versus agent performance

50:01 Retraining judges and keeping data fresh

55:59 Human preferences, sample sizes and audience questions

59:46 Evaluating final answers versus the agent's…

The skinny

The skinny isn't ready yet — notes appear once the transcript is processed.