Latest / Agentic Conversations (formally mlops.community)
Everything Hard About Building AI Agents Today
Willem Pienaar and Shreya Shankar discuss the challenge of evaluating agents in production where "ground truth" is ambiguous and subjective user feedback isn't enough to improve performance.
The discussion breaks down the three "gulfs" of human-AI interaction—Specification, Generalization, and Comprehension—and their impact on agent success.
Willem and Shreya cover the necessity of moving the human "out of the loop" for feedback, creating faster learning cycles through implicit signals rather than direct, manual review. The conversation details practical evaluation techniques, including…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.