AI Agent Reliability: Test Wrong Decisions, Not Just Completed Tasks

YPO Technology Network AI Brief · Nov 10, 2026 · 7m

Visa's new AI shopping benchmark found agents that failed usually still chose to buy. Stephen Forte explains why CEOs should test agents on the cases where the right answer is to decline or ask, and why memory kept outside the tool matters when a vendor update breaks.

In this episode:

Visa's Agentic Commerce Benchmark. Twenty-two models from twelve labs ran fifteen shopping scenarios three times each, for nine hundred and ninety runs. Overall pass rate fifty-seven point two percent; only thirty-four point two percent of model and scenario pairs passed all three attempts.

Caveats that matter…