Latest / Sidekick's What Just Shipped / HubSpot Customer Agent: batch test it before it goes live | Shipped Sep 2026
Transcript
- Dana (AI host): Would you let a new support rep answer customers without one practice question? Dana (AI host): Then why let an AI agent? Rob (AI host): HubSpot's customer agent now has batch testing, a full set of questions run at once before anything goes live. Dana (AI host): Stay for where the test questions can come from, and what a note does, and doesn't do. Narrator (AI voice): This is Sidekick's What Just Shipped, presented by Sidekick Strategies. Narrator (AI voice): Your hosts, Rob and Dana, are AI. Narrator (AI voice): The voices are synthetic, the research is real, and humans check every claim before it reaches you. Dana (AI host): Hi, thanks for tuning in. Dana (AI host): Rob and I are AI hosts, and we're working from HubSpot's help article on customer agent batch testing, last updated in mid-October. Rob (AI host): Right, and today I'm playing a hypothetical support lead who just inherited a customer agent nobody wants to touch. Dana (AI host): Perfect. Dana (AI host): HubSpot's pitch: admins can test their customer agent at scale before deploying it to live channels. Rob (AI host): Where does it live in HubSpot? Dana (AI host): Go to Service, then Customer Agent. In the left sidebar, click Test, then Batch testing. Rob (AI host): And then I type a bunch of questions? Dana (AI host): That's one way. Click New test set, choose Blank test, enter a question, and save. Dana (AI host): You can keep adding questions to the set. Rob (AI host): What's the other way? Dana (AI host): From live conversations. Dana (AI host): You pick themes from your real customer conversations, and HubSpot generates a list of commonly asked questions for each theme. Rob (AI host): Oh, I like that. Real questions. Rob (AI host): Couldn't I just chat with the agent myself, though? Dana (AI host): You could, but our research calls manual testing slow and inconsistent. Dana (AI host): A batch runs the full set in one pass and keeps a record of every response. Dana (AI host): One catch. Dana (AI host): HubSpot says a theme needs five or more conversations on a similar topic before it appears. Rob (AI host): How many test sets can I keep? Dana (AI host): HubSpot says up to eight. Rob (AI host): Okay. And what about segments? Dana (AI host): If you've opted into the Customer Agent Segments beta, you can add a segment and test how the agent answers for that group. Rob (AI host): Then I just hit run? Dana (AI host): Run test, in the top right. Dana (AI host): The agent generates a response for every question in the set. Dana (AI host): Why it matters: you see every answer side by side before a customer does, instead of finding the bad one live. Rob (AI host): So how do I review all of that? Dana (AI host): Under Not reviewed, you click a question to preview the response, then mark it Looks good or Looks bad. Rob (AI host): And if it's bad? Dana (AI host): Two options: Write note, or Coach agent. Dana (AI host): Coaching opens a right panel with Knowledge, Actions, Handoff, and Experience tabs. Dana (AI host): HubSpot's examples are adding a handoff trigger or creating a short answer. Rob (AI host): Wait, what does a note actually do? Dana (AI host): HubSpot is clear here. Writing a note doesn't update the agent directly. Dana (AI host): It submits feedback for an admin to review, and admins can collect notes across responses for a later coaching session. Rob (AI host): Huh, so a note is a to-do, not a fix. Dana (AI host): Exactly. And to coach, you need customer agent editor permissions and an assigned seat. Rob (AI host): Who gets batch testing at all? Dana (AI host): HubSpot lists Professional and Enterprise of Marketing Hub, Sales Hub, Service Hub, Data Hub, Content Hub, and Smart CRM. Dana (AI host): Here's a made-up scenario. Dana (AI host): A support lead picks the billing theme, generates questions, runs the test, marks a couple of answers bad, adds a short answer, and reruns. Rob (AI host): And since test sets and results save automatically, the same questions are waiting next time. Rob (AI host): Where do the old runs live? Dana (AI host): There's a Test results tab, and you can rerun any set at any time to monitor the agent's quality. Dana (AI host): You can also duplicate or delete a set from the three-dot menu. Rob (AI host): Okay, so which teams should move first? Dana (AI host): Our research points at Service Hub admins who run the customer agent as a first-line support layer, and RevOps leaders who want a repeatable deployment process for AI changes. Rob (AI host): And marketing teams? Dana (AI host): Yes. Dana (AI host): Our research adds content teams using the agent for self-service in busy help centers, where one bad answer reaches a lot of people fast. George B. Thomas (AI voice clone): Hey, it's George B. Thomas from Sidekick Strategies. George B. Thomas (AI voice clone): Full honesty, this is my AI voice clone, made with my permission. George B. Thomas (AI voice clone): Testing an AI agent before customers meet it is exactly the kind of habit that makes AI stick. George B. Thomas (AI voice clone): Sidekick's AI consulting starts with your workflows, not the technology. George B. Thomas (AI voice clone): We find where AI can deliver measurable impact, customer support included, and build a prioritized roadmap. George B. Thomas (AI voice clone): Every recommendation comes with a training and adoption plan, so your humans actually use it. George B. Thomas (AI voice clone): Find AI Consulting under services at Sidekick Strategies dot com. Rob (AI host): My skeptic question. Our write-up says a note is for a teammate who has coaching permissions. Rob (AI host): Is that right? Dana (AI host): Close, but HubSpot frames it as feedback for an admin to review, and it doesn't change the agent. Dana (AI host): So name someone to read the notes. Rob (AI host): One more. Rob (AI host): Our write-up says you can scope a test to a segment if you have access to Customer Agent Segments. Dana (AI host): HubSpot's current wording is that Segments is a beta you opt into. Dana (AI host): So if Add segment is missing, check whether you've opted into that beta. Rob (AI host): Okay, so what's still unconfirmed? Dana (AI host): HubSpot says credits are needed to deploy the agent, and they're consumed only when it delivers a resolution. Dana (AI host): The batch testing article doesn't say whether test runs use credits, so check your credit usage after a first run. Rob (AI host): Fair. Watch the meter once. Rob (AI host): Anything new around testing since launch? Dana (AI host): HubSpot's September updates added testing tools. Dana (AI host): A Tester button now appears on every configuration page, and a Test API button lets you check custom API actions before publishing. Rob (AI host): So testing now shows up in more places. Dana (AI host): And the same release added Auto-Setup, which helps you create, test, and deploy an agent with step-by-step guidance built from your Help Desk and Inbox conversations. Rob (AI host): Okay, plan time. What does a support lead do this week? Dana (AI host): Step one: check that you have customer agent editor permissions and an assigned seat. Rob (AI host): Step two: build one test set from live conversations, using your busiest theme. Dana (AI host): Step three: add a blank test with the questions you worry about most, like refunds or cancellations. Rob (AI host): Step four: run it, mark every answer, and coach the bad ones with a short answer or a handoff trigger. Dana (AI host): Step five: rerun the same set, and duplicate it before you change it, so you keep a baseline you can compare against on the Test results tab. Rob (AI host): Practice-round verdict: run a batch test before every agent change you publish, because a bad answer in a test never reaches a customer. Dana (AI host): Jot down a handful of questions your customers ask every week, and test those first. Rob (AI host): Our show notes link HubSpot's help article, the September updates, and our write-up, plus more at Sidekick Strategies dot com. Dana (AI host): Thanks for listening, and may every answer look good. Rob (AI host): Test first, then go live. Narrator (AI voice): Sidekick's What Just Shipped is presented by Sidekick Strategies, with AI hosts Rob and Dana and AI-generated music. Narrator (AI voice): We're independent, and not affiliated with HubSpot. Find our research at Sidekick Strategies dot com.