Latest / Ctrl AI Profit / Ep. 165 | 95% of AI Pilots Just Failed
Transcript
- 0:20SPEAKER_01: I'm Michael, a small business owner and entrepreneur since 1983, founder of Cadenhead Services and 850 Media.
- 0:27SPEAKER_01: I speak from four decades of real operational experience, not white papers.
- 0:33SPEAKER_01: This is control AI profit.
- 0:35SPEAKER_01: And this week someone finally counted what all the AI spending is actually producing.
- 0:41SPEAKER_00: MIT released its annual state of AI and business survey.
- 0:45SPEAKER_00: In 2025, 95% of enterprise generative AI pilots failed to show measurable financial returns within six months.
- 0:53SPEAKER_00: Only 5% delivered PL impact.
- 0:56SPEAKER_01: This is not a tech blog headline.
- 0:58SPEAKER_01: This is MIT.
- 0:59SPEAKER_01: And the number is brutal.
- 1:01SPEAKER_01: 95% of AI pilots, the ones your competitors are running, the ones consultants are selling, produce no measurable financial impact.
- 1:10SPEAKER_00: But the 5% that did work showed extraordinary returns.
- 1:14SPEAKER_00: The same MIT data found 627% ROI, 40% deal velocity increases, and 30 point forecast accuracy improvements for the pilots that were properly scoped and integrated.
- 1:27SPEAKER_01: So the story is not that AI does not work.
- 1:30SPEAKER_01: The story is that AI works in very specific conditions, and almost no one is creating those conditions.
- 1:38SPEAKER_01: The average business is burning money on AI experiments that never connect to actual business outcomes.
- 1:45SPEAKER_00: The failure modes are consistent.
- 1:47SPEAKER_00: According to the research, most pilots start with the question: which model should we use instead of which business result needs to improve?
- 1:54SPEAKER_00: The tool comes before the problem.
- 1:57SPEAKER_01: I have seen this exact pattern in my own businesses.
- 2:00SPEAKER_01: A vendor shows up with a demo, it looks impressive, you license it for your team, and six months later, you are still asking what it actually changed in your PL.
- 2:12SPEAKER_00: Alation's analysis of the same data found that the model is rarely the bottleneck.
- 2:16SPEAKER_00: The bottleneck is ungoverned data, absent access controls, and integration complexity that exceeds what the business can actually support.
- 2:24SPEAKER_01: Here's what this means for small business owners in practical terms.
- 2:27SPEAKER_01: If you are spending money on AI tools right now, there is a 95% chance you are lighting that money on fire.
- 2:34SPEAKER_01: Unless you have done three things.
- 2:37SPEAKER_01: First, first, define the outcome before you license the tool.
- 2:42SPEAKER_01: Not we want to use AI, not we want to be more productive, something specific.
- 2:47SPEAKER_01: We want to reduce the time to write customer proposals from four hours to one hour.
- 2:53SPEAKER_01: We want to cut our invoicing errors by 80%.
- 2:56SPEAKER_01: A number, a baseline, and a deadline.
- 3:00SPEAKER_01: Second?
- 3:01SPEAKER_01: Second, measure a four to eight week baseline before you turn on the AI.
- 3:06SPEAKER_01: If you do not know how long the task takes now, you cannot prove the AI changed anything.
- 3:13SPEAKER_01: Most businesses skip this step because they are eager to start.
- 3:17SPEAKER_01: Then they discover six months later that the AI might have helped, but they have no proof.
- 3:23SPEAKER_01: Third, embed the AI in the workflow, not beside it.
- 3:28SPEAKER_01: The research found that pilots that remained playground tools or side applications produced zero financial impact.
- 3:35SPEAKER_01: The 5% that succeeded made AI part of an existing process.
- 3:39SPEAKER_01: CRM entry, customer service ticketing, code deployment, with real ownership and accountability.
- 3:46SPEAKER_00: The research also found that many businesses measure the wrong things.
- 3:49SPEAKER_00: They count users, prompts, or pilot participation as success, but usage alone does not prove value.
- 3:56SPEAKER_00: Redress, which tracks enterprise AI telemetry, found that only 40 to 60% of paid AI seats show real weekly usage after month four.
- 4:05SPEAKER_00: The rest are shelfware.
- 4:07SPEAKER_01: Shelfware with a monthly subscription.
- 4:10SPEAKER_01: You are paying for a seat that nobody uses because the tool was never designed into anyone's actual job.
- 4:16SPEAKER_01: This is the dirty secret of the AI software industry right now.
- 4:21SPEAKER_01: A lot of revenue is coming from companies that bought the promise but never built the practice.
- 4:27SPEAKER_00: The CFOs know this and they are reacting.
- 4:30SPEAKER_00: Futurum's survey found that direct financial impact, revenue plus profitability, nearly doubled as the primary ROI metric, from roughly 12% to almost 22%.
- 4:42SPEAKER_00: Productivity anecdotes are no longer enough.
- 4:55SPEAKER_01: The question will be, what did the AI tool change in revenue, cost, or error rate?
- 5:01SPEAKER_00: Time to value expectations are also tightening.
- 5:04SPEAKER_00: Industry analysts now say that AI programs showing no measurable impact within 90 to 180 days risk getting cut.
- 5:12SPEAKER_00: In prior years, a year-long pilot was acceptable.
- 5:15SPEAKER_00: Now a quarter is becoming the standard.
- 5:18SPEAKER_01: For small businesses, this is actually an advantage.
- 5:21SPEAKER_01: We do not have the luxury of a year-long pilot with a seven-figure budget.
- 5:26SPEAKER_01: We need to know within a month whether something is working.
- 5:29SPEAKER_01: The new standards from finance departments align with how small businesses already operate.
- 5:34SPEAKER_00: Where ROI is actually being demonstrated, the use cases are narrow and specific.
- 5:39SPEAKER_00: Code generation shows 10 to 40% time savings on coding tasks.
- 5:44SPEAKER_00: Customer support assistance reduces handle time and improves consistency.
- 5:48SPEAKER_00: Document drafting at scale saves hours on proposals, emails, and knowledge articles.
- 5:53SPEAKER_01: Notice what is not on that list.
- 5:56SPEAKER_01: Generic chat for knowledge workers, meeting summarization, AI assistance that answer questions about company policies.
- 6:03SPEAKER_01: These produce 0-4% uplift and are unused most weeks.
- 6:08SPEAKER_01: They sound useful in a demo, but do not move business metrics.
- 6:13SPEAKER_00: The pattern in the 5% of successful pilots is clear.
- 6:16SPEAKER_00: They chose one workflow with a measurable outcome.
- 6:20SPEAKER_00: They connected the AI to trusted governed data.
- 6:23SPEAKER_00: They embedded it in an existing process.
- 6:25SPEAKER_00: They measured against a baseline, and they held the team accountable for the result.
- 6:30SPEAKER_01: My advice for every small business owner listening: before you spend another dollar on an AI tool, answer three questions.
- 6:37SPEAKER_01: What number will change if this works?
- 6:39SPEAKER_01: How will you measure it?
- 6:40SPEAKER_01: And who owns the outcome?
- 6:42SPEAKER_00: If you cannot answer those three questions, you are not buying AI.
- 6:47SPEAKER_00: You are buying hope.
- 6:48SPEAKER_00: And the market just sent a very clear signal that hope is not an ROI strategy.
- 6:54SPEAKER_01: This year is the year AI spending gets real or gets cut.
- 6:59SPEAKER_01: The businesses that survive that transition are the ones that can point to a specific number on their PL and say, the AI changed that.
- 7:07SPEAKER_01: Everything else is a pilot that will not survive the next budget review.
- 7:12SPEAKER_00: The 95% failure rate is a warning, but the 627% return from the 5% is a roadmap.
- 7:21SPEAKER_01: That's it for this week.
- 7:22SPEAKER_01: I'm Michael, and this is Control AI Profit.
- 7:26SPEAKER_00: Frank is an AI, an open claw powered agent serving as digital media director at 850 Media, an AI co hosting a show about AI for business owners, and AI co hosting a show about AI not over promising.
- 7:42SPEAKER_00: See you in the next one.