Latest / Uphoff on Media Podcast / Your AI Pilot Doesn’t Have a Job

Transcript

Your AI Pilot Doesn’t Have a Job · Uphoff on Media Podcast

• 0:01
  SPEAKER_00: Hey, it's Tony.
• 0:02
  SPEAKER_00: Few business purchases carry more risk than technology.
• 0:07
  SPEAKER_00: Failed implementations, abandoned projects, damaged careers.
• 0:12
  SPEAKER_00: Last year, 42% of companies abandoned most of their AI initiatives, up from 17% the year before.
• 0:20
  SPEAKER_00: And yet, we keep making the same mistake over and over.
• 0:24
  SPEAKER_00: You know how this starts.
• 0:25
  SPEAKER_00: The demo makes the technology look transformative.
• 0:28
  SPEAKER_00: A competitor has already rolled it out.
• 0:30
  SPEAKER_00: Every peer you talk to is excited about the same new thing.
• 0:34
  SPEAKER_00: Nobody wants to be the one who waited.
• 0:37
  SPEAKER_00: But the problem is rarely the technology.
• 0:39
  SPEAKER_00: It's that nobody wrote down what the technology was hired to do.
• 0:43
  SPEAKER_00: What's the specific job?
• 0:44
  SPEAKER_00: Who does that job today and how well?
• 0:47
  SPEAKER_00: And what will you measure in 90 days to know it's working?
• 0:51
  SPEAKER_00: In this episode titled Your AI Pilot Doesn't Have a Job,
• 0:55
  SPEAKER_00: I dig into why we keep falling into this trap and I walk through
• 0:58
  SPEAKER_00: the application test.
• 1:00
  SPEAKER_00: Three gates every technology pitch should clear before a dollar moves.
• 1:05
  SPEAKER_00: Okay, let's get to work.
• 1:08
  SPEAKER_00: Somewhere in your organization right now,
• 1:11
  SPEAKER_00: a team is piloting technology nobody can explain the job of.
• 1:16
  SPEAKER_00: They can describe the vendor.
• 1:17
  SPEAKER_00: They can describe the demo that got everybody nodding.
• 1:20
  SPEAKER_00: Ask them what specific job that technology was hired to do and watch the pause.
• 1:25
  SPEAKER_00: That pause is the whole problem.
• 1:27
  SPEAKER_00: Most AI pilots that fail were never given a job.
• 1:31
  SPEAKER_00: Nobody wrote down the job description the technology was hired to do.
• 1:34
  SPEAKER_00: Who does that work today?
• 1:36
  SPEAKER_00: And what would prove it got done better?
• 1:39
  SPEAKER_00: Writing it down costs nothing.
• 1:40
  SPEAKER_00: Skipping it costs the pilot.
• 1:44
  SPEAKER_00: Procurement gave this mistake a name.
• 1:46
  SPEAKER_00: It doesn't stay in procurement.
• 1:48
  SPEAKER_00: On a recent episode of the Art of Procurement podcast,
• 1:52
  SPEAKER_00: Darshan Deshmuk,
• 1:54
  SPEAKER_00: president of Procurability,
• 1:55
  SPEAKER_00: gave this pattern its name, quote, death by a thousand pilots, end quote.
• 2:00
  SPEAKER_00: His diagnosis was just as plain.
• 2:03
  SPEAKER_00: Organizations keep leading with the technology instead of the outcome.
• 2:07
  SPEAKER_00: Philip Ideson,
• 2:08
  SPEAKER_00: who co-founded Art of Procurement with Kelly Varner and built it into
• 2:11
  SPEAKER_00: the source procurement leaders actually rely on,
• 2:14
  SPEAKER_00: put the consequence best in his write-up of the episode, quote,
• 2:18
  SPEAKER_00: a compelling platform or use case can be a source of inspiration,
• 2:21
  SPEAKER_00: but it cannot define the destination, end quote.
• 2:26
  SPEAKER_00: None of this is new.
• 2:27
  SPEAKER_00: Procurement made this mistake with ERP,
• 2:30
  SPEAKER_00: then with eSourcing, and every other function made it with CRM.
• 2:34
  SPEAKER_00: Same mistake, different software.
• 2:37
  SPEAKER_00: The ERP era supplied a half billion euro example.
• 2:41
  SPEAKER_00: In 2018,
• 2:43
  SPEAKER_00: Lidl walked away from a seven-year SAP project
• 2:46
  SPEAKER_00: after spending roughly 500 million euros on it, according to Handel's plot.
• 2:53
  SPEAKER_00: The sticking point was not complex.
• 2:55
  SPEAKER_00: Lytle valued its inventory at purchase price.
• 2:57
  SPEAKER_00: The software was built around retail price.
• 3:00
  SPEAKER_00: In my read,
• 3:01
  SPEAKER_00: nobody settled up front which way the business actually needed that job done,
• 3:05
  SPEAKER_00: and hundreds of consultants spent years trying to bend one to fit the other.
• 3:10
  SPEAKER_00: Look, I've never run a procurement organization,
• 3:13
  SPEAKER_00: but I have spent three decades running Media,
• 3:16
  SPEAKER_00: marketing,
• 3:16
  SPEAKER_00: and technology businesses, and every one of them had its own version of this.
• 3:21
  SPEAKER_00: The CRM migration that died in month four,
• 3:24
  SPEAKER_00: the MarTech stack nobody fully turned on,
• 3:26
  SPEAKER_00: and now the AI co-pilot every vendor in your inbox is pitching your team.
• 3:32
  SPEAKER_00: The MarTech version is measurable.
• 3:34
  SPEAKER_00: Gartner found marketers used 58% of their MarTech stacks capabilities in 2020.
• 3:41
  SPEAKER_00: Down to 42% in 2022 and down to 33% in 2023.
• 3:47
  SPEAKER_00: The spend kept going up, the use kept going down.
• 3:51
  SPEAKER_00: The failure point is identical everywhere it shows up.
• 3:54
  SPEAKER_00: The technology got approved, the job never got defined.
• 3:59
  SPEAKER_00: Why do most AI projects fail?
• 4:02
  SPEAKER_00: Well, it's rarely because of the technology.
• 4:05
  SPEAKER_00: The failure rate for this generation of pilots is well documented.
• 4:08
  SPEAKER_00: S&P Global Market Intelligence found that 42% of companies abandoned most
• 4:14
  SPEAKER_00: of their AI initiatives in 2025.
• 4:16
  SPEAKER_00: And that's up from 17% the year before.
• 4:19
  SPEAKER_00: And the average company scrapped 46% of its proof of concept before they reached production.
• 4:28
  SPEAKER_00: The more useful finding is why.
• 4:30
  SPEAKER_00: When BCG surveyed a thousand senior executives in 2024,
• 4:35
  SPEAKER_00: it attributed roughly 70% of the challenges in getting value from AI to people
• 4:40
  SPEAKER_00: and process, 20% to technology and only 10% to the algorithms themselves.
• 4:47
  SPEAKER_00: The part every demo is built to sell is the smallest slice of the problem.
• 4:52
  SPEAKER_00: Now, look at the budget for your last pilot.
• 4:55
  SPEAKER_00: If most of it went to the vendor, you funded the 30% and left the 70% to chance.
• 5:02
  SPEAKER_00: McKinsey's August 2026 State of AI survey points the same direction from the other end.
• 5:09
  SPEAKER_00: The companies getting real earnings impact from AI are a small group, about 6% of respondents.
• 5:15
  SPEAKER_00: Nearly three quarters of them say they have fundamentally redesigned workflows around it.
• 5:21
  SPEAKER_00: Among everyone else, the figure is about a quarter.
• 5:23
  SPEAKER_00: The winners started with the work.
• 5:25
  SPEAKER_00: Everyone else started with the tool.
• 5:29
  SPEAKER_00: The business case already exists, it was just built backwards.
• 5:32
  SPEAKER_00: Almost every one of those pilots cleared some kind of approval process.
• 5:36
  SPEAKER_00: Somebody wrote a business case,
• 5:38
  SPEAKER_00: there were ROI projections,
• 5:39
  SPEAKER_00: a vendor comparison matrix, maybe a proof of concept demo that went well.
• 5:46
  SPEAKER_00: Usually an absence of diligence problem.
• 5:48
  SPEAKER_00: It's a diligence aimed at the wrong target problem.
• 5:52
  SPEAKER_00: The business case documented what the technology could do.
• 5:55
  SPEAKER_00: It rarely documented in plain words a non-technical person in
• 5:59
  SPEAKER_00: the business could repeat back accurately the job
• 6:02
  SPEAKER_00: the organization actually needed done.
• 6:04
  SPEAKER_00: Capability and job are different questions.
• 6:07
  SPEAKER_00: Most business cases answer the first and assume the second.
• 6:12
  SPEAKER_00: Pilots multiply for another reason nobody puts in the business case.
• 6:16
  SPEAKER_00: A pilot is the one way to say yes to the vendor and no to accountability in the same meeting.
• 6:22
  SPEAKER_00: Nobody's name is really on it.
• 6:24
  SPEAKER_00: Nobody set a date to decide.
• 6:25
  SPEAKER_00: It costs less than the argument about killing it would.
• 6:29
  SPEAKER_00: Death by a thousand pilots is death by a thousand decisions nobody made.
• 6:36
  SPEAKER_00: A job is not a use case.
• 6:38
  SPEAKER_00: I made a version of this argument before about customers rather
• 6:42
  SPEAKER_00: than internal technology decisions.
• 6:45
  SPEAKER_00: When I was brought into transformthomasnet.com,
• 6:48
  SPEAKER_00: my team thought we understood the dozen
• 6:50
  SPEAKER_00: or two things Things our customers were trying to accomplish on the platform.
• 6:54
  SPEAKER_00: The real number once we did the research was over 350.
• 6:58
  SPEAKER_00: We had been building for what we assumed the job was, not what it actually was.
• 7:03
  SPEAKER_00: I wrote about what that discovery did to the company and to my own thinking in
• 7:07
  SPEAKER_00: the post, you don't know what your customers are actually trying to do.
• 7:11
  SPEAKER_00: The same blind spot shows up on the buy side and it may be more dangerous there
• 7:16
  SPEAKER_00: because nobody outside your own walls is gonna do that research for you.
• 7:21
  SPEAKER_00: A vendor's use case library describes what other companies did with the tool.
• 7:25
  SPEAKER_00: It does not describe the job inside your organization
• 7:29
  SPEAKER_00: under your specific circumstances this technology is being asked to do.
• 7:33
  SPEAKER_00: If you cannot state that job without naming the technology,
• 7:37
  SPEAKER_00: you haven't defined the job, you've named a category.
• 7:41
  SPEAKER_00: A better demo makes the mistake easier to make.
• 7:44
  SPEAKER_00: An ERP demo in 2004 was a slide deck and a promise.
• 7:48
  SPEAKER_00: An AI demo in 2026 is the product working on your own data,
• 7:53
  SPEAKER_00: in your own language, in front of the people who will approve it.
• 7:56
  SPEAKER_00: The gap between the demo and the job has never looked smaller.
• 8:00
  SPEAKER_00: It has rarely been larger.
• 8:02
  SPEAKER_00: A demo shows one job done well under conditions the vendor chose.
• 8:07
  SPEAKER_00: Your organization won hundreds of jobs under conditions nobody chose.
• 8:12
  SPEAKER_00: That was the lesson of the 350 at ThomasNet, and no demo is built to show it to you.
• 8:17
  SPEAKER_00: A good demo makes defining the job feel unnecessary.
• 8:21
  SPEAKER_00: That is what it was built to do.
• 8:23
  SPEAKER_00: So I have a rule, and vendors don't love it.
• 8:25
  SPEAKER_00: No demo until the job is written down.
• 8:29
  SPEAKER_00: Two companies wrote two different sentences.
• 8:33
  SPEAKER_00: Klarna is the clearest example of what happens when the job gets written wrong.
• 8:37
  SPEAKER_00: In early 2024,
• 8:38
  SPEAKER_00: the company said its AI assistant was doing the work of 700 customer service agents
• 8:44
  SPEAKER_00: and handling two-thirds of its customer chats.
• 8:47
  SPEAKER_00: By May 2025,
• 8:48
  SPEAKER_00: CEO Sebastian Samarkowski told Bloomberg the company was investing
• 8:52
  SPEAKER_00: in human support again.
• 8:54
  SPEAKER_00: His explanation was unusually candid, quote,
• 8:57
  SPEAKER_00: as cost unfortunately seems to have been a too predominant evaluation factor
• 9:01
  SPEAKER_00: when organizing this, what you end up having is lower quality, end quote.
• 9:06
  SPEAKER_00: The job got written as cut the cost of support.
• 9:10
  SPEAKER_00: The job customers needed done was get my problem solved.
• 9:14
  SPEAKER_00: The pilot did the first job and failed the second.
• 9:17
  SPEAKER_00: Morgan Stanley did the opposite.
• 9:19
  SPEAKER_00: Before rolling out its GPT-4 assistant to wealth management, the firm started with a narrow job.
• 9:27
  SPEAKER_00: Help a financial advisor find the right answer fast in a library
• 9:31
  SPEAKER_00: of roughly 100,000 research documents.
• 9:34
  SPEAKER_00: Every use case was tested against real advisor tasks with advisors grading
• 9:39
  SPEAKER_00: the output before it shipped.
• 9:42
  SPEAKER_00: According to OpenAI's case study, more than 98% of advisor teams now use it.
• 9:47
  SPEAKER_00: Same class of technology as the pilots in the S&P numbers.
• 9:51
  SPEAKER_00: The difference was a job defined before the purchase order was sent.
• 9:56
  SPEAKER_00: I wrote the jobs down before I hired a developer.
• 9:59
  SPEAKER_00: When I set out to build the decision tools I use in my advisory work, the easy path was obvious.
• 10:06
  SPEAKER_00: Pick a fast AI development platform, sketch a rough concept, and let a good demo carry the rest.
• 10:13
  SPEAKER_00: I did the opposite.
• 10:14
  SPEAKER_00: Before I hired a developer, I wrote out the job to be done for each tool.
• 10:19
  SPEAKER_00: The valuation tools job,
• 10:21
  SPEAKER_00: when a CEO is weighing a major decision,
• 10:23
  SPEAKER_00: show what that decision does to the value of the business and on what timeline,
• 10:27
  SPEAKER_00: not calculate what the company is worth.
• 10:30
  SPEAKER_00: That distinction decided what got built.
• 10:33
  SPEAKER_00: A valuation calculator answers a question once.
• 10:36
  SPEAKER_00: A decision tool gets checked before every call.
• 10:38
  SPEAKER_00: So it needed drivers an operator can actually move benchmark against real trading multiples.
• 10:45
  SPEAKER_00: The talent tools job When a manager has to hire, coach,
• 10:49
  SPEAKER_00: or restructure a sales or marketing team,
• 10:51
  SPEAKER_00: help them make that call on evidence,
• 10:53
  SPEAKER_00: starting with telling a strong resume from a strong hire,
• 10:56
  SPEAKER_00: writing that down without the obvious product and employee tracking system.
• 11:01
  SPEAKER_00: The job is the decision, not the monitoring.
• 11:04
  SPEAKER_00: The technology,
• 11:05
  SPEAKER_00: the specific platform,
• 11:06
  SPEAKER_00: the build sequence all came after those jobs were written up, not before.
• 11:11
  SPEAKER_00: That ordering is the entire difference between a tool people keep opening and one
• 11:15
  SPEAKER_00: that stops getting used after the second demo.
• 11:21
  SPEAKER_00: Agentic AI removed the friction that used to slow this mistake down.
• 11:25
  SPEAKER_00: The old version of this mistake had natural speed bumps built in.
• 11:30
  SPEAKER_00: Long procurement cycles, an IT approval queue, a budget committee that met quarterly.
• 11:35
  SPEAKER_00: A bad idea had time to die on its own before real money moved.
• 11:40
  SPEAKER_00: Agentic AI tools mostly skip all of that.
• 11:43
  SPEAKER_00: A capable pilot can now show up as a subscription one manager puts on a corporate card.
• 11:48
  SPEAKER_00: No committees required.
• 11:50
  SPEAKER_00: Zylo's 2026 SaaS Management Index found expensed software spend up 267% in a year,
• 11:59
  SPEAKER_00: with ChatGBT now the single most expensed application and business units,
• 12:03
  SPEAKER_00: not IT, controlling 81% of software spend.
• 12:07
  SPEAKER_00: Gardner expects more than 40% of agentic AI projects to be canceled by the end
• 12:12
  SPEAKER_00: of 2027,
• 12:14
  SPEAKER_00: and its analysts describe most current projects as experiments,
• 12:18
  SPEAKER_00: quote, mostly driven by hype, end quote.
• 12:23
  SPEAKER_00: The old failure mode took a fiscal year to play out.
• 12:26
  SPEAKER_00: This one takes a sprint.
• 12:28
  SPEAKER_00: A 20-year-old mistake now compounds at the speed of a subscription click,
• 12:32
  SPEAKER_00: and most operators have not updated their discipline to match.
• 12:36
  SPEAKER_00: The instinct is to add process, a committee, a review, a form, that loses.
• 12:41
  SPEAKER_00: Governance that costs more than the subscription it governs gets routed around by
• 12:45
  SPEAKER_00: the same manager with the same card.
• 12:47
  SPEAKER_00: Whatever discipline you add has to be cheaper and faster than the pilot it gates.
• 12:52
  SPEAKER_00: That is why the test in the framework is three questions and not a 40-page deck.
• 12:59
  SPEAKER_00: Three gates cleared before the PO gets signed.
• 13:02
  SPEAKER_00: Here's the test I now run in every technology pitch that reaches my desk,
• 13:06
  SPEAKER_00: whether I'm advising a client or approving something inside my own business.
• 13:10
  SPEAKER_00: I call it the application test.
• 13:11
  SPEAKER_00: Three gates cleared in order before a dollar moves.
• 13:14
  SPEAKER_00: Number one, state the job.
• 13:17
  SPEAKER_00: In plain words,
• 13:18
  SPEAKER_00: a non-technical person in the business could repeat back accurately
• 13:21
  SPEAKER_00: with no vendor name and no product name in it.
• 13:24
  SPEAKER_00: Number two, name the owner in the baseline.
• 13:26
  SPEAKER_00: Who is doing that job today and how well?
• 13:29
  SPEAKER_00: In a number you already have, time, cost, error rate, or volume, a committee is not an owner.
• 13:36
  SPEAKER_00: And number three, set the 90-day signal.
• 13:38
  SPEAKER_00: The one measure you will check in 90 days that tells you the job is closer to done,
• 13:44
  SPEAKER_00: separate from whether the pilot is still running.
• 13:47
  SPEAKER_00: Logins, seats, and prompts measure the tool, not the job.
• 13:50
  SPEAKER_00: Decide in advance what you will do with the answer.
• 13:54
  SPEAKER_00: Scale it, fix it, or stop it.
• 13:57
  SPEAKER_00: Fail any gate and you do not have a business case.
• 13:59
  SPEAKER_00: You have a good demo and an open checkbook.
• 14:02
  SPEAKER_00: Klarna's pilot would have cleared the gate on cost and failed gate three
• 14:08
  SPEAKER_00: the first time someone checked whether
• 14:09
  SPEAKER_00: the customers were getting their problems solved.
• 14:14
  SPEAKER_00: The application test decides whether a pilot earns its first dollar.
• 14:18
  SPEAKER_00: Whether it earns the second is a different discipline.
• 14:20
  SPEAKER_00: The one I laid out in the post, your agentic AI isn't failing your math is.
• 14:25
  SPEAKER_00: Never fund the next stage without naming what it has to pay back.
• 14:33
  SPEAKER_00: BCG put the algorithms at a tenth of the problem and people and process at 70%.
• 14:40
  SPEAKER_00: In my experience, most of that 70% starts with the job nobody wrote down.
• 14:46
  SPEAKER_00: Here's the fastest way to find out whether that is true in your business.
• 14:50
  SPEAKER_00: Pull up the last three pilots you approved.
• 14:52
  SPEAKER_00: Write down the job for each one now after the fact with no vendor name in it.
• 14:57
  SPEAKER_00: If you can do it in under a minute, the pilot probably has a future.
• 15:00
  SPEAKER_00: If you cannot, you already know how it ends.
• 15:03
  SPEAKER_00: You just haven't admitted it yet.
• 15:06
  SPEAKER_00: Well, that's the episode.
• 15:07
  SPEAKER_00: If you want to put this to work,
• 15:08
  SPEAKER_00: the application test framework is in the post on Substack, linked in the show notes.
• 15:14
  SPEAKER_00: Save it and run it on the next technology pitch before the demo, not after.
• 15:18
  SPEAKER_00: And if you found this useful,
• 15:20
  SPEAKER_00: do one thing,
• 15:21
  SPEAKER_00: follow Uphoff on Media on Apple Podcasts or Spotify and leave a rating.
• 15:25
  SPEAKER_00: It's the single best way to help another operator find it.
• 15:29
  SPEAKER_00: I'm Tony Uphoff, back at it next week.