Latest / Excel at AI in Finance | CFO, FP&A, AI Tools, Claude, Microsoft Copilot / How Good Is AI in Finance, Actually? Manage It Like an Entry-Level FP&A Analyst.
Transcript
- 0:00There's a benchmark that tests AI models
- 0:02on entry-level financial analyst work, the stuff you'd hand someone in their first year.
- 0:06The best model on it gets about 6 in 10 right.
- 0:10Now, if I had an analyst getting 6 in 10 right, we'd be having a conversation
- 0:14about their future.
- 0:15And I still use AI every
- 0:17single day on real finance work.
- 0:20Because the more time I spend with it, the more it looks like the most average
- 0:23performer on the team.
- 0:25It does decent work, and it never stops needing to be managed.
- 0:28So today, I want to talk about
- 0:31managing AI like that, and the five-step
- 0:33setup I use to get the most out of it anyway.
- 0:36Welcome to Excel at AI in Finance, the show for finance pros who want to get home sooner and be
- 0:41the ones their company can't afford to lose.
- 0:43I'm Mike, I run F9 Finance, and I'm a senior
- 0:46finance leader working in the trenches.
- 0:49Each week, I share the AI tools, workflows, and mindsets that get you where you want to be.
- 0:53All right, let's get into it.
- 0:56If you believe the vendor demos or the AI companies, you can type in a prompt,
- 1:00push a button, and one-shot nearly
- 1:02any piece of work you want.
- 1:04But I've quickly seen in my own
- 1:07career and work that that doesn't happen.
- 1:09There is almost no process
- 1:12aside from writing a basic email where
- 1:15you can one-shot it with AI.
- 1:16I think my biggest learning was the first time
- 1:19that I attempted a forecast process.
- 1:21Now, I took a small forecast process.
- 1:22AI tools are still not great with
- 1:24all of that complexity.
- 1:26That's as we think about deterministic,
- 1:28where there's that one right answer, one plus one is two.
- 1:31And as you think about probabilistic,
- 1:33where it's making some of those judgment calls, it's still not good at doing that
- 1:37much deterministic versus
- 1:38probabilistic work as a forecast.
- 1:41But it was a small forecast.
- 1:42And I decided to one-shot it because that's what everyone was saying you could do.
- 1:46Put the prompt in, give it your data,
- 1:48and have it run the forecast.
- 1:49What came out was just about
- 1:51the biggest piece of crap you've ever seen.
- 1:54It was really no different than if I asked an intern to do a forecast for the first time.
- 1:59They're just going to throw a bunch of stuff in Excel, maybe use a technique
- 2:02they learned in college, and what you're going to get back is insane.
- 2:06So that's where I started realizing
- 2:09that there was a lot of work the AI could do.
- 2:12Because in that process, it did do really
- 2:14good work on curating the data.
- 2:16It did do really good work on finding
- 2:19outliers and exceptions.
- 2:20It was that across an entire process,
- 2:23it's start, get it out of whack.
- 2:25I didn't want to give up on the forecasting, so I started tweaking it.
- 2:29What can we do here to make this
- 2:30process run better?
- 2:32The first thing I did is I
- 2:34stopped having it one-shot.
- 2:36I started having it come back to
- 2:38me after major pieces of work so I
- 2:40could redirect if necessary.
- 2:43Go curate the data, organize it for the
- 2:45forecast, come back to me, and let it review.
- 2:48That was the first part.
- 2:49The second part was as we think about that deterministic,
- 2:52that one right answer versus the
- 2:54probabilistic, I started to break it down.
- 2:57Instead of using trend
- 2:59analysis or statistical analysis,
- 3:01I flipped it to driver-based forecasting,
- 3:04all built around formulas.
- 3:07So instead of this engine, this work being
- 3:09done all inside the AI tool, I had it being
- 3:12done inside of an Excel file with formulas
- 3:14where I could check every single number.
- 3:17And then the AI was setting the initial
- 3:20drivers for me for my final review,
- 3:22which became the second part of the gate.
- 3:25Now, every one of these adjustments,
- 3:27the more I walked away from one-shot,
- 3:29the more I became part of the process with the
- 3:32AI tool working side-by-side alongside me,
- 3:34that's where I started to see some phenomenal
- 3:37improvements in the performance.
- 3:39The last piece of it was this feedback loop,
- 3:42that every single time that it did
- 3:44something that I didn't like,
- 3:46I would provide the AI tool feedback.
- 3:48It's no different than giving that intern or that junior analyst on your team feedback.
- 3:52I gave the AI feedback, and I had it catalog
- 3:55it inside of a skill file and inside
- 3:58of my AI vault, the documentation where
- 4:00I keep context and references for AI.
- 4:03And the process just kept getting better
- 4:06and better over time, till it was 80%
- 4:08faster than me doing the work on my own.
- 4:11And the performance and the output was
- 4:13just as strong, if not stronger.
- 4:14And it's these same learnings I
- 4:16want to share with you today to get your AI
- 4:19tools away from being the most average performer on your team.
- 4:22Want to work with me? Then check out the Finance AI Lab.
- 4:25Get at least seven hours a week back every week forever.
- 4:28You pick a roadmap,
- 4:29reporting, data cleanup, commentary,
- 4:31forecasting, and two weeks later,
- 4:33that part of your job runs without you.
- 4:35I'll help you every step of the way.
- 4:37Plus, get at least 15
- 4:38credits of CPE every year.
- 4:41The members who finish their first four roadmaps get about seven hours a week back.
- 4:44$99 a month, cancel anytime.
- 4:47And most members have their company pay for
- 4:49it out of their CPE budget.
- 4:51Learn more at f9finance.com slash lab.
- 4:54F9 Finance LLC is registered with the National Association of State
- 4:58Boards of Accountancy as a sponsor of continuing professional education.
- 5:01State Boards of Accountancy have final authority on the acceptance of individual
- 5:05courses for CPE credits.
- 5:07Most of the AI conversations these days swing between genius or garbage.
- 5:11Somebody posts that they automated their whole close and somebody
- 5:14else posts a screenshot of it making up a number.
- 5:17What I see day to day is far simpler and more
- 5:19boring than either of those.
- 5:21AI is the most average performer on your team.
- 5:24And once you look at it that way, the useful question is how you manage it.
- 5:28So I want to walk through what managing an average performer looks like, and what I do
- 5:32about each part of the process.
- 5:35AI makes your weakest work decent, and it can make your best work a little worse.
- 5:39The best evidence I've seen on this is a
- 5:42study of 5,000 customer support agents.
- 5:44This was published last year in the Quarterly Journal of Economics.
- 5:47When they got an AI assistant,
- 5:50the number of problems they solved per hour
- 5:52went up 15% on average.
- 5:54But the gains weren't spread evenly.
- 5:57The newest and least experienced people got
- 6:00faster, and their work got better.
- 6:02The most skilled people got a little faster, but the quality of
- 6:06their work dipped slightly.
- 6:07Now you see this same shape in a study of
- 6:11400 professionals doing real writing tasks from their job.
- 6:15With ChatGPT, they finished 40% faster.
- 6:18Quality went up 18%, and the gap between the
- 6:20best and worst writers got smaller.
- 6:22So it pulls people toward
- 6:24the middle from both directions.
- 6:26And there's one more piece to this that matters a lot in finance.
- 6:30When people write with AI, their work comes
- 6:32out better, but it also comes out more alike.
- 6:37Researchers have now seen that across 19 different studies.
- 6:40It's a small effect, but it keeps showing up.
- 6:43Say you've got 40 cost centers to write up every month.
- 6:46A decent first draft of all 40
- 6:48is a great job for an average performer.
- 6:51But there are usually three lines in that pack your CFO is going to read twice.
- 6:54Those are the lines where sounding like everyone else costs you something.
- 6:58So give AI the work where average is
- 7:01good enough, and keep your own hands on the work where your judgment is the product.
- 7:06The AI you set up this spring is not the
- 7:08same AI you're using this fall.
- 7:09Back in 2023, researchers at Stanford
- 7:12and Berkeley tested the same version of
- 7:15GPT-4 twice, three months apart.
- 7:18On one simple task, telling prime numbers from everything else, it scored 84% in March.
- 7:22By June, it was down to
- 7:2451% on the exact same questions.
- 7:27And from the outside,
- 7:28you had no way of knowing it had changed.
- 7:30And that was on the same model.
- 7:32That study is three years
- 7:34old now, but it was the first time someone measured this cleanly.
- 7:38And newer isn't always better on your particular task.
- 7:41Researchers who study model updates have a name for an answer that used to be right,
- 7:44and now comes back wrong.
- 7:46They call it a negative flip.
- 7:48The model can get better overall and still get worse at the job you built around it.
- 7:54And then the models retire, which is the part that never makes the demo.
- 7:56On Anthropic's developer platform,
- 7:59Claude Opus 4 came out in May of 2025,
- 8:02and it was switched off in June of this year.
- 8:04Claude Sonnet 4.5 is next.
- 8:07It retires on November 30th,
- 8:08about 14 months after it launched.
- 8:11You get at least 60 days notice, which is fair, but anything you built on it still has to move.
- 8:17Even Anthropic's own advice is to test your work on the new model
- 8:20well before the old one retires.
- 8:22Back in episode 6, I talked about a
- 8:25process breaking the second time you run it. This one is different.
- 8:29The process is fine,
- 8:30but the model changed underneath it.
- 8:33So keep 5 to 10 questions you
- 8:34already know the answers to, and run them
- 8:36again every time your AI tool changes models.
- 8:40So here's how I manage an average performer that changes on me every
- 8:43few months, and I call it GATES.
- 8:45It builds on something Dan
- 8:47Martell calls 10-80-10.
- 8:49You do the first 10% of the job,
- 8:51AI does the middle 80, and you do the last 10.
- 8:55The trouble is that middle 80.
- 8:57Hand an average performer 80% of a job in
- 8:59one go, and you find out it was very
- 9:01average or worse at the very end.
- 9:05So my addition is GATES inside the 80,
- 9:07and that's what the letters spell out.
- 9:09G is ground it.
- 9:10Your first 10%, the goal,
- 9:12and what done looks like written down before AI touches anything.
- 9:17Episode 7 was all about doing that part
- 9:19well if you want to check it out.
- 9:21A is automate the step, where AI runs the
- 9:24next chunk of the work and only that chunk.
- 9:26T is test at the gate,
- 9:28where you check that output before the
- 9:30next step builds on it.
- 9:33E is encode it as a skill, so once a step works,
- 9:35the prompt, the context,
- 9:37and your corrections get saved and reused.
- 9:40And A, T, and E repeat for every gate,
- 9:42depending on the complexity.
- 9:44S is sign-off.
- 9:47Your last 10%, the polish,
- 9:49and the approval, and that part stays yours.
- 9:52Now, how many gates you
- 9:54need depends on the work.
- 9:56Flux commentary on one P&L
- 9:57probably gets one gate, right after AI pulls the drivers and before it writes a word.
- 10:02A forecast refresh, like mine above, gets two or three,
- 10:05after the data clean, after the driver assumptions, and after the bridge.
- 10:08A board deck or a budget build gets a gate
- 10:11at every handoff between sections, and this is where the other two problems get handled.
- 10:17The gates catch the average work before anything gets built on it.
- 10:21The skills are where the retraining lives, so you correct something once,
- 10:25you give the feedback, and it stays corrected.
- 10:27There's research behind that part too.
- 10:30A 2026 experiment found a simple written structure narrowed the
- 10:34gap between the best and worst results.
- 10:36And when the model changes,
- 10:38you run the gates again before you trust it.
- 10:41This is exactly the process that got
- 10:43my forecast to run well after
- 10:46that one shot failed miserably.
- 10:47So put a gate anywhere a mistake would get
- 10:50built on and turn every step that passes into a
- 10:53skill with a constant feedback loop.
- 10:56So pull all this together.
- 10:58You've got something that does average work and it changes on you every year or so,
- 11:02at this point every few months, and it needs someone to keep its instructions current.
- 11:07That's a description of an employee you'd manage and GATES is how I manage it.
- 11:11And most finance leaders I know are good at managing average performers.
- 11:14They've been doing it for years. They just haven't pointed that skill at AI yet.
- 11:19AI is the most average
- 11:21analyst you'll ever manage.
- 11:23And the finance pros who manage it well
- 11:25are the ones getting their hours back. So here's where I'd start with this today.
- 11:30Action one, Pick one light job
- 11:32like flux commentary on a single P&L.
- 11:35Write down the goal and what done looks like before AI touches it.
- 11:39Action two, put one gate in, just one.
- 11:42Let AI pull the drivers and check them yourself before it writes a word.
- 11:46Action three, when that gate passes,
- 11:48save the prompt and your corrections as a
- 11:51skill with today's date on it.
- 11:53Then the next time your AI tool changes models, run it through the gate again.
- 11:56If you can't do all three by Friday,
- 11:59your AI doesn't have a manager yet, and you'll see it come out of the work.
- 12:04Managing an average performer well is a skill, and you can build it on purpose.
- 12:08And finance pros who learn
- 12:10AI don't just keep their jobs.
- 12:11They become the ones no one can replace.
- 12:15And that's this week's show. If you got something out of this, do me a favor and
- 12:18rate it on Apple Podcasts, Spotify, or wherever you listen.
- 12:22It's the fastest way to help another finance pro find it.
- 12:25And if you want more from me between episodes, the AI Library is where
- 12:28I drop the prompts and templates I'm using.
- 12:30I'll also send you my free weekly
- 12:32newsletter, Finance AI Insider,
- 12:34with tips, tricks, and hacks just like this.
- 12:36The links are in the show notes, and I'll catch
- 12:38you over there. Until then, this is Mike signing off from F9 Finance. Cheers.