Latest / How I Work / Your AI gets dumber at 60%. Here's the free fix.
Transcript
- 0:00Speaker 1: So imagine you're 10 or 15 minutes into a chat
- 0:04Speaker 1: with AI and it has been brilliant. But then it
- 0:09Speaker 1: starts forgetting what you told it right at the beginning.
- 0:12Speaker 1: It makes things up and it wanders down paths that
- 0:16Speaker 1: you never asked for. Your work buddy has literally lost
- 0:20Speaker 1: the plot. But starting again feels painful because you put
- 0:24Speaker 1: so much into that chat thread and you don't want
- 0:27Speaker 1: to throw it all away. So in this episode, Neo
- 0:30Speaker 1: and I look at why this happens and what to
- 0:34Speaker 1: do about it. You're going to learn the point in
- 0:37Speaker 1: a chat where AI does become more likely to hallucinate.
- 0:41Speaker 1: And you're also going to hear why a chat thread
- 0:45Speaker 1: works best like a tissue that you use once and
- 0:48Speaker 1: throw it away. And importantly, how to hand your work
- 0:52Speaker 1: over to a fresh chat thread without losing what matters.
- 0:57Speaker 1: And as a little bonus, if you listen to the end,
- 0:59Speaker 1: Neo is giving away the handoff skill that he built
- 1:03Speaker 1: and uses multiple times a day. And by the end
- 1:07Speaker 1: of this episode, you will know exactly when to start
- 1:10Speaker 1: afresh and how to bring your best thinking with you.
- 1:14Speaker 1: And by the way, before we get into it, if
- 1:16Speaker 1: you are new around here or maybe you're not new
- 1:19Speaker 1: around here, I would absolutely love it if you would
- 1:23Speaker 1: take just like two or three seconds to leave a
- 1:26Speaker 1: review for this podcast wherever you happen to be listening it.
- 1:30Speaker 1: It is a wonderful way to help spread the word
- 1:33Speaker 1: about this podcast. And if you do leave a review,
- 1:37Speaker 1: you can trust me that we read every single one
- 1:40Speaker 1: of them. So thank you in advance. Welcome to How
- 1:48Speaker 1: I AI with me, Dr. Amantha Imber and Neo Applin,
- 1:52Speaker 1: head of Inventium AI. Each episode, we share one practical
- 1:56Speaker 1: way to use AI better at work and in life.
- 2:00Speaker 1: No fluff, no tech jargon, just things you can use
- 2:03Speaker 1: straight away. One of the most frustrating things, Neo, is
- 2:07Speaker 1: when you're having quite a lengthy conversation within a chat
- 2:11Speaker 1: thread in AI and And it's gradually starting to get dumber.
- 2:17Speaker 1: It's starting to forget things. It's starting to hallucinate all
- 2:21Speaker 1: over the place. Like, at what point can we predict
- 2:25Speaker 1: that this will happen in a chat thread?
- 2:27Speaker 2: 60% of the context window. But that doesn't really help anyone. Like,
- 2:33Speaker 2: what's a context window? And what is 60% of the
- 2:35Speaker 2: context window? It's quite annoying.
- 2:37Speaker 1: All right.
- 2:38Speaker 2: So let me give you a little bit of the
- 2:39Speaker 2: science and facts behind your favorite chat thread. But at
- 2:44Speaker 2: its heart, what I recommend you think of a chat
- 2:47Speaker 2: thread is more like a tissue that you use once
- 2:50Speaker 2: and throw away than a pet that you nurture forever.
- 2:54Speaker 2: And I know that some people have these pets of
- 2:56Speaker 2: chat threads. I come back there every Monday and every
- 2:58Speaker 2: time I talk to you about that. And yes, in
- 3:01Speaker 2: the future, some of the better models will be able
- 3:05Speaker 2: to have a pet chat thread. That's the way they're going.
- 3:08Speaker 2: But current technology, deal them like a tissue.
- 3:11Speaker 1: All right.
- 3:11Speaker 2: So there's the background. But What it is is this,
- 3:14Speaker 2: each chat thread has its own short-term memory. Each of
- 3:16Speaker 2: the tools have a different memory limit going from like
- 3:20Speaker 2: 200 kind of A4 pages wall-to-wall text up to like
- 3:25Speaker 2: 1,500 pages of wall-to-wall text. So Gemini's got the biggest
- 3:30Speaker 2: window here at the moment. I think that Copilot's probably
- 3:33Speaker 2: the smallest window here, the smallest number of texts. But
- 3:36Speaker 2: ultimately it comes to this. There are a couple of
- 3:39Speaker 2: mechanisms within each chat thread. Whatever you start talking about,
- 3:43Speaker 2: it kind of anchors on that a little bit. They
- 3:45Speaker 2: call it contextual inertia, which basically goes, okay, here's what
- 3:49Speaker 2: we're originally thinking about, and then it kind of holds
- 3:52Speaker 2: onto that, which is why if you get a bad
- 3:54Speaker 2: first chat, it's really hard to get it off that topic.
- 3:58Speaker 2: You've also got another thing, which is over time, your
- 4:01Speaker 2: AI goes into different avenues, you go into different avenues,
- 4:03Speaker 2: and then it gets confused. Why are we here? What
- 4:06Speaker 2: are we actually doing? And the two of them actually
- 4:08Speaker 2: pull apart in different ways. So this is some of
- 4:11Speaker 2: the background, but the 60% part is really important, which
- 4:14Speaker 2: is when you've got 60% of this context window full,
- 4:19Speaker 2: then it starts to hallucinate more. For some reason, that's
- 4:23Speaker 2: a magical bar. It kind of starts falling off the clip.
- 4:25Speaker 2: So it's not that at 60% it forgets everything, not
- 4:28Speaker 2: at all, It's just, it's more likely to lose the plot.
- 4:32Speaker 2: It's more likely to make up things and forget things.
- 4:35Speaker 2: And so it starts to become a really unreliable work
- 4:39Speaker 2: buddy at that point.
- 4:40Speaker 1: Okay, so the obvious question is, What can we do
- 4:45Speaker 1: about this problem?
- 4:47Speaker 2: You fire your work buddy. No, you can't fire your
- 4:49Speaker 2: work buddy. So what you do is I would start
- 4:53Speaker 2: a new chat thread. This is where you throw that
- 4:54Speaker 2: tissue away and start a new. But you've already got
- 4:56Speaker 2: a lot of work going on with this chat thread.
- 4:58Speaker 2: You don't want to throw all that goodness away. So
- 5:02Speaker 2: there's a really simple way to do it and there's
- 5:04Speaker 2: a bit of a tip way to do that as well.
- 5:07Speaker 2: So the simple way to do it is simply go, hey,
- 5:10Speaker 2: can you please summarise everything we've talked about in this
- 5:12Speaker 2: chat thread? And then you get that summary and then
- 5:15Speaker 2: you place that into a new chat thread and go, hey,
- 5:18Speaker 2: read this summary and then we're going to continue. So
- 5:20Speaker 2: it's basically summarize the first chat thread and then place
- 5:23Speaker 2: it into a new one so you can then go
- 5:24Speaker 2: off from that summary. Now, that works fine, but there
- 5:29Speaker 2: are better ways to do it because sometimes with a
- 5:33Speaker 2: summary of a chat thread, there may be an avenue
- 5:35Speaker 2: you went down and you realize, oh, no, that's a
- 5:37Speaker 2: dead end for this thing I'm trying to achieve. And
- 5:40Speaker 2: it's kind of important to tell the new chat thread
- 5:42Speaker 2: that was a dead end. Don't go down that dead end.
- 5:45Speaker 2: And so some of the things you've realized, some of
- 5:48Speaker 2: the things you're working on are important to get, absolutely.
- 5:50Speaker 2: But what are the other things that are really crucial
- 5:53Speaker 2: for a brand new chat thread, a brand new brain
- 5:55Speaker 2: to be able to help you with? And it's really
- 5:56Speaker 2: important to brief it well. Otherwise, you're then rebriefing again
- 5:59Speaker 2: and it becomes quite painful there.
- 6:01Speaker 1: So I know that you have got a special gift
- 6:04Speaker 1: for listeners that you're about to describe. And this is
- 6:07Speaker 1: a handoff skill that I know you spent quite a
- 6:11Speaker 1: bit of time creating and refining. And we're going to
- 6:15Speaker 1: put a link to get that as a little gift
- 6:18Speaker 1: from us to you listener in the show notes. But Neo,
- 6:23Speaker 1: can you talk about how you built this handoff skill
- 6:26Speaker 1: and what the ingredients are? Yeah.
- 6:28Speaker 2: So first off, this can be a skill or it
- 6:31Speaker 2: can be a prompt or it could be like a
- 6:33Speaker 2: copilot agent if you wanted it to be and up
- 6:37Speaker 2: to you how you want to use this. But really
- 6:40Speaker 2: what it's about is how do I structure what I
- 6:43Speaker 2: need to take from that old chat thread and put
- 6:45Speaker 2: it into a new chat thread? And so it includes
- 6:49Speaker 2: things that I've found to be quite useful on that,
- 6:53Speaker 2: how do I take these things over? So it can
- 6:54Speaker 2: be things like, What have I found? What have I learned?
- 6:58Speaker 2: Those kind of things, and they're pretty important. The context
- 7:03Speaker 2: and constraints of the things that I was working on
- 7:05Speaker 2: is pretty important. Where I got to in that old
- 7:08Speaker 2: chat thread, the decisions that I got to and why,
- 7:11Speaker 2: what I tried and discarded, also other open questions and
- 7:15Speaker 2: things to carry across as well. The other part as
- 7:18Speaker 2: well is really, it makes sense when you think about it,
- 7:22Speaker 2: which is the first part of it is really important,
- 7:24Speaker 2: which is, how to read this handoff brief. Because you
- 7:28Speaker 2: don't just give AI a brief. He goes, oh, that's fine.
- 7:30Speaker 2: It's an interesting brief. The start of the brief needs
- 7:32Speaker 2: to say, hey, this is we had a whole conversation
- 7:35Speaker 2: in another chat thread that you don't know about. That's cool.
- 7:37Speaker 2: But here's the summary I want you to start with
- 7:40Speaker 2: so we can then start talking turkey. So it's actually
- 7:42Speaker 2: telling it why it has the brief. Just giving it
- 7:45Speaker 2: a summary alone, it will not actually start off the
- 7:47Speaker 2: conversation particularly well because AI just goes, yeah, that's fine.
- 7:50Speaker 2: And I don't know what to do with this thing.
- 7:52Speaker 2: So Having a handoff that is structured in a way
- 7:55Speaker 2: that AI understands everything that's important from that old chat
- 7:58Speaker 2: thread is really important so you can start that conversation
- 8:01Speaker 2: quickly without missing all the goodness.
- 8:04Speaker 1: Amazing. So we are putting a link to access that
- 8:08Speaker 1: skill that Neo has built in the show notes. And
- 8:13Speaker 1: the time to use it is not at the end
- 8:15Speaker 1: of every chat thread. But Neo, when do we use it?
- 8:19Speaker 2: Whenever you think you need to is the short version.
- 8:23Speaker 2: So if you feel like this is a long chat
- 8:25Speaker 2: thread and it's starting to get a bit long, maybe
- 8:26Speaker 2: it hasn't done anything wacky or hallucination yet, maybe just go,
- 8:30Speaker 2: this is a natural point for me to start this
- 8:33Speaker 2: as a new chat thread because I've been talking on
- 8:35Speaker 2: this one for a while. Certainly, if it starts to hallucinate,
- 8:38Speaker 2: make things up, go into weird angles and weird avenues,
- 8:41Speaker 2: that's a great time for a handoff. Also, if it's
- 8:44Speaker 2: gone on a weird avenue, it's really sometimes hard to
- 8:47Speaker 2: stop it go down that line. It's kind of like
- 8:50Speaker 2: you go, if I said to you, don't think of
- 8:52Speaker 2: pink elephants, you're thinking of pink elephants. And now I go, no, no, no, no,
- 8:57Speaker 2: really don't think of pink elephants. Then you've actually just
- 9:00Speaker 2: repeated that thing that you don't want AI to know
- 9:02Speaker 2: about and actually embeds that thing even more. So if
- 9:06Speaker 2: it's starting to get a little bit annoying from those,
- 9:08Speaker 2: even if it's not hallucinating, it's gone down a weird path,
- 9:11Speaker 2: handoffs are great. So just paste in the prompt or
- 9:13Speaker 2: use the handoff skill and then you get that brief.
- 9:16Speaker 2: If you need to tweak anything about that, you can
- 9:18Speaker 2: as well by saying, look, I don't actually care about
- 9:21Speaker 2: that line. So you can take that out so you
- 9:22Speaker 2: can edit it so that your new brain is exactly
- 9:25Speaker 2: got the background that you want. So anytime it's hallucinating
- 9:28Speaker 2: or anytime you just go, this is time maybe to
- 9:31Speaker 2: protect myself from it hallucinating or it's gone down too
- 9:34Speaker 2: many weird paths. I just want to start from scratch
- 9:36Speaker 2: so it doesn't go down those paths again. These are
- 9:37Speaker 2: all good times for a handoff.
- 9:39Speaker 1: I love that. So listeners, grab that skill in the
- 9:42Speaker 1: show notes and send us a note. Tell us how
- 9:46Speaker 1: you go with it and if it is making your
- 9:49Speaker 1: life just a little bit easier and better with AI.
- 9:53Speaker 1: How I AI was hosted by me, Amantha Imber and
- 9:56Speaker 1: Neo Applin. A big thank you to Martin Imber who
- 9:58Speaker 1: does our sound editing. and Jem Rubio for production support.
- 10:03Speaker 1: And thank you to John Kilby, who composed the theme music.