Latest / CXOTalk / Prompt Engineering Advice | CXOtalk 883
Transcript
- 0:00Prompting is the secret skill that taps into a is real
- 0:05capabilities, transforming large language models from flashy
- 0:09demos into engines of real world productivity.
- 0:13Today on CXO Talk number 883, we unpack how prompting works, what
- 0:19it is, why it matters, and how to get it right.
- 0:23I'm Michael Krigsman, and with me is Nate B Jones, a widely
- 0:27respected AI expert whose sharp insights and no nonsense advice
- 0:33have earned him nearly 300,000 TikTok followers.
- 0:36We need to talk a lot about prompting for two reasons.
- 0:41One, human language is fairly vague.
- 0:45That's why we invented computer language back when we were
- 0:48programming computers in the 1st place, because it's much more
- 0:50precise. And now we're using effectively
- 0:53natural language to program computers again, and that's
- 0:57challenging. The second reason is that even
- 1:00though these models are very intelligent in certain respects,
- 1:03they are not incredibly reliable yet at inferring your intent.
- 1:10If you are not precise about what you mean or want, they
- 1:13don't do that reliably. They guess, and they might guess
- 1:16right, and they might guess wrong.
- 1:18And so both because we have to get clear with our language and
- 1:23because models don't yet infer with tremendous precision,
- 1:28prompting is what bridges that gap.
- 1:30So when we are prompting, we we're programming the AI.
- 1:36This is really going to take you back, but like in the old days
- 1:39in the 60s with punch card computing, you would literally
- 1:42bring your little, your little punch card and, and put into the
- 1:45computer and you would run it and you would see in 20 or 30
- 1:48minutes whether you got that right or not, or maybe longer
- 1:51than that if it was a big program.
- 1:54We're doing exactly the same thing with natural language now.
- 1:56We're handing the prompt to an inference model, maybe O3 Pro.
- 2:00It does take that long, 20 or 30 minutes.
- 2:02And we're going to come back and we're going to see if our little
- 2:04natural language program did anything.
- 2:07It's fascinating how time is a circle in that regard, where
- 2:10we're back to where we started. So the logic of prompting is
- 2:15effectively the logic of software development.
- 2:18Is that a correct way to say it? You could say it as like the
- 2:21marriage of software development and business intent.
- 2:25So in a sense, software development has been predicated
- 2:28primarily on building interfaces that allow business operations
- 2:34to be conducted, business logic to be encoded, etcetera.
- 2:37But now, because these models have the ability to sort of
- 2:40bring intelligence to bear, you're not just asking it to do
- 2:441 specific thing, You're not spending your time engineering a
- 2:47specific interface. Instead, you're asking the model
- 2:51to think with you. And so it's this weird mix of
- 2:54the principles of engineering with the business clarity of
- 2:59intent that has always characterized a very strong
- 3:03executive brief, for example. As the models get better, what
- 3:09does that do to prompting? Does it make prompting easier or
- 3:13more difficult? On the one hand, you don't have
- 3:17to do some of the stage management that you had to do in
- 3:232022 and 2023 anymore. You'll recall when ChatGPT first
- 3:27came out, the prompting guides were like, OK, tell it to
- 3:30pretend it's the best editor in the world.
- 3:33Tell it to pretend this or that. And then it began to sort of
- 3:36turn into chain of thought prompting.
- 3:38Tell it to think step by step, do this and do this and do this
- 3:41and do this. That stage management is
- 3:44thankfully no longer really necessary.
- 3:48You can instantiate the model by saying you are in a particular
- 3:52space, like you're in a consultative strategist space,
- 3:55you're in a CFO space, whatever. You can say that, but you don't
- 3:58have to like put in the adjectives and hope and pray
- 4:01that the model understands what you mean.
- 4:03You can just say this is where we are.
- 4:04You don't have to specify chain of thought anymore.
- 4:07The frontline models know to use chain of thought when they need
- 4:10to. And so in that regard, prompting
- 4:12has gotten simpler. On the other hand, the
- 4:16importance of specifying what you're looking for, what success
- 4:19criteria looks like, what the constraints are, that's only
- 4:23gotten more important because these models are much more
- 4:27powerful. And so before, like if it was a
- 4:31very simple ask that you had for a smaller model, you could go
- 4:35back and forth a few times and figure out what you wanted and
- 4:37it was fine. But if you give something to a
- 4:39frontline model and like it's running for 6 minutes, 8
- 4:43minutes, 10 minutes, 20 minutes, it comes back and you just did
- 4:47not clearly specify the scope, you're going to be frustrated
- 4:52because you wasted all of that compute.
- 4:54And so in that sense, some of the some of the, the, the stage
- 5:00management and scripting that you're used to, you don't have
- 5:02to do anymore, but the importance of specifying the
- 5:04work very clearly has grown. Like you have to really take
- 5:08that seriously now. So you really need to match the
- 5:12prompt to the model. A lot of the art of it is in
- 5:16figuring out what is this subject, what is my intent, what
- 5:20is the right model for that? And once I have all of that
- 5:23figured out, now how do I craft a prompt and then bring in the
- 5:26context the model needs so it can do a good job for me?
- 5:30For example, with open AIA, number of the models allow you
- 5:37to either include deep reasoning or research or not.
- 5:42And actually other other companies as well other LLM same
- 5:46thing. So give us some examples of
- 5:48this. This is one of the things where
- 5:51model makers have not done a great job at the product surface
- 5:55of explaining what their models do.
- 5:57For example, deep research is really a very narrow web agent
- 6:02that is trained as a research assistant to go out to look
- 6:07across the entire browsable web, it doesn't yet look behind
- 6:11paywalls and to come back with a consolidated view.
- 6:15And they train it specifically on citation.
- 6:18So it's good at citations. It lists what it knows and why.
- 6:22Open AI pioneered this with Deep Research, but Deep Research is
- 6:26now available on Perplexity. It's available on Claude.
- 6:29It's available with Google. Lots of others have picked this
- 6:32up because it turns out that reasoning across the web is a
- 6:34lot of what we do. And so there's just inherent
- 6:37value in report generation. But people don't realize that
- 6:43all you're getting with Deep Research is the O3 model if it's
- 6:48ChatGPT specifically tuned to web search.
- 6:52And that is different from whatever else you've been
- 6:57talking about with whatever model you've been talking about
- 7:00in ChatGPT previously. So if you've been having a
- 7:02conversation with four O for a bit and then you turn on deep
- 7:05research, it's not that four O suddenly picks up a Cape and
- 7:09becomes a superhero and turns into deep research.
- 7:12It's that you are invoking a separate agentic tool, getting a
- 7:16separate prompt in, starting a new flow, and then that report
- 7:21is going to come back and you're going to be able to continue the
- 7:23chat. And I think that a lot of people
- 7:25don't think about it that way. And it's become even more
- 7:28confusing in the last week because O3 Pro on the surface
- 7:32looks very, very similar. It's got a long thinking time.
- 7:36You give it a prompt, it goes away, it comes back.
- 7:39And so people have asked me, did they just release a clone of
- 7:42Deep Research and rename it? And the answer is no.
- 7:46The answer is that O3 Pro is a generalizable model with a lot
- 7:50of different tool calls under the surface.
- 7:52But precisely because it's under the surface, it's difficult to
- 7:56know that staring at the chat window when it takes a similar
- 7:59amount of time and comes back. And so I think that some of what
- 8:04I do is just try and convey the nuances of these models and how
- 8:08understanding them with a little bit of a fingertip feel can
- 8:11shape the way we prompt. Again, there's a level of
- 8:16confusion here. I mean, I use so many different
- 8:21models every single day and I am on an ongoing basis having to
- 8:29kind of experiment. You know, it's like this whole
- 8:32domain is very immature because the models are changing and the
- 8:37models give indeterminate results in any case.
- 8:40And that means you keep having to adjust your prompts on an
- 8:43ongoing basis. It's it's really a waste of
- 8:46time. I think if it was a waste of
- 8:48time, we wouldn't be seeing the kind of tremendous uptake we see
- 8:53on ground groundswell usage with these models.
- 8:57One of the biggest challenges with IT and security this year
- 9:01is shadow IT where people are finding these models so useful
- 9:06for the work that they do that they are using them even outside
- 9:10traditional IT security practices.
- 9:12And in that sense I share your frustration.
- 9:17I find that like when I am not getting what I want, there's,
- 9:20there's nothing more frustrating than sort of pounding my head on
- 9:23the wall and trying to figure out what the model needs to hear
- 9:25from me so that it can give me what I want.
- 9:30But net, net, if I look across my overall productivity for the
- 9:34day, for the week, I am so much more productive now, even with
- 9:40all of that factored end than I was two years ago.
- 9:42And it's because I'm learning enough about how to work with
- 9:46these models that I'm able to get a tremendous amount of value
- 9:49back. And I think a lot of people are
- 9:51having that experience. And maybe I shouldn't say it's a
- 9:54waste of time, although I do think it's a waste of time, but
- 9:56let's just say that there's a lot of overhead that seems like
- 10:03it shouldn't be there. That's a really fair call.
- 10:06That's basically a complaint and not that it makes any difference
- 10:09at all, because that's the nature of the maturity of these
- 10:14models as products at this point in time.
- 10:18What I am curious to see answered by the model makers in
- 10:22the next probably 18 months is the extent to which prompting
- 10:28remains a durable skill set that provides tremendous alpha to
- 10:33people who know how to use it well versus the extent to which
- 10:36it commoditizes. Not necessarily because everyone
- 10:39learns the same amount, but because models get very, very
- 10:42good at inferring intent across a range of prompts for the same
- 10:46subject, and people are widely divergent on what they think
- 10:51will happen. My own view is I'm trying to
- 10:54take seriously the fact that I expected initially prompting to
- 10:58be a very one off 2022-2023 edge, and that's not been the
- 11:03case. It's been stronger and stronger
- 11:05over time instead. So I tend to lean toward the
- 11:08idea that at least for the intermediate term, prompting is
- 11:11going to continue to have a tremendous amount of value
- 11:13because that's what we've seen as a trend so far.
- 11:16There are people who think that if we can get to a level of
- 11:19generalizability with these models, we will suddenly unlock
- 11:22a tipping point and we will find a way to infer very reliably
- 11:26where we haven't before. And that might be.
- 11:29And if that's the case, then suddenly prompting will become
- 11:32less painful and less needed somewhere in the next 18 months.
- 11:36Subscribe to the CXO Talk newsletter so you can be part of
- 11:41our community. We have amazing shows coming up.
- 11:44What makes a good prompt? Do you have any practical
- 11:49advice? Number one, be really clear
- 11:52about the outcome that you are looking for and about how the
- 11:56model can know that it's done. I think a lot of people will be
- 12:01fairly loose about specifying the outcome or they'll be loose
- 12:05about the goal. They'll be very, very loose or
- 12:06non existent about how the model can know that it's finished
- 12:09adequately. And the more you can specify and
- 12:12be clear about what you're looking for and what good looks
- 12:14like, the better off you're going to be for the rest of the
- 12:17prompt #2 you want the model to have all the context that it
- 12:23needs to do that job, and you would prefer it to not have any
- 12:28extra context that it doesn't need.
- 12:32A lot of what we call hallucinations are effectively
- 12:36models reasoning outside your desired context window.
- 12:41And so if you can be more clean and clear about this is what I
- 12:45want you to focus on in a web search or here's some documents
- 12:48I want you to review. I want you to keep your thinking
- 12:50focused around this particular, you know, set of meeting
- 12:53transcripts or whatever it is. It will really help the model to
- 12:56be confident that it's doing the right job and able to deliver a
- 13:02reasoned results that closely matches the kind of work you
- 13:08were looking for. And so the context piece is
- 13:09another one. And then the third is really
- 13:11making sure that you understand the the constraints and
- 13:18guardrails that you want to put around.
- 13:19So if you have a, if you have an outcome or goal, if you have
- 13:22context, you feed it. You then need to make sure that
- 13:25the model knows don't do this. Where do I not go?
- 13:30And I find that that is often one that people either barely
- 13:35put in or tend to avoid because we tend to be thinking in a
- 13:37positive stance of like, hey, this is what I want done.
- 13:40Let me just give the task and go.
- 13:43And maybe this is because we're anthropomorphizing models.
- 13:46Anthropomorphizing models. We don't tend to regard a senior
- 13:49colleague as someone who needs a tremendous number of warnings
- 13:54and constraints for a task. We just say, hey, go tackle
- 13:57this. I'm sure you'll do a great job.
- 13:58Come back and let me think. Think about what you get.
- 14:01These models need those constraints still.
- 14:03Even if they in many ways are very senior in their thinking,
- 14:07they still need helpful constraints so that they know
- 14:11where the guardrails are in the space.
- 14:13And they don't start to reason off the rails into a direction
- 14:17that that isn't helpful. Because at the end of the day,
- 14:20what they're really trying to do is just infer from your
- 14:22utterance what they think you mean.
- 14:24Figure out where in latent space they can go and get a reasonable
- 14:29pattern match. Do some searching across the
- 14:31web. In the case of an inference
- 14:33model, do a lot of that iteratively so they can figure
- 14:36out what's best and then put together something.
- 14:38And so they do need those guardrails to constrain.
- 14:40Models are people too, and you can't.
- 14:44Just as you can't expect your friend or your spouse to read
- 14:52your mind, how can we expect models to anticipate every
- 14:58possibility that's out there and map it to what happens to be in
- 15:02your mind at this given time, what you want when you write
- 15:05this prompt? And that's the need to be
- 15:07explicit. And that's why I say models are
- 15:10people too. We as humans are very, very good
- 15:13at retaining long context from multiple conversations with our
- 15:18colleagues and extracting what's really important out of that and
- 15:22and getting to clear points of discussion.
- 15:25Like I can talk with a software development manager about a
- 15:27project that's been going for six months.
- 15:30We can have a really meaningful discussion about the sticking
- 15:32point decisions we've made in the past.
- 15:34What we need to change. That is how humans have done
- 15:37work for a long time. We we iterate over time
- 15:40effectively, the prompt evolves through conversation over time.
- 15:43It's shared work together with the model.
- 15:46We can't have the same iterative conversation.
- 15:49We actually have to front load all of that thinking and give it
- 15:53to it in a really clean prompt so we can get a really clean
- 15:56answer. And I think part of what's hard
- 15:57for us about prompting is we're conversational people.
- 16:00We like to chat just like you and I are chatting.
- 16:03We make meaning that way, but the model needs us to sort of
- 16:06compress that semantic meaning into a, like a really clean
- 16:10initial prompt that will help it to work effectively.
- 16:13I think this is a very important point that you're making is
- 16:18there is the need to, as you said, to compress the context
- 16:25into a digestible set of words and chunks that the model can
- 16:32then use to execute the explicit task.
- 16:36Going back to an earlier comment, you made that in effect
- 16:39you are programming into the model, into the and driving the
- 16:43conversation through that programming essentially.
- 16:46Right, because we humans effectively, collectively derive
- 16:50intent and collectively reach decisions through conversation.
- 16:53But the model needs you to be the one that provides the
- 16:55intent, that provides the driving force.
- 16:57There's a higher expectation of human agency in prompting.
- 17:01Let's jump to some questions. If you're watching on Twitter,
- 17:06pop your questions into Twitter using the hashtag CXO Talk.
- 17:11If you're watching on LinkedIn, pop your questions into the
- 17:14LinkedIn chat. And so this first question on
- 17:18Twitter X goes to Arsalan Khan, who says it seems like asking
- 17:25specific questions in your industry to the AI would deter
- 17:30fake experts. But how would an end client know
- 17:34the difference between real experts using AI versus just
- 17:39good or fake experts or salespeople using AI?
- 17:44And if I can restate that question in terms of prompting,
- 17:49if somebody is really, really good at prompting, can't they
- 17:53present the appearance of being an expert where it's almost
- 17:57impossible to tell them apart from somebody that has the PhD
- 18:02in whatever the subject might be?
- 18:04That is more true than many of us would like to admit.
- 18:08I think it's part of why there are so many, many consultants
- 18:13springing up, so many tools springing up.
- 18:16The industry has a need for authenticity, but AI by its
- 18:20nature is enabling many people to claim expertise that they
- 18:24don't genuinely have. And there's just like there's
- 18:28not a silver bullet solution for detecting text in student essays
- 18:34and saying who, who wrote which which bit of text.
- 18:38There's also not a silver bullet solution for detecting
- 18:42expertise. I find in practice, what tends
- 18:46to be most helpful at distinguishing a true expert
- 18:49from the sort of AI generated straw man expert is acknowledge
- 18:56the source material. It's probably going to be very
- 18:58good because AI helped prepare it, it's very thorough,
- 19:01etcetera. Make sure you understand it and
- 19:04then ask a question that's designed to push off balance,
- 19:09push out of the comfort zone, and a true expert will be able
- 19:13to adjust and have an interesting and thoughtful
- 19:15perspective and not get too frustrated or flustered.
- 19:19And someone who's depending heavily on the prompting is
- 19:23often going to struggle because they won't be able to actually
- 19:26have that flexible intelligence across the domain that
- 19:28characterizes true expertise. So you're saying that there is a
- 19:31level of depth that the model doesn't, or that a person who's
- 19:37simply relying on the model doesn't have.
- 19:39Is that another way of saying? I'll give you an example.
- 19:41So I was doing an article on prompting an O3 Pro versus O3,
- 19:47right, Because O3 Pro came out this week and I asked it to
- 19:51prepare a road map because I'm very familiar with Rd. maps that
- 19:54came up through product management.
- 19:55I've seen more of them than I would care to admit.
- 19:58And I asked, I asked for that because I knew I could judge it.
- 20:02I knew I had the expertise to assess it.
- 20:05And I was talking with someone afterward and I was saying O3
- 20:09Pro did a much better job on the road map.
- 20:12And they were like, well, how did you know?
- 20:14Aren't Rd. maps subjective? And I immediately pulled up 3 or
- 20:184 reasons why Rd. maps are not subjective.
- 20:20Why? It's actually a craft you can
- 20:22understand and and 9 out of 10 experts will agree with you that
- 20:24a particular road map is better than another because it's a
- 20:27proactive stance. It takes into account all of the
- 20:31strategic advantages the company has.
- 20:33It thoroughly understands the marketplace.
- 20:34I could just go on and on. It's all at the top of my head.
- 20:37And so having that expertise helps you to assess the true
- 20:40quality of model response. And in a sense, what we're
- 20:43seeing here is that these models are getting to a level of
- 20:46intelligence where their very best work takes an expert to
- 20:51truly understand and appreciate. We have some really interesting
- 20:54questions that are coming in on LinkedIn right now.
- 20:58And Greg Walters is responding from the to the point you made
- 21:03earlier. Nate, where you where you were
- 21:06describing the need for a compressed highly efficient
- 21:09prompt. And Greg says this isn't the
- 21:12magic in prompt iteration. Instead of having one
- 21:17compressed, highly efficient and explicit explicit task or
- 21:22prompts shouldn't there? Shouldn't we be collectively
- 21:28prompting prompting? It depends on the kind of task
- 21:32that you're looking for. So this gets back to the
- 21:36relationship between prompting and model selection.
- 21:40For certain kinds of models, they're more suitable to
- 21:46iterative thinking and iterative brainstorming.
- 21:48We haven't really talked about the relationship between model
- 21:51and interface, but I find if I'm using advanced voice mode, it's
- 21:57just a very different experience for my brain because I'm talking
- 22:01instead of typing and I am much looser and it's much more
- 22:04conversational. It is in a sense much more
- 22:07iterative and I keep it that way on purpose.
- 22:09But if I'm working with a long inference time model, and it's
- 22:16not just that ChatGPT has a monopoly on those.
- 22:19Opus 4 is a great example from Claude.
- 22:22I want to be clear in what I'm looking for because frankly, it
- 22:26is expensive to iterate when the cycle times take that long.
- 22:30And so I pick the problem and I pick the model and that guides
- 22:33me to a prompting style. We have a really interesting
- 22:37question from Wayne Anderson on LinkedIn and he says this.
- 22:43How would you address the fear that leaders using large
- 22:47language models could inhibit and erode decision making and
- 22:53critical thinking? When does effective prompting
- 22:57help and when do you think leader should avoid using AI?
- 23:01It's kind of like asking, do you want your doctor to avoid using
- 23:05AI? If we have studies that show
- 23:07that medical reasoning is something that these models are
- 23:10very good at, I would love my doctor to use AI as long as my
- 23:14doctor understands how to use it well.
- 23:16And so in that sense, my response is I want AI leaders to
- 23:20be using AI all the time. I just want them to understand
- 23:24the limitations of these models and where they need to think
- 23:27beyond the edges. And so really, I think it's more
- 23:30precise to say these are extraordinary models.
- 23:34In some places, they are advancing the far edges of human
- 23:38thought and research. We have AI developed drugs and
- 23:40pipeline, but they're narrow. They have like particular ways
- 23:46in which we can prompt them that generate extraordinarily
- 23:49effective results. And the strength of a good
- 23:51leader is not being only narrow, it's at that T shaped leader
- 23:55where you have that breadth of experience as well.
- 23:57And So what I would look for a great leader to do with AI is to
- 24:00know when he or she needs to go to AI for a deep, precise,
- 24:06thoughtful perspective on something and then to bring that
- 24:09generalized experience of the business to bear to say this is
- 24:13how I would contextualize that and understand it for my broader
- 24:18problem set. But let me just go back to the
- 24:21comment that Wayne Anderson made.
- 24:24When I write certain things, I'll write something and I'll
- 24:29ask ChatGPT or whatever the model is.
- 24:33What do you think? And it will make suggestions and
- 24:37this new canvas feature, I guess it's not so new anymore of
- 24:41ChatGPT makes it really easy to like drill down to very small
- 24:45segments. It produces good results.
- 24:48But in the back of my mind, I'm thinking to myself, it's giving
- 24:54me kind of the least common denominator, mass market
- 25:00generalized. Solution.
- 25:03Not necessarily. That might be your prompting.
- 25:06And I think that's what's interesting about these models
- 25:08is that you, you are correct that if you're not intentional
- 25:14about how you frame the models position in latent space, it
- 25:20will default towards something that's more highly probable,
- 25:23which we often translate as the least common denominator.
- 25:29If you are intentional though, and you want to lean in and say
- 25:32I don't want a mid answer, I don't want a common answer.
- 25:36I want a really creative answer. I want a really thoughtful
- 25:38answer. I want an answer that you
- 25:40haven't heard or seen elsewhere. Models are perfectly capable of
- 25:44going that far and thinking more creatively, thinking more
- 25:48substantively, but they don't do it by default because the way
- 25:53they're trained is to be helpful for as much of the population as
- 25:57possible. And so in a sense, our own
- 25:58population distribution shapes the way the model makers are
- 26:02tuning these models for general helpfulness.
- 26:05And so it's up to us if we want something more on the far side
- 26:08of the distribution to push for it.
- 26:10We're drifting from prompting here.
- 26:13Oh, you. Do that with prompting.
- 26:14It's prompting going to create the next, you know, set of Bach
- 26:20inventions. No, I don't think so.
- 26:23And I think especially in the creative arts, like I would say
- 26:26that like humans have tried, I've actually a huge fan of Bach
- 26:30in the cello suites. I love them.
- 26:31I listen to them almost every weekend.
- 26:34And people have tried to expand on to invent after Bach even
- 26:42through the 20th century. And in my view, no one has done
- 26:46for the cello what Bach has done for the cello.
- 26:49And so no, I don't believe that we are in any danger of a
- 26:53machine coming along and doing a better job than Bach at Cello
- 26:56Suites. Let's jump over to Twitter from
- 26:59Chris Peterson, who says tokens and time measures are all very
- 27:05well, but doesn't every round trip of prompting eat up more
- 27:09electricity and water for cooling, thus making some of the
- 27:13numbers from Open AI and others highly misleading?
- 27:17No or yes it does, and yes it matters in aggregate, and yes we
- 27:22should talk about power use in aggregate.
- 27:23I think it's an appropriate conversation to have.
- 27:26But individual, individual prompt usage by people doesn't
- 27:31compare to some of the other things we do day-to-day that use
- 27:35energy and water. So taking a hot bath is much,
- 27:38much more expensive in water than any kind of ChatGPT prompt
- 27:42you're going to run. Watching an hour of football on
- 27:44the big screen is much, much more expensive in electricity.
- 27:48Then I think it runs up to a couple of 100 ChatGPT prompts.
- 27:52And so does it matter in aggregate?
- 27:55Yes, because suddenly a billion of us are using this.
- 27:57It's important, we should talk about it.
- 27:59Not saying that we don't have relevant conversations to have,
- 28:02but I think the idea that an individual prompt is
- 28:07fantastically expensive is incorrect when we actually
- 28:12factor in the energy usage of a day-to-day life.
- 28:16Let's jump over to another question.
- 28:18This is from Chris Chablonsky on LinkedIn who says Do you have
- 28:23any tips for using Gen. AI for a data analyst to process
- 28:28a large data set and generate visualizations?
- 28:33I'm not sure if it's the right tool for the job.
- 28:35I've sort of talked about this a little bit with folks who are
- 28:39managing large data sets. And what I find AI
- 28:45extraordinarily good at is handling data sets that don't
- 28:51have clean numeric, numeric data, right?
- 28:54If you have clean numeric data, we have fantastic tools for
- 28:56that, and they may include machine learning or they may be
- 29:00just traditional sequel, but we're very, very good at
- 29:04handling that efficiently with compute.
- 29:07I don't know why we would switch that out and ask a large
- 29:12language model to do that when the language model wasn't even
- 29:15designed primarily to be numbers driven.
- 29:20They use Python And other tools to handle numbers now, and
- 29:25that's great. But if you're talking about a
- 29:26truly large data set, we have tools that handle those data
- 29:32sets and visualizations really effectively.
- 29:33And what I find people using in practice when they're looking at
- 29:36large data sets and AI is they're using AI to help them
- 29:40craft SQL statements. They're using AI to help them
- 29:44think through the data schema that they want to set up.
- 29:47Sometimes they're using AI to help them prototype
- 29:52visualizations that they will want to get to quickly.
- 29:54Claude is great for that. And all of those are sort of, by
- 29:59the way, uses of AI that help you to use that data more
- 30:01effectively. But that's different from the
- 30:03traditional assumption that you can just sort of type the query
- 30:06in and you will magically get a better answer than you would get
- 30:09with really efficient sequel. It's a really, really good point
- 30:15that you've got to have an understanding of the particular
- 30:19tool that you're using and what will be the most effective use
- 30:24of that tool. And as you said, prompts are
- 30:28great if you have a body of data and you're trying to figure out
- 30:32what have I got here and how can I present it?
- 30:36And is there something that I'm missing?
- 30:39Prototyping, as you say. But there are tools out there
- 30:41that are designed for, you know, millions of records and that do
- 30:46it really well. I don't think you'd want to put
- 30:48millions of records into ChatGPT.
- 30:50It's not really designed, if you think about what we mean when we
- 30:53talk about context and prompts. It's designed to look across the
- 30:58overall picture. And oftentimes with data, we
- 31:00don't just want an overall picture.
- 31:03We want precision. And that's something that we
- 31:05have existing tools that do very, very well at.
- 31:08Let's get into the structure, the nature of large language
- 31:13models, how they think, think in quotes and operate, and what
- 31:18that means for prompts. Maybe take us down that path a
- 31:22little bit. It's probably worth calling out
- 31:25that a lot of the difference in how prompting has evolved is
- 31:29being driven by this movement from large language models that
- 31:33are what I would call vanilla. So that it's just coming back
- 31:37with a response based on weights and vector space developed
- 31:41through pre training data. Which is what we had into 2024.
- 31:46And then the newer version, which is inference time,
- 31:49computes models where they have that same underlying
- 31:53architecture, but at the time you press enter and send in your
- 31:57query, they are running threads in the background trying to
- 32:01figure out what the correct response is.
- 32:03And there's different ways of doing that.
- 32:04Sometimes it's a combination of expert models in the background
- 32:08that are sort of coming up with answers and deciding amongst
- 32:10themselves. Sometimes it's running the same
- 32:12query multiple times in parallel in the background trying to find
- 32:16the most common answer. Regardless of the underlying
- 32:19architecture, the effect of having more time to run cycles
- 32:25on your query is tremendous. It's it's a night and day
- 32:29difference in terms of the intelligence that the model is
- 32:31able to respond with. And so that is a lot of what has
- 32:36shaped different prompting. A lot of the reason we don't
- 32:39have to give chain of thought instructions anymore is because
- 32:43the models already have a way of deeply processing the queries we
- 32:49give them when they are inference models and they don't
- 32:51need our help to do so anymore. And so when I say you don't need
- 32:55chain of thought, but you want to be clear on your goals and
- 32:57basically saying try and write a prompt that understands that you
- 33:03are going to be running multiple parallel streams of thought in
- 33:07the background or multiple parallel streams of tokens in
- 33:10the background. Constrain it.
- 33:13Like if you if you have 10 that are going to run, you don't know
- 33:15it's time. But let's pretend it's time for
- 33:16simplicity's sake. Make sure all 10 are focused on
- 33:21what you care about because you want to constrain the scope of
- 33:23the query so that it's actually focused on where you want to go
- 33:27with a conversation. And so that's why I emphasize so
- 33:30much. Set a goal, make sure the model
- 33:32knows what good looks like. Make sure you set guardrails,
- 33:34etcetera, etcetera. Describe to us what you mean by
- 33:38a chain of thought prompt. It's where you said I want you
- 33:42to answer my query to a traditional model using pre
- 33:46training data weights and it would come back and answer.
- 33:49But you wanted the token stream to go through a particular
- 33:53sequence. And so it's going to go through.
- 33:56And from a transformer perspective, like the
- 33:58Transformers there, it's basically using your query, it's
- 34:01matching it in vector space. Once it vectorizes it with what
- 34:04it has for weights and pre training data, it's coming back.
- 34:07And you're basically saying, let me give you a deeper query with
- 34:11a lot of things I want you to think about and do.
- 34:13So start with, this is who you are.
- 34:16You're an expert on marketing. Second, I want you to think very
- 34:20deliberately about this campaign that I want to launch.
- 34:24Third, these are the steps of thinking I want you to go
- 34:27through. First, develop a plan.
- 34:29Second, critique your plan. Third, understand the
- 34:32consequences of the plan in the market and you can kind of go
- 34:34through. And that's like chain of
- 34:35thought, right? When you do that, you're
- 34:38basically being very particular about the places in vector space
- 34:43that you want the model to go and hit when it's generating the
- 34:46response. And because models read like
- 34:49humans do, they read top down, when the model hits that point,
- 34:53it's going to be effectively sequentially reasoning back to
- 34:58you because of the way you programmed it.
- 35:01And so this gets back earlier in our conversation, Michael, when
- 35:04we talked about this idea of natural language programming,
- 35:07that we are effectively programming the model, that was
- 35:10sort of what we were doing. And all we're saying now is we
- 35:14still have to program the model. We don't have to program it
- 35:17quite that way anymore. How do we program it today?
- 35:20Today when we program the model, we want to be focused more on
- 35:24outcomes and goals. And in the past it was focused
- 35:28more on process. And so today if I'm looking for
- 35:34a report like I digested a, you know, 130, a 140 page economic
- 35:40report this morning from the world, I think it was the World
- 35:43Economic Forum, something like that.
- 35:45It I I wanted the model to understand what I wanted out of
- 35:49the report and the goal of the summary.
- 35:52I didn't just want a vanilla summary.
- 35:54And so my focus was on making sure it knew the the angle I
- 35:57wanted on the report. And I trusted it to know how to
- 36:01read, digest, summarize, think through all the things I would
- 36:04have had to specify earlier. In that case, again, the the
- 36:08context, giving it the background and the goals becomes
- 36:14the key focus of the prompt as opposed to telling the LLM how
- 36:20to do its job. That's right, we have another
- 36:23question from Arsalan Khan on Twitter.
- 36:26Arsalan says to prompt or not to prompt?
- 36:29When is it appropriate and when is it just a rabbit hole for
- 36:33your confirmation bias? I think one of the biggest
- 36:36differences in the way people use models right now is people
- 36:40who are focused with their models can use the model as a
- 36:44mirror that focuses on a particular subject really
- 36:46effectively. And there are people who are
- 36:48less focused and the mirror becomes a scatterer for them.
- 36:51It scatters their thinking. They become more confused as
- 36:53they use it. And I've seen both.
- 36:57What I find is interesting about the critical thinking piece.
- 37:00Imagine the mirror, and typically it faces you right.
- 37:03Then it becomes a reflection of yourself.
- 37:05You're absolutely right. There's no critical thinking
- 37:07there. It's just coming back with
- 37:09confirmation. But if you're smart, you can
- 37:11turn the mirror away from yourself and you can focus it on
- 37:14something else and you can come back with a disconfirming or
- 37:17divergent opinion. And so I will frequently ask the
- 37:20model to fight with me. I will ask it to disagree.
- 37:24I will ask it to come up with a Steel Man argument because I
- 37:27think it's much more interesting and my thinking gets sharpened
- 37:31when I do that. Actually I do the same thing.
- 37:34I very often will say to the model, be very critical.
- 37:39Don't worry about hurting my feelings, be sharp.
- 37:43That's right, like an iron sharpens iron vibe is what I
- 37:46like to go go for. Makes sense.
- 37:48What's the best way to again craft that prompt?
- 37:53We were trying to accomplish something.
- 37:56Shall I show my screen? Would it be helpful to just kind
- 37:59of take a peek at a prompt I wrote?
- 38:01Sure, let's do that. All right.
- 38:03This is a real prompt that I wrote and this is an example of
- 38:08me picking something where I feel pretty good about sort of
- 38:10my overall ability to assess quality of response.
- 38:14But I don't have a direct answer to this question.
- 38:18And this was part of a sub stack article that I was writing to
- 38:20test O3 Pro. So this is four O 3 Pro.
- 38:23You can see it up there and I'm asking it to step through this
- 38:26analysis with me. So think you're a senior product
- 38:29leader brought in to design a 12 month AI adoption road map for a
- 38:32real firm. First I I could have given the
- 38:37model the choice of firm and I tried a separate prompt where I
- 38:40gave it that option. That was very interesting.
- 38:42In the end, I wanted something with a company that I was
- 38:45familiar with since I was working on it for testing
- 38:47purposes. So I used Datadog.
- 38:49I ask it to do some very specific information gathering.
- 38:54So build the source corpus. I want publicly available
- 38:57information, I want 10 KS, I want job postings, I want SEC
- 39:00FINRA guidance. And then I want 3 responses.
- 39:06And I actually specify the word count output and I specify what
- 39:09I want there, right? There's a strategy memo first,
- 39:12there's a tech stack overview, and there's a regulatory
- 39:15constraints piece. And So what what's interesting
- 39:19is by using the word internal, I am suggesting to the model that
- 39:23the model can craft these inside the chain of thought that it's
- 39:27running behind the scenes without me having to see it.
- 39:31And then Step 2, produce. Now I'm starting to ask for
- 39:34output. I'm starting to ask the model to
- 39:37come back with one document with an executive summary, month by
- 39:41month road map, a KPI per quarter, anticipated failure
- 39:46modes and mitigations, and an advisor briefing.
- 39:49And then I'm giving it styling that I want.
- 39:51So I want it to be really brutally honest.
- 39:53This is an example of not looking for confirmatory
- 39:55thinking. I do not want tables, I want
- 40:00just bullets if need be and I would like.
- 40:03To get a sense of what shaped your recommendations right, I
- 40:07want to know where you got some of this thinking from.
- 40:10And then I give it a limit at the top.
- 40:12It can't be more than 7500 words.
- 40:14So it ran, it thought about it, it was a 6-7 minute, basically a
- 40:187 minute run. So it chooses Datadog, which I
- 40:22specified, It does a little bit of basic research on Datadog.
- 40:27It builds the source corpus, so it gives you a sense of what's
- 40:31in the box there. Market contacts, Datadog's Edge.
- 40:36It's starting to adopt the persona.
- 40:38So it's saying where we lag and talking about sort of other
- 40:42updates in the competitive space, getting into growth
- 40:46goals. And what I love here is that it
- 40:48actually called out like the statement by the CEO by by
- 40:52Olivier around what they're looking for and why.
- 40:55And it's taking that into effect and taking that into account the
- 40:58way, frankly, a good road map builder should.
- 41:02It's looking at client mix. And this is a situation where
- 41:07it's done its own research to come up with that assessment and
- 41:10given sort of a very rough assessment of that.
- 41:13It's looking at the AI aspirations it can find from
- 41:18each of the different C-Suite members.
- 41:20It's looking at strategic gaps to close.
- 41:22This is all just in preparation and hasn't even really started
- 41:26the assignment yet. It's just kind of thinking it
- 41:27through. It's now going into the current
- 41:30stack in great detail, looking at vendor contracts, security
- 41:35posture. You can see where we're going
- 41:37here. Eventually it's actually going
- 41:40to get to what it wants to say. And it's actually a very cogent
- 41:45thesis. It talks about how you sort of
- 41:49dominate the data exhaust space and what that means.
- 41:51And then it starts to get into the the road map piece.
- 41:54But my point here, like we could go through this, but we don't
- 41:56have time. The the point is basically
- 41:58because I structure the prompt carefully, I got exactly what I
- 42:02was looking for back. What if you're not trying to get
- 42:05a research report, but you're trying to do, say, small, small
- 42:09research? Find out the answer to some set
- 42:12of questions, for example. I did 1.
- 42:15I don't know if I have it handy or not.
- 42:17I will see if it's there may be emerging trends and investing.
- 42:22Yeah, I think it's this one. This is a much shorter report.
- 42:25See that? That's the whole thing right
- 42:26there like that. It feels short comparatively and
- 42:29it was a very short ask. Please analyze this economic
- 42:33report and I'm really interested in again, I'm trying to push it.
- 42:36I want to understand emerging trends and I want to understand
- 42:41areas not commonly discussed, right?
- 42:43I'm looking for it to sort of push beyond, but it's not a very
- 42:46long prompt per SE. And then it just jumps right in.
- 42:50It reads all 138 pages. It gives me a snapshot.
- 42:53It talks about commodities and where they're at globally, miss
- 42:57pricing issues driven by LNG, and it basically goes through
- 43:03themes it's seen. And then at the end, it Nets it
- 43:05out like this is the big picture assessment for the next 6
- 43:08months. This is the macro assessment and
- 43:12this is how you start to lean in.
- 43:14And what's interesting is this is much more specific.
- 43:17I ran the same prompt with O3 and O3 Pro and it was
- 43:21interesting to see the relationship between the two
- 43:23because O3 focused on on this sort of bifurcation piece and O3
- 43:28Pro had a slightly different perspective.
- 43:29And I know we have like 8 minutes so we probably don't
- 43:31have time to get into it, but I thought it was fascinating to
- 43:34run a short prompt on both and see the differences.
- 43:36We have a question from LinkedIn and this, and I was going to ask
- 43:42something very similar, which is this is from Laura Finlayson and
- 43:47she says with a prompt like this, which model will do the
- 43:51best at retaining the prompt information for future use?
- 43:55She built and refined her job application prompt in Gemini,
- 44:01but it seems to forget some of the deliverables each time she
- 44:05goes back with a new job description.
- 44:08Claude allows her to save a project, but she doesn't love
- 44:12its writing. And so we need to talk about
- 44:19which model is the better model to use and how do you choose.
- 44:23And we only have a few minutes left, but this, you know, we
- 44:26could go on forever here. So what do we do?
- 44:29There's two things going on there.
- 44:30The 1st is memory. And ChatGPT really has a killer
- 44:36feature edge with memory right now because they do have, it's
- 44:40not perfect, but they have a memory feature that enables the
- 44:43models to start to actually have a living context of information
- 44:47about other chats you've had inside the same model surface,
- 44:51right? So if it's in ChatGPT, it
- 44:53doesn't matter if you're talking with O3 or 4-O or whatever.
- 44:56There's going to be a loose understanding of recent
- 44:58conversations you've had along with some specific facts that
- 45:02the model has remembered about you that you can actually audit
- 45:05and check in the settings section.
- 45:07That turns out to be very useful for problems like this where you
- 45:11want it to do a repetitive task and you want it to have a sense
- 45:14that it's done the task before. Even so, I still find I want to
- 45:20be precise about each of the assets I needed to process if I
- 45:24need that. And that is one of the reasons
- 45:26why I do tend to favor long prompts that I will keep in a
- 45:29Notion page or keep elsewhere that I can just copy and paste
- 45:33in as needed because I don't want it to forget anything.
- 45:36I don't want to go to that trouble of writing out that
- 45:39prompt again. I just wanted to remember every
- 45:42single thing and do it again. And I wish that I had an answer
- 45:46for you, that these were going to be flexible deep memory
- 45:51models that just would remember that you did exactly like this
- 45:53and never forget that step. We're just not there yet.
- 45:57And so prompts are part of how we bridge that gap.
- 45:59One of the problems that I have is I like to try prompts on
- 46:04different models to see the results and compare the results.
- 46:08I think it's it leads to a lot of creative thinking and it
- 46:14becomes a a real burden and an obstacle because I've interacted
- 46:19for 20 minutes with Model A and now I want to go to Model B and
- 46:26I've got to start all over again.
- 46:29There is a way to make that slightly less painful.
- 46:32So what I like to do is if I want to transition, I like to
- 46:36ask the model I've been chatting with.
- 46:38Could you please give me a very detailed summary of our
- 46:43conversation so far and make sure it's as clean and clear as
- 46:48you possibly can make it? And then it will do that and it
- 46:51will give me a great summary of the conversation.
- 46:52I then pull that summary into a new model conversation I'm
- 46:56starting and say, here's where we're at right now.
- 46:59I would love to continue this conversation with you.
- 47:02And this is what I'm looking for.
- 47:03And it's still a little bit painful, but it's it's less
- 47:06painful than it would be otherwise.
- 47:08Which model or which company do you gravitate towards the most?
- 47:14You have access to everything. What do you use the most?
- 47:17The memory feature is one of the most powerful product features I
- 47:24can remember on ChatGPT because I find that the fact that it
- 47:28remembers something about me drives a recursive behavioral
- 47:32loop for me. I'm very aware of it, right?
- 47:34Like I've I've worked in product for a long time, I know what
- 47:36they're doing, but it still works because I find that having
- 47:40a model that remembers me a bit is super, super helpful.
- 47:43And so Chet GPT drives a lot of interest for me.
- 47:45O3 is a daily driver for me. I pick it up by default, but
- 47:50that doesn't stop me from going other places.
- 47:53Like when I am working on a complex piece of writing, I will
- 47:58use perplexity. I will use both of the new
- 48:01Claude models, Opus 4 and Sonnet 4I will sometimes go to Gemini
- 48:052.5 Pro. And so I almost look at those as
- 48:08like additional pieces that I want to go to for specific
- 48:11things. Sonnet 4 is great for writing.
- 48:15I love it. Opus 4, I love the way it does
- 48:18like very thoughtfully considered reading and research.
- 48:22There's something qualitative about it that's very strong
- 48:25there. And so even though I end up in
- 48:29Chat GPTA lot, that doesn't stop me from reaching my fingers into
- 48:34the rest of the ecosystem and and grabbing what's useful.
- 48:37I also really like ChatGPT in general.
- 48:40They make it pretty easy. And that memory feature, you
- 48:44know, you start, you start with your prompt and you feel like it
- 48:49has so bizarre to say this. It has like this intuition of
- 48:53what makes sense for you. Right.
- 48:57And that sense of being recognized, I think is very
- 49:02powerful from a product experience perspective and
- 49:04people respond to it. Marketers talk about
- 49:07personalization and usually personalization is well, we've
- 49:12watched their shopping cart and in the past they've bought XYZ
- 49:16product and so we'll recommend the next product that they'll
- 49:20really, really like or the next movie or whatever.
- 49:22But we're talking here about a level of subtlety with
- 49:26personalization that's like light years beyond the typical
- 49:30marketing personalization that it that we know of.
- 49:33It really is and it's going to be super interesting in the next
- 49:366 months or 1818 months to see how the product platform evolves
- 49:42for ChatGPT as they build on this memory feature and they add
- 49:45more you know new models, etcetera.
- 49:48I, my sense is especially as they lean into the partnership
- 49:52with Shopify, they're going to learn more into commerce.
- 49:55They're going to be opportunities for
- 49:57personalization with commerce that we've never seen before.
- 49:59But we'll just have to see how that evolves.
- 50:01Should I pay the $200 a month for ChatGPT pro?
- 50:07I know you do, but should I? Is it worth it or should I just
- 50:10pay the 20 bucks a month that I pay right now?
- 50:13That depends on the kind of user that you are.
- 50:17And so I have seen the article that came out, I think it was
- 50:20this week that basically said ChatGPT has done such a
- 50:24phenomenal job pushing value down the chain to the free tier.
- 50:28Why would we pay it all? Because it's it's so impressive.
- 50:31And I think for a lot of average daily use, that is the correct
- 50:36assessment That is true for me. I want, it's not, it's not even
- 50:41just that I want it so I can test it and show it to people.
- 50:43It's that I want to have no token limits and no usage limits
- 50:49on the smartest models out there because I find myself doing
- 50:54better with my own brain if I have the smartest thinking
- 50:57partner possible. And so we've talked a lot around
- 51:00the edges of what sort of thinking and intelligence means.
- 51:02That's probably a conversation for another day, but from a
- 51:07economics perspective, if I have a thinking partner like that and
- 51:11I push the edges like that and it helps me make one or two
- 51:14better decisions in a given month, the ROI is off the
- 51:17charts. At $200 a month, it's very, very
- 51:21easy to do that math. And so I think it depends on
- 51:23what you're looking for it to achieve.
- 51:25I want the best possible results from the thinking because if I'm
- 51:30spending time. Pay the 200.
- 51:32You'll get O3 Pro and it's, it's qualitatively better in a way
- 51:38that you will notice it. It has a resonance to it where
- 51:41like the insights it has stick in my head and I'm like chewing
- 51:46them over in a way that I haven't had with other models,
- 51:48which is super interesting. Well.
- 51:49I'll have to try it. Well, with that, we are out of
- 51:52time. A huge thank you to Nate Jones.
- 51:56Thank you so much for taking your time to be with us today.
- 51:59It's so valuable for us when you're here.
- 52:02It was such a delight. I enjoyed it.
- 52:04A tremendous thank you for having me, Michael.
- 52:06I'm glad we got to talk about prompting.
- 52:08I felt like we could have gone on for hours because it's such
- 52:10an interesting topic, but I think we really got to a lot of
- 52:13cool stuff over the course of this 60 minutes together.
- 52:16And thank you to everybody who watched.
- 52:17Now, before you Go, subscribe to the CXO Talk newsletter so you
- 52:23can be part of our community. We have amazing shows coming up.
- 52:27We have the chief technology officer of AMD coming up, and we
- 52:34have all kinds of amazing people.
- 52:36So go to cxotalk.com, subscribe to the newsletter, and we'll see
- 52:42you again next time. Thanks so much everybody and
- 52:44hope you have a great day.