Latest / In The Loop / What Is Context Engineering And Why Should You Care?

Transcript

What Is Context Engineering And Why Should You Care? · In The Loop

• 0:00
  The term context engineering has become the biggest buzzword in
• 0:02
  AI in the past 10 days, and it's a word that if industry leaders
• 0:07
  are to be trusted, you must now start to learn, with many people
• 0:11
  calling it the next big evolution from prompt
• 0:13
  engineering. In very simple terms, prompt
• 0:16
  engineering is what you ask an AI model to do for you, whereas
• 0:19
  context engineering is what a model knows when you ask it
• 0:23
  something. This shift reflects very hard
• 0:25
  one lessons when building AI agent systems.
• 0:28
  You can have something that has all the bells and whistles, but
• 0:31
  if it doesn't give the right context, it can never deliver
• 0:34
  good outcomes for people. In today's episode, we're going
• 0:36
  to discuss the evolution from prompt engineering to context
• 0:39
  engineering, and I'll breakdown what context means when it comes
• 0:42
  to AI. And finally, I'll actually give
• 0:44
  a breakdown of some of the real things that you as a person are
• 0:48
  going to start noticing as a result of improved context
• 0:51
  engineering. This is in the Loop with Jack
• 0:53
  Halton. I hope you enjoy the show.
• 1:06
  So I'm going to assume most of you already know what prompt
• 1:08
  engineering is, but just just for the sake of the episode, I'm
• 1:10
  going to spend 20 seconds on it. Prompt engineering is a phrase
• 1:14
  that gained mainstream adoption and traction generally after the
• 1:18
  launch of ChatGPT in about late 2022.
• 1:21
  And this is because millions of people are suddenly trying to
• 1:24
  figure out how to talk to an AI system.
• 1:27
  It was one of the very early job roles that were advertised for
• 1:29
  by big companies. Prompt engineering became a
• 1:32
  skill set that people went into courses about generally became a
• 1:35
  really important skill. And engineering a prompt
• 1:37
  basically meant using trial and error and intuition really to
• 1:42
  find a set of I guess instructions and also the way
• 1:44
  the instructions are worded to make the model do what you want
• 1:48
  it to do. A really famous example of this
• 1:50
  was the very famous prompt that literally just said, let's think
• 1:54
  step by step. And this led to train of
• 1:56
  thought, which is a form of AI reasoning, which typically
• 1:59
  improves the performance of how an AI model performs when you
• 2:02
  just ask it a simple question because it's making itself a
• 2:05
  plan and following it step by step.
• 2:07
  However, as the use cases for AI have become increasingly
• 2:10
  advanced, so have the steps and the tasks that an agent must
• 2:16
  follow to actually, I guess, deliver value today.
• 2:18
  And as a result, there were very clear limitations with prompt
• 2:21
  engineering because early prompt engineering is essentially the
• 2:25
  A1 shot attempt. So meaning you write a single
• 2:28
  prompt, which could maybe include a few examples or
• 2:31
  instructions, and the model then give an answer as a result.
• 2:33
  But many tasks require dialogue with users.
• 2:37
  It requires the agent to memorize things.
• 2:39
  It requires the agent to use tools, so integration into
• 2:42
  separate systems to capture data from those systems.
• 2:45
  And so, for example, asking a model to, say, answer this
• 2:49
  question by searching the web and then write a summary can't
• 2:52
  be done in a single one prompt attempt.
• 2:55
  The model needs to take many different actions, like doing a
• 2:58
  search and remembering what it found and then doing something
• 3:00
  with that information. And many builders in the space,
• 3:03
  including ourselves, do things called prompt chaining, which
• 3:06
  essentially means chaining prompts together to give the
• 3:08
  model more context to every single step.
• 3:11
  So take a step and then follow another step and use the
• 3:14
  information from that previous step.
• 3:16
  And there was many different innovations that include RAG, so
• 3:19
  retrieval, augmented generation, including what's called React,
• 3:23
  So reasoning and action, which essentially are ways of getting
• 3:28
  a model to prompt and capture subsequent information and use
• 3:32
  that as context to provide a better answer.
• 3:34
  The point being is that it's not actually just the instruction
• 3:37
  sent to a language model that matters.
• 3:39
  It's actually the context the language model has and how it
• 3:42
  carries context throughout many different steps.
• 3:46
  And this is where context engineering really starts to
• 3:48
  take center stage. Because context engineering is
• 3:51
  all about deciding what a language model should know
• 3:53
  before it speaks and answers a human.
• 3:55
  Not just what you ask, but what data or tools or memories,
• 3:59
  instructions, and history should be fed into that system to guide
• 4:03
  its answer. And the term itself became
• 4:05
  largely popularized by Shopify CEO very recently on Twitter.
• 4:10
  He said, I prefer the name context engineering to prompt
• 4:13
  engineering. It better describes the core
• 4:15
  skill set, which is giving the model the context it needs to
• 4:18
  plausibly solve the task. And there are other big names
• 4:21
  out there like Andre Kapathy who was one of the Co founders or
• 4:24
  let's say founding team members of open AI.
• 4:27
  And he's very famous and even coined the term vibe coding.
• 4:31
  He described it more as an art and science of filling the
• 4:34
  models context window just enough to make the next step in
• 4:38
  its sequence work, but not too much and not too little.
• 4:42
  It's just the right amount of information.
• 4:43
  So you can almost consider context engineering like
• 4:46
  orchestration. You're orchestrating an agent
• 4:48
  over many subsequent steps and trying to control what
• 4:52
  information and context it should have after each step and
• 4:55
  how it should use that context in the subsequent one.
• 4:58
  Let's go one level deeper and actually unpack what counts as
• 5:01
  context when it comes to an AI model.
• 5:04
  Because every piece of software built over a language model,
• 5:08
  that's mindset, that's every provider is essentially this
• 5:10
  fantastic piece of software that is dictating the context and the
• 5:15
  instructions sent to a model and how that information is surfaced
• 5:18
  to a user. And that is where all the hard
• 5:20
  work comes from because it's flipping difficult to get it all
• 5:23
  to work. And you'll see why in a second.
• 5:26
  There's many layers to context. So you've got the first piece of
• 5:29
  context, which is the immediate prompt.
• 5:31
  So this is the direct question or instruction by a user, for
• 5:36
  example, creating a marketing strategy.
• 5:39
  And this is where prompt engineering has typically been
• 5:41
  focused #2 you've got systems instructions, which essentially
• 5:45
  a system message sent to the large language model to set a
• 5:50
  behavior. So you are an expert travel
• 5:52
  panel. The second layer is called
• 5:53
  system instructions and essentially this is where you
• 5:56
  send a system message to a large language model such as you are
• 6:00
  an expert travel planner. So this is the context that
• 6:03
  influences say the tone, the scope, and this alone can be
• 6:06
  quite detailed. We've got an entire agent
• 6:08
  builder around these which dictates how to behave, not to
• 6:12
  behave, guard rails, policy, personality, how it should
• 6:15
  structure its responses, but that's all system messages.
• 6:18
  Again, the software built on top of these things a highly complex
• 6:23
  in advance just to make sure the right thing is sent at the right
• 6:25
  time with the right amount of context.
• 6:27
  So third piece of context is the conversation history, which is
• 6:30
  often called the short term memory.
• 6:32
  So in a conversation which they ChatGPT, you want parts of that
• 6:35
  chat to be included in every single prompt that goes to the
• 6:39
  LLM. So when you ask a question, it
• 6:41
  has the context of all previous conversations you've had in that
• 6:45
  thread again, so you might say as I was speaking about earlier,
• 6:50
  and therefore it knows what earlier means.
• 6:52
  And that's because every time a message by a user is sent
• 6:56
  ChatGPT is a piece of software also takes the message history,
• 7:00
  condenses it down and sends it to the language model.
• 7:03
  So it then has that context of the conversation.
• 7:05
  The 4th type of context that's absolutely critical is long term
• 7:09
  memory or persistent memory. This can be a summary of
• 7:12
  previous interactions. This could be stored facts about
• 7:15
  a user, such as my preferences or information about my profile
• 7:19
  or even previous conversations between different threads.
• 7:23
  So it's that context that persists across different
• 7:25
  messages. So it knows that you work in a
• 7:29
  certain job, you like response in a certain way, you hate
• 7:32
  things that sound like this. We're actually as mindset
• 7:35
  launching an entire long term memory layer next month, which
• 7:39
  enables our customers to dictate what an agent should remember.
• 7:43
  Then you've got retrieving external information.
• 7:45
  So this might be both RAG or integrations into third party
• 7:48
  systems. So sometimes you'll ask
• 7:49
  questions that will require context of external documents or
• 7:53
  code snippets or videos. And so the software that you'll
• 7:56
  be interacting with will then call that information, bring it
• 8:00
  into the prompt that is sent. So your message plus that
• 8:03
  information from an external data source sent to the LLM, and
• 8:07
  then the LLM will use that to answer your question.
• 8:09
  Another example could be a customer support bot that pulls
• 8:12
  in user's account data whenever it answers a question about
• 8:15
  billing. You've then got another layer
• 8:16
  down, which is tool definitions. So if the models can use tools
• 8:19
  like say, making API calls or running calculations, to do
• 8:24
  that, it needs to understand how to actually call those tools or
• 8:27
  interact with external tools, and it must use the information
• 8:30
  it gets back to answer or support what it's doing.
• 8:33
  And so whenever it uses one of those tools, let's say
• 8:36
  summarizes all my WhatsApp conversations, you then want it
• 8:39
  to be able to retain the output of that query into WhatsApp for
• 8:43
  the next step in the process. So yeah, that's different layers
• 8:46
  of what context actually means in AI system.
• 8:48
  Again, it just sounds like a human that requires context to
• 8:51
  do good work, but it actually has very big implications on
• 8:55
  what it means for a genetic systems.
• 8:57
  And it's one of the big reasons we're launching a new
• 8:59
  orchestration system that enables our customers to really
• 9:03
  give models the right context at the right time.
• 9:06
  And I think in summary, and I'll explain why in just a second,
• 9:10
  simpler processes or sequential step by step processes are
• 9:13
  better than multi agent systems right now.
• 9:26
  We've talked about multi agent systems before in previous
• 9:28
  episodes, but this is where you have teams of agents doing many
• 9:31
  different things. And it's really relevant to this
• 9:34
  conversation because it's all about the concept of context
• 9:38
  engineering and really highlights why context is so
• 9:41
  important. So let me delve a level deeper
• 9:44
  by explaining the problem of having teams of agents doing
• 9:46
  many different things at once. There are many different use
• 9:49
  case ideas that people often think, oh, let's get a team of
• 9:53
  agents to do this. Let's get a team of agents also
• 9:56
  working on tasks in parallel to each other and then breaking
• 10:00
  down those tasks into subtasks and getting other agents to help
• 10:03
  with those subtasks. So the idea might be to have,
• 10:06
  say, Agent A break a problem into small little parts,
• 10:09
  delegate it to agent B&C, and then have a say combine all of
• 10:13
  their information into one answer.
• 10:15
  Because in theory it would just make the responses faster
• 10:18
  because you have people working, or so I say, agents working in
• 10:21
  parallel. And those agents could be
• 10:23
  specialized, breaking down tasks, for example, or
• 10:27
  summarizing information. So in theory it makes sense, but
• 10:31
  often it will fail because of coordination.
• 10:33
  And when we say coordination, what we actually mean is
• 10:37
  context, contacts not being shared between different agents.
• 10:41
  Now, there was a fantastic paper recently released by Cognition
• 10:44
  Labs again, the one of the labs behind what's called the Devin
• 10:47
  agent, one of the very famous coding agents before the new age
• 10:51
  coding apps emerged. And that paper was called Don't
• 10:55
  build multi agent systems. And to show why they actually
• 10:59
  tasked a set of agents to build a Flappy Bird clone.
• 11:02
  To accomplish this. The I created sub agents want to
• 11:06
  create the background graphics and another one to create that
• 11:08
  player bird, the thing that you're going to control.
• 11:11
  Now without the full context, the first sub agent might
• 11:14
  interpret the task a little bit different and generate a Mario
• 11:18
  like background instead of a Flappy Bird pipes background.
• 11:21
  Now they're very similar but slightly different.
• 11:24
  But the second agent might then produce a bird that doesn't
• 11:26
  therefore fit with the style of graphics.
• 11:30
  So when these two agents merge the outputs to combine it into a
• 11:33
  single game, it obviously doesn't work.
• 11:38
  It feels like separate products. And the root cause of this is
• 11:41
  that the sub agents lack the full context of the original
• 11:44
  objective and the work of other agents going on at the same
• 11:48
  time. Now the straightforward solution
• 11:50
  you might say would just to be give all these sub agents all
• 11:52
  the information they need, like the information on the task, the
• 11:56
  description, any decisions of the agents have made so they can
• 11:59
  go and do a good job. In practice, that means when
• 12:01
  agent A, say, delegate something, it should pass along
• 12:04
  not just the sub task, but also all of the information, the
• 12:08
  conversation history, everything that's happened up until that
• 12:11
  point, and all the decisions that led there.
• 12:14
  However, even sharing the entire context isn't enough sometimes
• 12:18
  because the sub agents then might work with full knowledge
• 12:21
  of say the main objective, yet still produce something that's
• 12:24
  incompatible because they may make assumptions throughout that
• 12:28
  process that conflict with each other.
• 12:30
  Like 1 deciding that the game should have a cartoon style and
• 12:33
  the other a realistic style. Again, you'll notice that every
• 12:36
  single decision or action by an agent will always have a knock
• 12:40
  on effect. And so overall what Cognition
• 12:42
  Labs have claimed and believed after much testing is that
• 12:46
  single agent architectures, which means that one agent
• 12:50
  follows sequentially a few steps is far easier to get a fantastic
• 12:54
  output than multi agent systems. And it's exactly as I said, why
• 12:57
  mindset is enabling this exact use case making enabling our
• 13:01
  customers to dictate all the different steps and separate
• 13:05
  integrations and the long term memory that an agent should
• 13:08
  carry throughout a conversation and essentially what it should
• 13:11
  do with each part of its data and output at each step.
• 13:14
  And they're not the only ones that reported this because
• 13:16
  Anthropic did a lot of research into multi agent systems for
• 13:19
  their research assistance and hire the exact same problem.
• 13:24
  We've personally found that same problem when trying to use multi
• 13:27
  agent systems in our own business.
• 13:29
  So for all the excitement of multi agent systems with teams
• 13:32
  of agents doing things and and even a single agent doing
• 13:35
  something really powerful, it all comes down to how well the
• 13:39
  software product on top of those large language models can prompt
• 13:44
  and give it context every single time it prompts.
• 13:47
  So I think this is going to be a growing area of innovation over
• 13:49
  the next sort of 12 months. And obviously I believe
• 13:53
  everything will be solved and multi agent systems will
• 13:55
  probably be possible in I'd say 12 months, but we can always
• 13:59
  half it and everything happens faster than we thought.
• 14:12
  So I guess the interesting question therefore, is as a
• 14:15
  listener right now, how does this impacts you?
• 14:18
  Like what should you know about this?
• 14:20
  Why is this relevant to you? But really simply, I think good
• 14:23
  context engineering, either as a user using just something simple
• 14:26
  as ChatGPT, or using a product that's an AI product.
• 14:30
  Good context engineering is where you're left with a sense
• 14:33
  of that was actually helpful. It also massively influences the
• 14:37
  types of experiences and features and options as a user
• 14:41
  that you're going to be given by products.
• 14:42
  So one of those are going to be personalization.
• 14:45
  The system is going to increasingly be able to remember
• 14:47
  everything about you, your preferences, and previous
• 14:49
  conversations. You're going to start to see
• 14:51
  more user interface and user experience patterns emerge.
• 14:55
  For example, as AI becomes integrated and say, apps,
• 14:58
  designers are going to have to come up with ways to give
• 15:02
  context to agents without users actually realizing it.
• 15:06
  For example, many chat interfaces have an upload file
• 15:09
  button, which enables you to upload a document.
• 15:12
  From a user's perspective, it's straightforward.
• 15:13
  I want to talk to a document. But actually what you're doing
• 15:17
  is giving the agent or the AI language model context.
• 15:21
  You're also going to have different modes of working.
• 15:23
  So even if you go on the ChatGPT app, you have the deep research
• 15:25
  prompt. If you're a pro user, you have
• 15:28
  the Google search, you now have study with me.
• 15:30
  There's many different options. And what that does is
• 15:33
  essentially give context to an agent that they built.
• 15:37
  And it might just be a single agent architecture with a series
• 15:39
  of steps. It gives it context on how it
• 15:42
  should behave in that situation. Another big area is managing
• 15:45
  context in multi tenant environments.
• 15:48
  So many of our customers are technology providers, software
• 15:50
  providers who have say thousands of enterprise customers and each
• 15:55
  one of them have their own space in these platforms.
• 15:58
  How therefore does an agent or an AI retain conversations and
• 16:01
  contacts between all these people and stay secure?
• 16:05
  So it's an area we've been working really harder and we're
• 16:07
  excited to bring some out for. But this is the area that's
• 16:11
  really important because let's take a SAS platform in sales.
• 16:14
  It might integrate with every single salesperson's contacts.
• 16:19
  It's CRM, it's e-mail. So when a salesperson asks to
• 16:23
  say, let's say draft an e-mail, it'll just use all of that data
• 16:27
  seamlessly, very cleverly, magically and answer you.
• 16:31
  But all of that magic actually had to be done quite securely.
• 16:36
  I think the other one is going to be privacy and trust for you
• 16:38
  because I would be worried about what happens with all this data.
• 16:43
  They're capturing a lot of data on us now.
• 16:45
  I personally use it as a life coach as ever.
• 16:48
  I use it for almost any question that I've got.
• 16:50
  I will almost always go to ChatGPT or another AI system
• 16:53
  first. So I guess let's just conclude,
• 17:07
  early on we were obsessed with the model itself and how to talk
• 17:10
  to it with prompt engineering. And over time we've realized
• 17:13
  that an AI agent isn't just the model, it's the whole system
• 17:16
  that feeds it contacts and interpret its output and uses
• 17:20
  that output to inform its next set of contacts.
• 17:24
  So in a way, we're moving one level above in terms of
• 17:27
  abstraction. How do I get the model to output
• 17:30
  X? Response is more about thinking
• 17:34
  what information do I need to give it and guidance to get that
• 17:38
  best response. Anyway, I hope you enjoyed the
• 17:41
  episode, I certainly did, and I'll see you next week.