Latest / The Adults In The AI Room / The Earley AI Podcast with Seth Earley - Episode #57 Understanding AI and Data Readiness with Camden Swita

Transcript

The Earley AI Podcast with Seth Earley - Episode #57 Understanding AI and Data Readiness with Camden Swita · The Adults In The AI Room

• 0:00
  SPEAKER_02: Good afternoon, good evening, good morning, depending upon your time zone.
• 0:03
  SPEAKER_02: Welcome to the Early AI Podcast.
• 0:05
  SPEAKER_02: My name is Seth Early.
• 0:07
  SPEAKER_02: We're without Chris Featherstone today, but I'm really excited to introduce our guest for today to discuss the critical importance of agentic oversight and an AI, AI mechanisms and processes to monitor complex AI stacks and complex technology stacks.
• 0:24
  SPEAKER_02: The significance, of course, organizing, curating data sources, and again, being able to make sure that we can do troubleshooting, traceability, monitoring of AI systems.
• 0:34
  SPEAKER_02: It's something that's fairly overlooked.
• 0:36
  SPEAKER_02: And as our systems get more and more complex, it gets more difficult to actually define exactly where problems might be.
• 0:44
  SPEAKER_02: Our guest today is an expert who seamlessly bridged the gap between journalism and technology.
• 0:48
  SPEAKER_02: He carved out a niche niche in the AI space.
• 0:51
  SPEAKER_02: He currently is the head of AI and ML innovation at New Relic, a company that is renowned for observability and dedicated to empowering organizations by monitoring AI-enabled systems.
• 1:04
  SPEAKER_02: With a rich background in product management and a key player in various technology roles, he's been instrumental in pushing New Relic's capabilities forward.
• 1:12
  SPEAKER_02: Camden Sweda, welcome to the show.
• 1:17
  SPEAKER_01: So happy to be here.
• 1:18
  SPEAKER_01: It's quite the intro.
• 1:19
  SPEAKER_01: Looking forward to this conversation.
• 1:20
  SPEAKER_02: Great, great.
• 1:21
  SPEAKER_02: So a lot of times we like to start with kind of misconceptions in the industry.
• 1:25
  SPEAKER_02: And when it comes to AI environments and especially the work that you do within monitoring, you know, tell me what do you run it up against in terms of lack of understanding, misunderstanding, myths, you know, thing.
• 1:38
  SPEAKER_02: What do you how do you have to educate the executives and the customers that you talk to?
• 1:43
  SPEAKER_02: And what are the things that they don't necessarily understand when you kind when you walk in the door, your organization walks in the door?
• 1:49
  SPEAKER_02: What are you seeing in the industry?
• 1:52
  SPEAKER_01: Sure, sure.
• 1:52
  SPEAKER_01: Yeah.
• 1:52
  SPEAKER_01: Um, yeah, makes sense.
• 1:54
  SPEAKER_01: I think there's still a lot of magical thinking going on around large language models, generative AI assistance in general.
• 2:01
  SPEAKER_01: Um, you know, there I think there's still the perception that, you know, especially the foundational models are sort of black boxes that are capable of doing multi-step reasoning with a high degree of accuracy right out of the box, and that they don't need any additional context to complete these tasks or that, you know, and and things of that nature.
• 2:22
  SPEAKER_01: So I think that the the thing that I still try to emphasize in any of these conversations with with executives or other people who are interested in investing in, especially agentic automation, is that context is still really king.
• 2:35
  SPEAKER_01: Uh, you still really have to think hard about how you're going to organize and serve up that additional information and context to the assistant to make sure that it's uh got the right information to make the right decisions and give the whoever the end user is the right insights.
• 2:51
  SPEAKER_02: Yeah, and I think you're you're right in that uh people still think, oh, you just plug this in in many cases.
• 2:57
  SPEAKER_02: They're starting to learn the hard way that no, you don't plug it in.
• 3:00
  SPEAKER_02: You do need that data foundation.
• 3:02
  SPEAKER_02: And then building that data foundation requires a lot of a lot of uh investment, but more frequently you can use large language models and age approaches to actually fix the data.
• 3:13
  SPEAKER_02: That's some of the experience I've had.
• 3:14
  SPEAKER_02: Have you seen organizations doing that actually using the models uh themselves, say programmatic approaches or templated uh uh prompting to fix data?
• 3:24
  SPEAKER_01: Oh, sure.
• 3:24
  SPEAKER_01: Yeah.
• 3:24
  SPEAKER_01: I mean, everything from you know, there's there's a few different ways in which, you know, the the data may need to be fixed, right?
• 3:30
  SPEAKER_01: You can use the large language models, of course, to synthesize additional context or alignment data for other models.
• 3:37
  SPEAKER_01: So, in other words, you want model A to be better at writing queries in this specific query language.
• 3:43
  SPEAKER_01: Well, maybe you'll use some other model to write a bunch of examples of queries in there and natural language examples.
• 3:50
  SPEAKER_01: You also see it, of course, with um prompt adjustment after the user uh enters something.
• 3:56
  SPEAKER_01: You see this with some of the uh foundational language models, uh chat platforms, especially some of the text-to-image generators, where they'll take the user prompt and the model will sort of rewrite it in a way that's gonna be a little bit more successful for the text to image uh writer.
• 4:12
  SPEAKER_01: And then uh, you know, you can even use these models as sort of judge models, right?
• 4:16
  SPEAKER_01: So uh oftentimes what you'll see is you'll see a customer using a ling a large language model that for whatever reason they've determined is a good enough quality and kind of economical enough for them to use in production.
• 4:28
  SPEAKER_01: And maybe they'll use a more expensive and better language model as a judge LLM that will sort of look at at least some subset of the responses that that first large uh language model is giving and assess whether or not it thinks the answer is good or not or meets the criteria that they've set for it.
• 4:44
  SPEAKER_01: So two different ways, right?
• 4:46
  SPEAKER_02: And what's interesting is you said that you can use a more expensive language model to kind of look at the results of a less costly model.
• 4:54
  SPEAKER_02: So that kind of saves the money by leveraging them appropriately.
• 4:58
  SPEAKER_02: So this may be it may not be the best language model, but it produces pretty good results.
• 5:02
  SPEAKER_02: But now you're using another language model, and then of course, modular rag, which is using different algorithms for retrieval, using can handle that through, say, fusion types of uh programs.
• 5:13
  SPEAKER_02: And I don't want to get too deep into the weeds there, but but um you know this this whole approach of the modular retrieval augmented generation is giving us a lot of capabilities, but they they are agentic approaches.
• 5:24
  SPEAKER_02: In other words, they will cause the LLM to call out and an API have an API call out to talk to another system to produce information.
• 5:33
  SPEAKER_02: Now, what you're starting to see is a very complex environment where there's so many dependencies between one system and another, and there's so many interrelationships that it does become a very complex beast.
• 5:45
  SPEAKER_02: So let's talk about AI ops.
• 5:48
  SPEAKER_02: And when you start thinking about AI ops, first of all, uh, why don't you break that down and define that for us?
• 5:53
  SPEAKER_02: And then we can talk about how that fits in with some of these more complex environments and heterogeneous uh agentic environments that start to get really complex.
• 6:03
  SPEAKER_01: Sure, sure.
• 6:04
  SPEAKER_01: Yeah, I mean, uh, I guess uh, you know, starting from the beginning here, when I think about AI ops, right, there's there's obviously it can be a broad set of definitions there, and there's an official definition and there's a Gardner definition and all these different things.
• 6:17
  SPEAKER_01: But, you know, I I really just think about the the the automation of tasks that an IT operations individual typically historically may have performed to restore a system to its normal state after some problem is detected, right?
• 6:33
  SPEAKER_01: So some problem is detected, uh, the mitigating action is identified, the mitigating action is taken and it's confirmed that the system has returned to a good state.
• 6:43
  SPEAKER_01: You can automate all of that, right?
• 6:45
  SPEAKER_01: You have you have AI ops.
• 6:46
  SPEAKER_01: Now, obviously, embedded in that is is is tons of complexity to your point.
• 6:51
  SPEAKER_01: And uh, you know, in a in a modern software toolchain or SDLC or ITSM toolchain, there may be four, five, six or more different kind of platforms or tools involved in all of that, right?
• 7:04
  SPEAKER_01: And maybe IT operations folks are using one and engineers are using another, and SREs are using something else, and support engineers are using something else.
• 7:13
  SPEAKER_01: And so, you know, so your point, it's it's when we talk about you know automation of all of that, especially automation with some amount of reasoning that you get by using modern day foundational large language models, you're really talking about kind of an enterprise data issue still, which has always been the problem, and then an integration issue still, which has always been the problem with automation across those platforms.
• 7:37
  SPEAKER_01: Now, I will say the introduction of generative AI agents has sort of, in my opinion, begun to change the integration game a little bit.
• 7:45
  SPEAKER_01: And by that I mean we can make natural language APIs now, where every one of these platforms is going to have its own assistant.
• 7:53
  SPEAKER_01: They already do, right?
• 7:55
  SPEAKER_01: Now, those assistants can send natural language prompts to one another and get back the data that they need, the payload that they need.
• 8:02
  SPEAKER_01: Does that introduce some additional potential for error or misunderstanding?
• 8:06
  SPEAKER_01: Yes, but it also simplifies that maintenance and upkeep and construction of all of these APIs as well.
• 8:14
  SPEAKER_02: So yeah, that's really interesting.
• 8:16
  SPEAKER_02: So I I also think of the area of AI DevOps, right?
• 8:21
  SPEAKER_02: So where you're building those systems and testing those systems and iterating.
• 8:25
  SPEAKER_02: So AI ops is a little bit different in that you're saying you're supporting your your sort of maintenance and keeping the lights on using AI versus AI DevOps is really building the algorithms and building the systems and then testing and iterating and uh doing impact analysis and so on.
• 8:43
  SPEAKER_02: Do you see that those as two distinct areas are just very interrelated?
• 8:47
  SPEAKER_02: I mean, obviously they're there's they're different uh aspects of similar uh processes.
• 8:53
  SPEAKER_01: Sure, sure.
• 8:54
  SPEAKER_01: Yeah, definitely, definitely related.
• 8:55
  SPEAKER_01: I mean, the you know, the obviously problems can occur and and mitigating actions for for problems need to be identified throughout the construction process of of modern software.
• 9:05
  SPEAKER_01: Software.
• 9:06
  SPEAKER_01: And I think, of course, one of the areas where we're seeing kind of the the the most effective adoption, you might say, is in this space of writing software and and DevOps, right?
• 9:17
  SPEAKER_01: And increasingly you see a greater percentage of accepted code at a lot of organizations coming from a large language model.
• 9:25
  SPEAKER_01: I think that'll get I think that'll obviously that will continue.
• 9:27
  SPEAKER_01: I have no doubt that that the percentage of that code will increase.
• 9:31
  SPEAKER_01: I think it will be interesting too to see how as these foundational models or as the frameworks around them get better at writing software, not just code, but actually building out the software as a total package.
• 9:47
  SPEAKER_01: You know, are they gonna get good at understanding like software architectures?
• 9:51
  SPEAKER_01: Are they gonna get good at just really sort of like going from zero to a working application with infrastructure and all that in one go?
• 9:59
  SPEAKER_01: And I guess what I'm kind of curious to see is, you know, historically, or I mean, over the last 10, 15 years, we've moved to this very microservice-y kind of architecture as the norm.
• 10:09
  SPEAKER_01: And I wonder as we as engineers release more and more of the actual writing and and management of the code itself to these large language models, is there as much reason uh to have such a microservice-y architecture?
• 10:26
  SPEAKER_01: In other words, if you have a foundational language model that is very good at writing code, and every time you need to make an update, it can just regenerate the entirety of your code base.
• 10:36
  SPEAKER_01: You don't necessarily need to outsource all of these different services, right?
• 10:40
  SPEAKER_01: As microservices.
• 10:41
  SPEAKER_02: Anyway, not to go on a tangent, but that is interesting because it does call out, you know, how we're architecting these environments.
• 10:49
  SPEAKER_02: And yes, there has been a great dependencies on on uh microservices and and the integration of different systems using microservices.
• 10:58
  SPEAKER_02: And you're saying that that might not necessarily be as important down the road.
• 11:03
  SPEAKER_01: Yeah, yeah.
• 11:03
  SPEAKER_01: I mean, I I think I mean what, you know, of course, we'll see, but but you know, one of course, you as you know, one of the main reasons why microservice architecture has has become popular in the first place is individual organizations or software teams don't necessarily want to own and manage all of the different aspects of their system.
• 11:22
  SPEAKER_01: You know, maybe that's the checkout process or the payment processing or the authentication or whatever it is.
• 11:28
  SPEAKER_01: But at the end of the day, every single one of those services that we kind of outsource because it's toilsome for a human to manage all of that and risky.
• 11:37
  SPEAKER_01: Well, if these foundational language models are increasingly good at just generating every single one of these services with every deployment, or maybe it does it in chunks or something, right?
• 11:49
  SPEAKER_01: No need to waste the energy regenerating all of them.
• 11:52
  SPEAKER_01: But if all that toil is sort of now managed by an automated system run by a foundational language model, then yeah, why would you need to depend on those external parties for these individual microservices to manage your system?
• 12:07
  SPEAKER_01: Why not just have the foundational language model that you've chosen to manage your code, just regenerate that payment processor module, that whatever it is, you know, every single time?
• 12:17
  SPEAKER_01: And it's interesting because you know, in in some ways it could mean like a little bit more uh uh of shift back to like a monolithic code base, whereas you know, every time you every time you you you make a significant update or an update, your entire pillar of of code that represents your entire system is just getting re-deployed.
• 12:37
  SPEAKER_01: Um, and and so I wonder myself, like, are are we see are we potentially eventually going to see as these foundational language models take over more of this work, a shift away from microservice and external dependencies to everything's in-house, it's a monolithic code block.
• 12:53
  SPEAKER_01: And every time you deploy, it's just you have this cheap and effective language model that's regenerating it all.
• 12:59
  SPEAKER_02: That's really interesting.
• 13:01
  SPEAKER_02: And in a way, it almost uh addresses one of the other challenges.
• 13:05
  SPEAKER_02: I mean, when you when you're uh deploying, and maybe you could just talk a little bit about not a sales pitch, but talk a little bit about what your your solutions give us the the 30,000 foot.
• 13:14
  SPEAKER_02: Uh I guess I do want the sales pitch, but this isn't about selling, this is about educating.
• 13:20
  SPEAKER_02: But um, you know, when you go into an organization and you you you talk about the complexities, you know, what is your 30,000 foot sales pitch for what you do?
• 13:30
  SPEAKER_02: Because you're using agents, you're deploying uh agent approaches to monitor lots of different systems and processes.
• 13:37
  SPEAKER_02: And in a way, if you, as you say, if you have fewer dependencies and you have more of a monolithic uh code uh uh stack, there's fewer dependencies and fewer places where things can go wrong.
• 13:47
  SPEAKER_02: So, in a way, that's kind of uh an evolution away from the increasing complexity, but it seems that we're going in the other direction, we're going into greater complexity.
• 13:56
  SPEAKER_02: So, do you want to talk a little bit about that evolution and then how your approach to monitoring, troubleshooting, uh you know, diagnosing those complex environments uh has evolved or is evolved?
• 14:09
  SPEAKER_01: Sure, sure.
• 14:09
  SPEAKER_01: Yeah, makes sense.
• 14:10
  SPEAKER_01: I mean, I think the where I would start here is you know, with the rise of using large language models in modern software stacks, you know, New Relic itself has is evolved to treat that layer of the stack.
• 14:25
  SPEAKER_01: In other words, the the generative AI layer of the stack as a first-class citizen is really kind of part of our APM solution, right?
• 14:33
  SPEAKER_01: So New Relic historically has been known for application performance monitoring and in a lot of ways monitoring the performance of you know the generative AI layer of all that is much the same.
• 14:43
  SPEAKER_01: You know, you want to understand just at a very basic level, and then we'll spiral into the complexities here.
• 14:49
  SPEAKER_01: At a very basic level, what is the latency around this layer of your of your service stack?
• 14:56
  SPEAKER_01: What is the cost?
• 14:57
  SPEAKER_01: What is the, you know, what's the throughput there?
• 15:00
  SPEAKER_01: Are there, you know, any logs that are describing errors that you might need to know about and and so on and so forth?
• 15:06
  SPEAKER_01: So that in and of itself is is important for for I think any organization adopting large language models as part of its of how it serves up its products to customers to really treat that LLM portion as another layer, just like they would treat any service or browser app or mobile app or host you know in in their stack.
• 15:29
  SPEAKER_01: And of course, that's that's kind of square one as far as this AI monitoring goes.
• 15:34
  SPEAKER_01: But of course, organizations become more sophisticated in how they're using, especially generative AI to uh create value for their customers.
• 15:43
  SPEAKER_01: That's where a lot of the fun and games begin, right?
• 15:46
  SPEAKER_01: Because if you think about what a lot of these AI agents are doing behind the scenes, especially using the more popular frameworks, is you know, there's usually an interface with a user, and then there's usually a some kind of like parent or router agent that is connected to the large language model that is receiving input.
• 16:03
  SPEAKER_01: And that's, you know, let's for simplicity just say it's a prompt.
• 16:07
  SPEAKER_01: And based on that prompt, it's deciding what to do.
• 16:09
  SPEAKER_01: It's saying, I think the user wants this, and and I, you know, I need to go figure out how to do that.
• 16:14
  SPEAKER_01: And in order to help it know how to do that thing, what a lot of organizations, New Relic included, to create this kind of agentic experience, we're giving these things individual tools, or sometimes they're called skills.
• 16:27
  SPEAKER_01: And these individual tools or skills are sometimes they're just another prompt that kind of like helps focus the large language model's attention in a specific domain on a specific task.
• 16:38
  SPEAKER_01: Sometimes they're specific queries that we want it to run.
• 16:41
  SPEAKER_01: So, in other words, you know, the user says, like, go get me logs for the last hour, and the parent agent says, Oh, I've got a logs tool for that.
• 16:50
  SPEAKER_01: And it goes and hits that tool, and that tool says, Okay, I've I have a query I know I can run to get logs for the last hour.
• 16:57
  SPEAKER_01: I'm gonna go do that and return the results back up to the parent agent who will take that back to the user.
• 17:02
  SPEAKER_01: And maybe that parent or routing agent does some additional work on it to kind of summarize them or analyze them or whatever.
• 17:08
  SPEAKER_01: And uh, you know, you can imagine that that kind of architecture can get quite complex quite fast, especially because some of the more valuable use cases and the things that save a lot of users time are when you start chaining the use of those individual tools together.
• 17:27
  SPEAKER_01: And so you say something like, you know, I want logs for the last hour, and also go find, go find like correlated like anomalies in my service metrics or something like that.
• 17:37
  SPEAKER_01: And so this assistant starts piecing these things together.
• 17:40
  SPEAKER_01: And in order to understand what's happening in your with your agent, and and and if you're a uh an agent engineer, as I've I've heard them referred to recently, or uh, you know, an ML engineer, and you want to really understand where things are potentially going wrong or where there's room for improvement, what you're really talking about is being able to have visibility in this actual, like essentially like a round trip trace of this input to routing to tool use to response and so on.
• 18:11
  SPEAKER_01: And so it's in it's a game of telephone.
• 18:13
  SPEAKER_01: And so what you really have to understand is what's going into each of these steps, what context was there, what, and then what was the output that was generated as that step, and what was and and what you know, what additional context was retrieved, and how did this happen all along the way?
• 18:28
  SPEAKER_01: And you have to, you know, you're you're you're on the you're on the lookout, of course, for for standard software type errors, but you're also on the lookout for essentially almost like misunderstandings, right?
• 18:38
  SPEAKER_01: Just like any game of telephone, like things are misinterpreted, and the same applies here.
• 18:43
  SPEAKER_01: If you really want to debug why the assistant's giving poor answers to this specific type of preampt, you have to look at each of these steps and say, I think I see that it's maybe not getting like enough context about this specific topic here, right?
• 18:57
  SPEAKER_01: Um, it's fascinating, it's a really fascinating uh kind of evolution where it's almost like this like semantic layer on top of debugging that you really have to pay attention to.
• 19:08
  SPEAKER_02: Yeah, yeah.
• 19:08
  SPEAKER_02: And in some ways it makes it easier, as you say, because it's more understandable.
• 19:13
  SPEAKER_02: The inputs and the outputs are more understandable, but then there's more complexity and there's more more areas for misinterpretation, hallucination, incorrect answers, and correct interpretation, ambiguous queries, and then of course the data sources have to be uh the right data and the right structure and right quality and so on.
• 19:31
  SPEAKER_02: Uh so so talk uh what what is your understanding of modular rag?
• 19:36
  SPEAKER_02: So modular rag is something that we're increasingly seeing, you know, there's rag is just saying, I'm going to use my corporate data source for this query.
• 19:46
  SPEAKER_02: In other words, instead of relying upon the large language model for its understanding of the world, you're you're saying you can interpret the query, but go to the source to find the answer and then make it more conversational.
• 20:00
  SPEAKER_02: But but that in-between piece where you're retrieving, you know, I've seen some people use just full text search, right?
• 20:07
  SPEAKER_02: And that's going to give you as much uh value as regular full-text search will.
• 20:11
  SPEAKER_02: And if you ask anybody on any corporate intranet how well search works, it's usually not very good, right?
• 20:17
  SPEAKER_02: I mean, unless your content is really good and well curated and so on.
• 20:21
  SPEAKER_02: So, but then we have more other iterations.
• 20:23
  SPEAKER_02: Um, do you want to talk a little bit about reg, advanced reg, modular reg, and then how that kind of fits in with your world?
• 20:31
  SPEAKER_01: Sure, yeah.
• 20:32
  SPEAKER_01: I mean, retrieval augmented generation is still probably the leading way to reduce the likelihood that your LLM-enabled application will hallucinate, right?
• 20:42
  SPEAKER_01: Because you're essentially filling in knowledge gaps that the language models may have, in that, you know, the a lot of these foundational models are trained on the entirety of the internet, but maybe they're not trained on your specific, you know, business practices in this one area, because of course, how how could it be, right?
• 20:59
  SPEAKER_01: So you you want to supplement its knowledge base with RAG.
• 21:02
  SPEAKER_01: And as you've already said, you know, initially it was, it was, it was uh a pr, you know, RAG was kind of like, I'm gonna put as much context as I possibly can, as well organized as I can kind of get it into one big kind of slushy vector database.
• 21:16
  SPEAKER_01: And then my assistant, when it gets a prompt, it's going to go and do some kind of like, you know, search over that to find the right bits of information and use that to augment its its response here.
• 21:29
  SPEAKER_01: And modular rag, right, or advanced rag is sort of making that breaking that kind of uh that big wide open.
• 21:36
  SPEAKER_01: I'm gonna search across this entire vector database more into like a componentized approach to this, which I I think makes it at least in my experience and my understanding, it's going to significantly improve an agent developer's or machine learning engineer's ability to actually, in a very targeted and fast way, kind of improve the certain like components or certain parts of the context that it.
• 22:00
  SPEAKER_01: That they want to improve to improve the agent's response.
• 22:03
  SPEAKER_01: So, of course, understanding how how and where to supplement your agent with more context in and itself is a big problem and a big piece of this puzzle.
• 22:11
  SPEAKER_01: And modular rag or advanced rag, I think is trying to solve some of that problem by putting a little bit more of a standardization around the components that go into RAG and allow you to kind of improve these individual components and say, I understand that, like in order for to get the right context to my assistant to answer these types of prompts, I need to improve like the data processing unit, or I need to improve like this specific knowledge base or this specific data store that it draws from to answer questions about my, you know, I don't know, my fall clothing line or whatever it is.
• 22:43
  SPEAKER_01: Um and and so I think it's, you know, we're gonna see a lot of evolution in RAG over time.
• 22:47
  SPEAKER_01: And I think you're seeing evolutions both in terms of what you're talking about with modular RAG, but also the the means by which the assistants actually look out across the uh provided context, you know, adding like additional uh semantic search on top of that or just sort of improving how these agents actually locate the right context at the right time to answer these questions.
• 23:08
  SPEAKER_01: But, you know, to kind of get back to the core of the problem, and I know you've mentioned this before, it's like at the end of the day, you know, it this is still very much a data game in a way.
• 23:19
  SPEAKER_01: It's like in order for these assistants to be successful, you you oftentimes still have to make sure that they're going to have access to that specific context you need them to have access to, and you have to make sure that it's organized in a way that's going to make it easier for them to locate, right?
• 23:37
  SPEAKER_01: And sorry, one other thing I'll mention on this that I think is kind of interesting too is there's there's there's a lot of opportunity in observability of rag, right?
• 23:47
  SPEAKER_01: Because if your assistant is giving poor answers to a specific type of question about a specific topic, you know, I guess let me back up.
• 23:55
  SPEAKER_01: How do you know that it's giving poor answers to a specific, you know, on a specific topic?
• 24:01
  SPEAKER_01: Like, what if you could kind of look at a map of all of the context that you've vectorized and you've provided to your assistants, and you can actually see, like, we're getting a lot of prompts about this kind of specific topic area, but look how sparse our actual like rag context is for the specific topic.
• 24:19
  SPEAKER_01: We need to spend time sort of filling in that hole, or you know, we have a bunch of context down here, and we never get asked any prompts that force the agent to utilize it.
• 24:28
  SPEAKER_01: Maybe we can just leave that alone for now.
• 24:30
  SPEAKER_01: Right.
• 24:30
  SPEAKER_01: So I think there's some there's a lot of opportunity and kind of untapped opportunity right now for helping people understand where they need to actually improve their their rag system.
• 24:40
  SPEAKER_02: And and you're you have the ability to visualize that, or how how are you actually doing that?
• 24:46
  SPEAKER_01: Oh, sure.
• 24:46
  SPEAKER_01: Yeah.
• 24:47
  SPEAKER_02: I mean, the vector spaces and looking at how often they are uh leveraged or looking at how the prompts are accessing uh those vector spaces.
• 24:56
  SPEAKER_01: Sure.
• 24:56
  SPEAKER_01: I mean, one one thing that New Relic is looking at, you know, this isn't this isn't on the market yet, but yeah, it would be you know essentially like a a heat map or a scatter plot of like, here's the entirety of my rag system and and here's a visualization, right, of of clusters of context by topic or however the vectors are organized.
• 25:16
  SPEAKER_02: Using dimensionality reduction to use.
• 25:18
  SPEAKER_01: Yeah, right.
• 25:19
  SPEAKER_01: Yeah.
• 25:19
  SPEAKER_01: It's uh I think it's gonna be very interesting for sure.
• 25:22
  SPEAKER_02: Yeah, and I think that at the end of the day, when you say it is about context, uh, you know, that's such a critically important topic.
• 25:29
  SPEAKER_02: And I think it's bandied about quite a bit.
• 25:31
  SPEAKER_02: But even if you said the prompt was, you know, send me logs for the half-less hour, you know, there's an implicit assumption about what that context is, right?
• 25:41
  SPEAKER_02: What system are you in?
• 25:42
  SPEAKER_02: Or the model has to say, well, logs of what, right?
• 25:45
  SPEAKER_02: Yeah.
• 25:46
  SPEAKER_02: Um, and and so what organizations need to understand is that cuts across all of their content.
• 25:51
  SPEAKER_02: If you're asking support questions and you're asking about a particular model, right?
• 25:55
  SPEAKER_02: Just like a human would need context, the LLM needs the context.
• 25:59
  SPEAKER_02: You can't just give it the world, right?
• 26:01
  SPEAKER_02: Because even at that granular and atomic level of those pieces of content, especially if you're doing question-answering types of systems, you need to have you know that that tagging and that metadata at that level of nuance and detail so that the so that the agent can return the right information and for that right person in that right context.
• 26:23
  SPEAKER_02: And it's such a very trivialized uh uh uh area, right?
• 26:28
  SPEAKER_02: People don't necessarily think about that granularity in those use cases.
• 26:32
  SPEAKER_02: And so, how do you uh talk to people about data and content curation and uh alignment with things like use cases and and how do you how do they get to the right level of detail and granularity, or do you not run into that?
• 26:47
  SPEAKER_02: I mean, tell me about your world because if an organization is saying, well, I'm not getting the right results from my uh from my uh uh LLM, we're trying to troubleshoot that, right?
• 26:58
  SPEAKER_02: So, how do you kind of bump into that and then how do you address that or help organizations address that?
• 27:04
  SPEAKER_01: Sure, yeah.
• 27:05
  SPEAKER_01: Uh a very common problem, as you've noted, there's a lot a lot of modern-day software systems and especially user interfaces have have all been built, obviously, around the assumption that a human is sitting there looking at it and is able to uh interpret certain you know things or control certain things about the UI in order to understand where they are, why they're there, and that sort of thing.
• 27:29
  SPEAKER_02: So just a very simple example.
• 27:30
  SPEAKER_02: UI is giving them context.
• 27:31
  SPEAKER_02: Yeah, exactly.
• 27:33
  SPEAKER_02: I mean, anytime we try to do an uh usability improvement, user experience improvement, it's always helping people match the system with their mental model, which is contextualizing the information and their uh according to their mental model.
• 27:46
  SPEAKER_01: Right, exactly.
• 27:47
  SPEAKER_01: And now if you want an as if you want a generative AI assistant to automate some of whatever maybe the end user is doing, or maybe automate some of the support aspects of what somebody else would do.
• 27:57
  SPEAKER_01: It's like, you know, historically, you know, in order to help say a support engineer or a service desk person understand what was going on when a user encountered a problem, for example, which is you know a very common use case for a lot of generative AI assistance is kind of this frontline support.
• 28:13
  SPEAKER_01: Uh, you know, you might send the support human a screenshot of what the user was seeing or something like that.
• 28:21
  SPEAKER_01: Or oftentimes they'll ask for screenshots.
• 28:23
  SPEAKER_01: So they'll ask for a copy of the error message or something like that.
• 28:26
  SPEAKER_01: And I think that to your point, that from there, the the support human would oftentimes need to like go look up the user's email address to figure out like what account they belong to and all this different stuff.
• 28:37
  SPEAKER_01: And I think what we've encountered at New Relax because we've built our own agent, is right, is we've we've done a lot of work to try to see how much of the individual context of that user, you know, all the way up to their account, like or or what page they were on, what part of the page they were on, we can actually serve to the assistant just as part of the context it automatically gets from the page.
• 29:00
  SPEAKER_01: So we we kind of click, we'll call it this like context API, and it exists on all of New Relic's platform pages right now, and it's constantly sort of serving up to this assistant a variety of of metadata about the the location, the account, the user, the entity, you name it.
• 29:16
  SPEAKER_01: And there, but there's there's it's a we still haven't like quite cracked the code, I will say, because there's a lot of you know, we're constantly finding places where it's like, ah, well, we need to give the assistant this additional metadata to do this.
• 29:30
  SPEAKER_01: And I think that the what I would what I would say, if if if somebody were starting fresh here, and I think we we're we're looking into something like this too, but there's a lot of really interesting things you can do here around, you know, with like image to text models, right?
• 29:43
  SPEAKER_01: Like as an example.
• 29:45
  SPEAKER_01: Originally, when we were first building building our generative AI assistant, I don't think you could have reliably taken a screenshot of a page in New Relic's platform and sent it to an image to text model and gotten really good results.
• 29:58
  SPEAKER_01: Like it wouldn't, it would have been very error-prone.
• 30:00
  SPEAKER_01: But what we're seeing increasingly is like, well, rather than trying to like slurp all of this context from the page individually, if we can avoid it, why not like take an image of the page and have that image to text model actually like just like see the account ID at the top of the screen and like see the username in the bottom left and all this different stuff.
• 30:20
  SPEAKER_01: I think there's some really interesting things that we're gonna be able to do as far as serving these agents more location-specific context, just using automatically snapped, you know, screenshots of what the user was doing.
• 30:33
  SPEAKER_01: Now, that may or may not be possible depending on the kind of like regulations and constraints that an organization is under.
• 30:38
  SPEAKER_01: It's like some companies are not going to be willing to take a screenshot of what the user was doing when they encountered a problem, but it's still interesting.
• 30:46
  SPEAKER_01: And then I think the other advice I would give anybody who's interested in creating kind of like sophisticated agentic flows using context and all of this different stuff is like really, really narrow, narrow the use case initially.
• 30:59
  SPEAKER_01: I mean, I think it's really tempting with this technology to start broad to go and to try to solve all of your problems or solve all of your frontline support problems with one of these agents.
• 31:10
  SPEAKER_01: And I think, especially for the reasons that you've cited around the difficulty serving up the right context at the right time and the adjustments you'll need to make along the way, picking that one use case that, hey, if we can solve this, it's really going to prove the value.
• 31:24
  SPEAKER_01: And then we can kind of add on additional use cases over time.
• 31:27
  SPEAKER_01: And that'll allow you to kind of solve individual problems with some of this context, you know, along the way.
• 31:33
  SPEAKER_01: It's like maybe start with a use case that helps you solve the user's account context problem first and then move on, whatever it is, right?
• 31:41
  SPEAKER_02: I call that the excuse case.
• 31:43
  SPEAKER_02: It's the use case that gives you the reason to solve a lot of other things, right?
• 31:47
  SPEAKER_02: When you have one use case that can say justify the revamping of an information architecture or an ontology, now all of a sudden you have all these other applications and all these other problem areas that you can solve once you have that infrastructure.
• 32:02
  SPEAKER_02: And so you're essentially saying the same thing.
• 32:04
  SPEAKER_02: Pick a very narrow set of tasks or problems, solve that, and then you have the ability to expand that scope and expand those use cases to broader and broader contexts or broader and broader sets of problems.
• 32:17
  SPEAKER_02: And then it makes it much easier for you to kind of uh uh have wins that you can build upon.
• 32:24
  SPEAKER_02: That's right.
• 32:25
  SPEAKER_02: That's right.
• 32:26
  SPEAKER_02: What kind of advice?
• 32:27
  SPEAKER_02: Go ahead.
• 32:27
  SPEAKER_02: You were gonna say something else.
• 32:28
  SPEAKER_02: No, no, no.
• 32:29
  SPEAKER_02: No, I was going to say, what kind of advice would you give to CEOs and leadership teams who are wrestling?
• 32:36
  SPEAKER_02: They're hearing tons and tons of pitches from vendors every day.
• 32:40
  SPEAKER_02: And I find that, you know, a lot of vendors have what I call aspirational functionality, right?
• 32:46
  SPEAKER_02: They say they they want to do it, they intend to do it, maybe they'll get there one day, but they don't have it today.
• 32:51
  SPEAKER_02: So the stuff that's showing up in slides is uh a little bit farther ahead and then the reality of what they can do.
• 32:57
  SPEAKER_02: And so you have to separate the wheat from the chafe, right?
• 33:00
  SPEAKER_02: You have to have the BS detector turned up on it to hide.
• 33:03
  SPEAKER_02: And so, and and of course, the safe way to go is with the big consultancies and the big uh services firms, but they don't necessarily have the answers either.
• 33:12
  SPEAKER_02: So, what kind of advice would you give to uh leadership teams who are wrestling with this stuff, trying to find the right resources, the right partners?
• 33:20
  SPEAKER_02: So, how do they go about that?
• 33:22
  SPEAKER_02: And this is kind of it's not like choose us, right?
• 33:25
  SPEAKER_02: But because you're talking about you know, looking at this field more broadly and saying, how do you separate the hype from reality?
• 33:31
  SPEAKER_02: And what are some things that that executives and leadership teams need to do?
• 33:37
  SPEAKER_01: Yeah, a good question.
• 33:40
  SPEAKER_01: I think I think what I would say is is there a bubble in with with this technology, especially businesses anchored to this technology right now?
• 33:50
  SPEAKER_01: And and I think this is particularly relevant to to a lot of executives who are wondering whether or not they should invest more of their budget in generative AI initiatives because they're you know, there's all this news around like I don't know, however many tens of billions of dollars spent over the last year and how much revenue was generated, and it's hugely disproportionate.
• 34:09
  SPEAKER_01: And I think that's I hear from some executives at companies that that's kind of turning them off of this.
• 34:16
  SPEAKER_01: But I guess my advice would be don't get too cynical.
• 34:20
  SPEAKER_01: I mean, if when you look at the internet, the world wide web, the exact same thing was true.
• 34:25
  SPEAKER_01: Initially, the amount of money that was getting poured into essentially the development of the internet and websites vast vastly outweighed the actual benefit anybody was getting.
• 34:35
  SPEAKER_01: And and look where we are now.
• 34:37
  SPEAKER_01: Yeah.
• 34:37
  SPEAKER_01: And look where some of the companies that went big into development in that space relatively early, you know, are now, right?
• 34:46
  SPEAKER_01: And so I think my advice would be, you know, I think a a certain amount of of like you say, uh have your BS detector up, make sure that you're really taking the time to understand what a vendor is really offering you, and if it's something that is essentially like a kind of like a reskin of something that you can get otherwise.
• 35:06
  SPEAKER_01: Like, you know, you see a lot of these like chat bots out of the box kind of offering these days of like, hey, we'll come in and index your docs site and give you like a docs bot, you know, and it's like, well, that's not really that difficult to build in-house.
• 35:19
  SPEAKER_01: And I would be kind of wary of things like that.
• 35:21
  SPEAKER_01: But I also would would encourage executives, especially those that are holding the purse strings, to not be overly cynical or overly skeptical when they're considering whether they should allocate more of their budgets into research and development using this technology.
• 35:35
  SPEAKER_01: Because I do think that even if there's a bubble now, we'll we'll start to see the the value to investment you know curve really kind of turn around and not too far in the future.
• 35:45
  SPEAKER_01: But the but the thing I would kind of advise, and I think where a lot of companies who are investing in this hit a pretty significant wall, is first and foremost talk to legitimate, you know, data science and data engineers, whether that's via firm or in-house or just you know, just bring in an advisor and understand is your company's you know, knowledge bases, is is your company's sort of like data layer, whatever it is.
• 36:19
  SPEAKER_01: And by data, I mean it can be marketing data, it can be business intelligence, it can be the telemetry, it can be whatever it is.
• 36:26
  SPEAKER_01: Get an honest and expert opinion on whether or not you're even set up at a data level to really like take advantage of a lot of the things these agents can do.
• 36:36
  SPEAKER_01: Because very often, as you know, organizations go down this path of sort of saying we're gonna build this autumn, you know, we're gonna build this bot that's gonna automate a lot of what we do internally, or maybe automate things that our customers do.
• 36:48
  SPEAKER_01: And then they find out that, oh Lord, we have a huge, massive underlying, you know, data organization or data storage or data queryability problem that they need to solve.
• 37:00
  SPEAKER_01: And so that's where it's like, I would strongly advise that every organization that believes that there's something to this AI stuff starts there and takes a really cold hard look, getting real advice from real data scientists and data engineers and chief data officers and that sort of thing about like where they stand in terms of being able to actually use their data for machine learning.
• 37:24
  SPEAKER_02: Yeah, and it's it's data, it's content, it's knowledge, it's all of that stuff.
• 37:28
  SPEAKER_02: And I'm gonna do a shameless plug that we do an AI, uh, a CAM knowledge management and data assessment for AI workshop.
• 37:37
  SPEAKER_02: That's a short sprint that is exactly what you're talking about.
• 37:40
  SPEAKER_02: It's saying, you know, where are you?
• 37:42
  SPEAKER_02: And you know, many organizations are saying we're not ready to do AI, but we are ready to do IA, right?
• 37:48
  SPEAKER_02: I had coined that phrase, there's no AI without IA.
• 37:51
  SPEAKER_02: And you know, there's no artificial intelligence without information architecture.
• 37:54
  SPEAKER_02: Well, that is the critical piece.
• 37:55
  SPEAKER_02: So many times we go in and fix those pieces, which will solve lots of other problems anyway, right?
• 38:01
  SPEAKER_02: Yeah, you know, because that's basic blocking and tackling.
• 38:04
  SPEAKER_02: And then you're building the foundation for doing that future stuff, right?
• 38:07
  SPEAKER_02: Right.
• 38:07
  SPEAKER_02: So I get to throw in a uh uh once in a while.
• 38:13
  SPEAKER_02: But uh, but no, I think it's important.
• 38:14
  SPEAKER_02: I think it's important for organizations to understand.
• 38:17
  SPEAKER_02: And you know, hearing it coming from other folks to say that your data infrastructure, your content, your your knowledge, that is the stuff that these tools are going to operate on and retrieve.
• 38:26
  SPEAKER_02: And if you don't have that, then there's there's a problem because they're gonna hallucinate, they're not gonna be able to have the right answers.
• 38:33
  SPEAKER_02: So just want to shift gears a little bit.
• 38:35
  SPEAKER_02: Tell me a little bit about your background.
• 38:36
  SPEAKER_02: I know you you came from the journalism area.
• 38:38
  SPEAKER_02: What what do you do for fun?
• 38:40
  SPEAKER_02: What do you tell us a little bit about Kendon the the person uh as opposed to Gandon the the role?
• 38:46
  SPEAKER_01: Sure, yeah.
• 38:46
  SPEAKER_01: I've I've had kind of a wild and tumbling journey through the tech industry.
• 38:50
  SPEAKER_01: I I it's funny actually, when I first started working in tech, it was like peak crowdsource uh praise, uh circa, I don't know, 2011, 2012 or something like that.
• 39:02
  SPEAKER_01: And that was also back in the um, you know, SEO cramming days.
• 39:07
  SPEAKER_01: The first startup I worked at, for example, was we, you know, overstocks and Amazons and other large retailers of the world realized that they had 100,000 product SKUs that they need needed rewritten and re-optimized like fairly regularly.
• 39:24
  SPEAKER_01: Enter crowdsourcing and we had a platform and all this stuff.
• 39:26
  SPEAKER_01: And the reason why I was able to make a hop from journalism to that was because there's this startup that was sort of anchored to the idea that they were going to produce high-quality content for e-commerce, but nobody on the staff knew anything about content.
• 39:40
  SPEAKER_01: Like they none of them were writers, none of them were editors.
• 39:43
  SPEAKER_01: So they needed someone who could come in and kind of like almost look at the assembly line of for crowdsourcing and apply like QA principles to it to make sure it was it was coming out the the the other end and looking good, right?
• 39:58
  SPEAKER_01: And so it's funny I mentioned that because I think there's, you know, as we you hear all this recent news about as generative AI begins writing more and more content, what is that doing to like search results optimization and and things like that online?
• 40:13
  SPEAKER_01: And I started feel some of the same themes kind of come back to the surface as were happening back then.
• 40:20
  SPEAKER_01: And then along the way, you know, I've done everything in in technology from from marketing to business operations to account management.
• 40:28
  SPEAKER_01: Now, of course, product management, which is where I think really really my happy place is here.
• 40:32
  SPEAKER_01: And I think what I've really appreciated about working on AI and ML initiatives as the as the as the head of AI ML here at New Relic is that it's a it's such a generally applicable technology to a lot of different problems.
• 40:44
  SPEAKER_01: And it's it's it's it's relatively easy to kind of like understand and understand the value of, you know, it's it's it's it's a great technology, I think, especially for higher level leaders who maybe aren't uh in the in the technical weeds as much to think about and to devise kind of visions around.
• 41:01
  SPEAKER_01: Because while there's always the risk of magical thinking, I think it really does lend itself well to beginning to kind of reconsider whether certain things that happen on your platform or in your products or in your internal processes could, in fact, be automated or at least greatly reduced as far as the toil goes.
• 41:18
  SPEAKER_01: Um, outside of work, anyway, though.
• 41:20
  SPEAKER_01: I mean, outside of work, you know, interestingly, actually, I still still love to write.
• 41:23
  SPEAKER_01: I still love creative writing.
• 41:25
  SPEAKER_01: I write short fiction, which has been another really interesting space with the rise of large language models, of course, because you know, suddenly writers can potentially train their own small language models using all of the things that they've written before to now generate more content that sounds just like what they're writing.
• 41:43
  SPEAKER_01: And I think there's there's gonna be some, I don't know, in the arts, especially, we see this a lot with image generation and video generation.
• 41:50
  SPEAKER_01: But I think the true the same thing will be true of writing.
• 41:52
  SPEAKER_01: It's like as the maybe more arduous or toilsome aspects of art generation, whether it's written or otherwise, are increasingly automated.
• 42:03
  SPEAKER_01: What's going to distinguish good, let's say, written fiction from bad written fiction?
• 42:10
  SPEAKER_01: Or will we start to see uh you know entirely different requirements for what is good written fiction emerge?
• 42:18
  SPEAKER_01: Anyway, it's it's just a fascinating feel.
• 42:20
  SPEAKER_01: So that's that's kind of how I spend a lot of time.
• 42:23
  SPEAKER_02: Uh any any other hobbies or uh any uh kids or animals, pets?
• 42:28
  SPEAKER_01: I've got two cats, frog and toad, love running.
• 42:31
  SPEAKER_02: What are the names?
• 42:31
  SPEAKER_02: What are the names?
• 42:32
  SPEAKER_02: Frog and toad.
• 42:33
  SPEAKER_02: Frog and toad.
• 42:34
  SPEAKER_02: Yeah, yeah.
• 42:36
  SPEAKER_01: Um love running, love hiking.
• 42:38
  SPEAKER_01: Live in I live in the Pacific Northwest, so we're we're we're big on the outdoors up here.
• 42:43
  SPEAKER_01: And you know, outside of of writing and work, there's there's not much time left in the day, otherwise.
• 42:48
  SPEAKER_02: I get it.
• 42:49
  SPEAKER_02: It's funny, you know, back to your question or your comment about um AI enhancing uh creative writing.
• 42:58
  SPEAKER_02: I mean, I do a lot of professional writing and I find that it slows me down because I I don't use uh AI generated content for my writing.
• 43:07
  SPEAKER_02: I I write my writing, but I'll I might use it to do a little editing, I might use it, but sometimes it messes up the content because it makes it sound like AI, right?
• 43:16
  SPEAKER_02: And I don't want that.
• 43:17
  SPEAKER_02: Uh, but I'll use it for research, I'll use it for outlines, but then I end up with more stuff that I need to go through to synthesize from, you know, for some paradoxical reason.
• 43:28
  SPEAKER_02: I find it reduces my productivity, um, which is a strange thing.
• 43:32
  SPEAKER_02: You know, it's easier for me to do some research than it is for me to try to generate stuff.
• 43:36
  SPEAKER_02: So I don't know.
• 43:37
  SPEAKER_02: It's but I do need to kind of uh do what you said is train it with all my prior content so that it has more of my voice.
• 43:44
  SPEAKER_02: Um I've gotten around to yet.
• 43:46
  SPEAKER_01: But yeah, it's it's worth trying.
• 43:48
  SPEAKER_01: And I think you you may also find, I mean, to your point, it's like some some aspects of your process are probably good candidates for for using an LLM and others not so much.
• 43:57
  SPEAKER_01: I mean, I I I personally not not so much for creative writing, it's more of just kind of tinkering with that file, but at least for professional writing, it's like uh you know that you can either start with a blank page for me, or I I start with something that's like even 40% acceptable.
• 44:13
  SPEAKER_01: And and if I can get 40% acceptable out of an LLM and just take it from there, right?
• 44:17
  SPEAKER_01: A lot of times it's time savings, but miss for sure.
• 44:21
  SPEAKER_02: What kind of fiction do you write?
• 44:22
  SPEAKER_01: Oh, I would I would consider it just kind of you know uh general literary fiction.
• 44:26
  SPEAKER_01: So it's not any kind of genre fiction, it's just um yeah, kind of the short fiction you'd see in magazines.
• 44:32
  SPEAKER_02: Well, Hamden, thank you so much for sharing your insights here on this uh episode of our Air Early AI podcast.
• 44:39
  SPEAKER_02: I really appreciate it.
• 44:41
  SPEAKER_02: Thank you for your time.
• 44:41
  SPEAKER_02: Thanks for your insights.
• 44:43
  SPEAKER_01: Of course, yeah, happy to be here.
• 44:45
  SPEAKER_01: Thank you so much for having me.
• 44:46
  SPEAKER_02: And uh thanks everyone for tuning in, and Carolyn for your work in the background and for the other folks who've been helping out.
• 44:53
  SPEAKER_02: And we will see you next time.
• 44:54
  SPEAKER_02: And by the way, in the show notes, you can find where to locate Camden.
• 44:58
  SPEAKER_02: Uh, by the way, on Twitter, it's sweet at Camden's S W I T A C A D E N on Twitter.
• 45:05
  SPEAKER_02: The company uh website is New Relic, N-E-W-R-E-L-I-C.com.
• 45:12
  SPEAKER_02: And again, thank you for being here.
• 45:14
  SPEAKER_02: And we will talk to you next time on the next episode of the Early A podcast.
• 45:19
  SPEAKER_02: Thanks again.
• 45:20
  SPEAKER_00: Visit early.com to find links to the full podcast on all audio platforms to listen on the go.
• 45:27
  SPEAKER_00: Thank you for watching.
• 45:29
  SPEAKER_00: Contact us today at infoearly.com.