Latest / Practical AI / We've all done RAG, now what?

Transcript

We've all done RAG, now what? · Practical AI

• 0:04
  Jerod: Welcome to the Practical AI podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind the scenes content, and AI insights. You can learn more at practicalai.fm.
• 0:36
  Now, onto the show.
• 0:49
  Daniel: Welcome to another episode of the Practical AI Podcast. I'm Daniel Wightnack. I am CEO at Prediction Guard and not joined by Chris today, but very happy to be joined by a longtime friend of of the podcast and and friend of all things AI and data science online, Rajeev Shah from contextual AI, where he's the chief evangelist, and also, of course, TikToker extraordinaire all things. So welcome back to the show, Rajeev. It's great to have you.
• 1:23
  Rajiv: It's great to be back. Excited to talk.
• 1:25
  Daniel: Yeah. And we're we're close neighbors around the the Midwest as well. I'm looking forward to seeing you at the the Midwest AI Summit, which if if listeners don't know, both Chris and I will be at, the Midwest AI Summit, which is happening November 13 in Indianapolis. So I think as you put it on LinkedIn, Rajeev, if you live anywhere near Corn and you like AI, then this is the place to be. And actually Rajeev will be there giving a great talk about kind of this idea of, I think you're going to be talking about this stat, the ninety five percent of pilots that fail, why that might be.
• 2:05
  Yeah, so great to have a fellow Midwest AI Silicon Prairie friend on the show as well.
• 2:12
  Rajiv: No, it's great to have these in person events here in the Midwest. Usually you have to kind of fly out to some large city to get that. It's nice to be able to kind of get that right in Indy.
• 2:21
  Daniel: Yeah, yeah. So if you're interested in that, go to midwestaisummit.com and make sure and register for that. Hopefully we'll see you there in person. It's going to be a fun time. But it has been a while since we've had you on the show, Rajeev, and I'm sure there's a lot to catch up on.
• 2:39
  It almost seems like last time we talked, there was kind of we were all getting into Rag and thinking about agents and reasoning models. We're trying things. People were gradually developing maybe what they thought would be a direction for best practices or ways to approach problems. It would be interesting to hear from then to now how you've seen that world advance in these archetypal use cases that people are addressing with retrieval related frameworks?
• 3:18
  Rajiv: Yeah, no, it's been amazing. About two and a half years ago, we were really giddy about the possibility of AI, but there was a lot of different things that we can do. And kind of since then, we've seen AI develop in certain ways where, for example, in code completion, code development tools, it's accelerated dramatically, even over the last year like that. And we're all kind of using those tools on a regular basis. And then in other areas, right, like chatbots, for example, we've seen some progress, but we've also seen that there's some things that are difficult to do with chatbots as well.
• 3:50
  But I think, you know, it's been incredible to see just the continued development of AI in terms of the capabilities, the smartness, the ability to kind of use tools to find search information that there's still so much for all of us to do in our lives, kind of for those of us that work with AI.
• 4:07
  Daniel: Yeah, yeah. And I guess, certainly there's a lot of people still I guess when I interact with customers or hear from other people in the industry, still a lot of people are wanting, as their first things, to kind of whatever is build a rag chatbot, which maybe for those listeners out there that are maybe not as familiar with this, they've maybe heard rag this and raggedy rag that and all of that. Maybe just remind listeners what rag is and how retrieval may fit into some of this AI stuff.
• 4:46
  Rajiv: Yeah. And I'll start simply. Even a couple of years ago, I was kind of working with the earliest large language models and people would be like, I like ChatGPT, but it doesn't know anything about me or my company. Like, how can I train it to know all that knowledge? And what we figured out is that it's not about really training that model about the knowledge.
• 5:05
  It's instead finding that knowledge, searching that knowledge, passing that on to the model, and then be able to use that. And that's where kind of this idea of retrieval, finding stuff, and then augmenting the generation, augmenting the written response for that came about. And I think RAG today is kind of one of the most important or it's one of the most widely used use cases, I'd say,
• 5:29
  Daniel: the general AI space. Yeah.
• 5:31
  Rajiv: Every company is probably running some type of rag at this point for searching its internal knowledge, helping folks figure out through HR documentation, using it for customer support. There's a lot of use cases where we have huge amounts of information. We want to be able to find things from that. But we also want to use the smarts of AI for that. We don't want a Google search results of 20 things.
• 5:56
  We want a nice AI summary of it or we want to extract out all the information. I just want the names and dates out of all this. I don't need to read every one. And so that's where the AI really comes into it and why Rag is so powerful and we see it so widely used.
• 6:11
  Daniel: Yeah, and I love what you were saying, just sort of contrasting this with training or even fine tuning. I find this, of course, to be a very widespread misconception about how these tools work. Even ChatGPT, the application, it does seem like it kind of quote trains on your information that you put in, your previous chat history, etcetera. And then it's almost like you have a model of your own, right? It almost seems like OpenAI has a separate model for every person on the planet, which is not feasible from the data science training perspective.
• 6:52
  So could you highlight some of those things and highlight, I know you educate a lot as well, so I'm sure you've seen this as well. There's kind of this jargon of training thrown around a lot, which is confusing maybe in how it's used. Yeah.
• 7:08
  Rajiv: Yeah. And I think partly is a lot of these companies want to make you think that they have one thing that does everything in the world. But if you actually take a look at something like ChatGPT, it's a system. It actually composes of multiple parts. So if you were trying to build your own ChatGPT, these are some of the things that you would have to think about.
• 7:28
  So one is I need to be able to retrieve from lots of different sources. I have all my knowledge inside my companies, inside of Confluence. I need to be able to access and retrieve that. That could be one thing. So one thing is using Rag, being able to access tools.
• 7:45
  Another thing is memory. Like I wanted to remember that last conversation I had, whether it was two things I said ago or like one of the nice things about ChatGPT is you can come back a conversation or a day later and it remembers a lot about you. It could maybe a little bit scary sometimes how much it remembers about you and how well it can profile you. And so this is kind of what we see inside of AI engineering as context engineering, where there's a number of different parts of managing interactions with these models, whether it's rag, whether it's remembering the memory, when it's knowing how to summarize conversations to do things like multi turn so we can have those repeated conversations, but keep track of what was said earlier in our conversations as well.
• 8:31
  Daniel: Yeah, so I love that idea of context engineering. How is that idea of context engineering kind of different from what we would maybe from the traditional data science side consider model training, I guess? So there's almost like a pipelining mindset versus an actual GPU model training piece.
• 8:56
  Rajiv: Yeah. I think what's remarkable nowadays is the amount of information that's stored inside of these large language models. I do a thing when I kind of talk about large language models where imagine if you were sitting outside and kind of reading a book and you could read all day, all night. If you read for ten years, I think that's on the order of something like about a billion tokens. And these models are trained on the order of like 15,000,000,000,000 tokens.
• 9:23
  So it's just an inconceivable amount of information that these models hold inside there. And I think one of the biggest improvements we've seen in these models is not only kind of stuffing all that information in there, it's effectively using all of that information in there, that these models really today are untapped in terms of everything they can do. And it's one of these things on the technical side is why you see that the capabilities of the models have continued to grow without them necessarily having to grow immensely in size simply because we're better tapping into all the abilities that they have inside them.
• 9:59
  Daniel: Yeah. And I know you're up for this because we know each other, but I'll present a scenario to you that sometimes comes up for us, whether I'm teaching a workshop or something, and you can help us understand how this retrieval type of thing fits in. So I often Maybe I'm talking to folks in healthcare, and what's interesting is there's all of this huge amount of data that these models have been trained on. And there is good, let's say, medical information in that data, right? Or there's care guidelines for nurses or medics or whatever it is.
• 10:40
  Let's take that as an example. The interesting thing is there's even if you think about factuality in a situation, it is kind of relative in the sense that if I'm a medic that works for this provider, let's say their care guidelines are different from another provider. Or if I'm treating maybe children, those guidelines are probably different than if I'm treating or senior adults or something like that. So there can be these facts out there that actually conflict with one another. It's really You talked about the context.
• 11:16
  It's really about the context that you're working in, what it should pull. So these models, assuming that all that information is public, right, like you said, and it's been scraped off the internet, all of that's kind of You could imagine that maybe somehow that is embedded somewhere in the model, the base model, so whether we're talking about LAMA or GPT or whatever. Let's say I'm a healthcare company, but I'm interested context. Now, how do I make that connection? Still use maybe, I'm not going to retrain that model, but how do I use the model then in this context of kind of conflicting facts?
• 11:56
  Rajiv: Yeah, absolutely. As much as the models know and been trained so widely, they don't know everything. And so sometimes that's you have your own information. You want to pass that into the model. And so that's where that retrieval augmented generation comes in.
• 12:10
  We want to grab that information and and then we want to pass it into the model. And this is where for folks who are in the space for a while prompt engineering comes into play, where we think about the inputs that go into the model, where we take the facts that we know about our own given situation, along with maybe some instructions for what we want along with it. And that all of that is manipulating the context, right? The larger world that this model is sitting within there. And by manipulating and giving it information, instructions and other facts, we can get outputs that better match kind of what we're looking for like that.
• 12:46
  Daniel: Yeah, yeah, that's great. And I guess there's another term that's been thrown around the last couple of years. One would be this kind of retrieval piece, which you've talked about, and this idea that in the context of a certain question or something, I'm able to connect to one of my internal data sources, get that right There's this other thing that would be related to reasoning. Now, some people might just consider that all generative AI models reason in one way or another, but it is now almost like a term of art. It means a certain thing talking about these models.
• 13:26
  Could you help highlight that also to put that in context? And then maybe we'll circle around and combine some of these things.
• 13:32
  Rajiv: Yeah. A lot of times when we talk about models, we want to make it convenient to help people explain things until we kind of anthropomorphize them, right? We give them human like qualities when they're not actually human like qualities. And so, for example, reasoning is something we say for these models. Now they're not reasoning like a human would through a problem.
• 13:51
  But what we typically mean kind of when we think about reasoning with these models is they're doing lots of extra steps and they're doing these steps in a logical way to better solve a problem. So if you take kind of a math word problem, a great way to see this is if you take a word problem like the train is moving east, you know, at 50 miles an hour and another train is moving west another 40 miles an hour and you have to figure out like what point they're going to cross, right? There's no kind of quick answer to that. You need to kind of calculate the first train, calculate the second train, and then you can figure out the solution. And what we've done, and that's what these reasoning models have done, is we've trained them that don't come up with the answer right away.
• 14:34
  Think through it. Think about the first thing. Think about the second thing. Connect all the dots and then put an answer in. And the way we've done this is we've literally given the model.
• 14:45
  Well, we've trained it in multiple ways, but in some ways we've literally give the model an example of, hey, this is how I solve this word problem. I did this all these steps here. I want you to learn how to go through these problems step by step. And this is where we differ from two and a half years ago when the first models didn't know how to do that. Those first models, all we had trained it up was like, hey, tell me if a movie is a good movie or a bad movie or tell me if this sentiment is happy or sad.
• 15:10
  But since then, we've had time to develop more training data. We've given them more complex training. And the cool thing is the models have picked up. They've been able to learn this capability for doing this. And so now we're able to do this much more complex kind of I'm doing the air quotes of reasoning to solve these problems.
• 15:28
  Daniel: Yeah, I guess one thing that people, and this is part of why I really enjoy watching your videos online, is you often break down a lot of this jargon and kind of help. It's almost like people feel the shock of all of this terminology and a new model coming out every 30 minutes or whatever it is, and just not really knowing how to deal with that. And there's kind of all of these things that have happened. So there's reasoning models, there's small language models, there's tool calling, there's retrieval, there's all of these different mix of things. Based on your experience, both what you've implemented personally, also what you've seen and interacted with folks on, if people could bring in some focus and maybe imagine a company's getting into AI, however they view AI transformation within their organization, and they're thinking about some of the use cases that are the initial ones on their roadmap at the time being, what would you encourage them to pick the signal out of the noise?
• 16:43
  So what would you encourage them to maybe focus on to see some of that time to value upfront? Not that they don't want to explore some of the other topics that are happening or read about them or whatever that is, but how would you at least recommend to get that best time to value, or maybe just the things that are producing the most value? And you could maybe flip that as well and say, what are some of those things that are cool things, but maybe let's wait and see what happens and maybe just sort of don't get distracted at the moment.
• 17:18
  Rajiv: Yeah. Mean, you're taking me away from all the fun, cool technologies that are the latest, like automation, things that I can kind of fire up like that. I think when you're thinking about this from the perspective of kind of a company, if you're kind of a manager in these situations, you have to figure out what use cases you want to put on a very different hat than just thinking about the technology itself. And I can tell you this, like I was burned from personal experience when I was just starting out in data science. I was entranced by the latest technology.
• 17:51
  So I remember, right, we were talking about code development. Like ten years ago, I was working at State Farm and I think Andrij Karpathy has written his paper about the unreasonable effectiveness of LSTMs. It was something like that. But part of that paper had the idea of you could translate code, you could complete code from one language to another. And I was like, hey, come on, guys.
• 18:15
  Like, we've got a lot of this cobalt code sitting around. Like, give me some give me a few GPUs and some data and I can
• 18:22
  Daniel: do it.
• 18:22
  Rajiv: I'll solve this. Naive. They didn't fund that project or anything, but I think, yes, it can be very easy to be kind of seduced by the technologies, kind of what a shiny demo is versus when you're in an organization, you really have to think about kind of the problems that you have. Part of it will be, you know, how complicated this is from a technical point of view to get it up and running. That's one factor, But that's not the only factor.
• 18:49
  We also need to consider what's the value to this organization. I talk to lots of enterprises on a regular basis and I see often what I call science experiments where teams like the latest technology, they go out and kind of run this stuff, but there's no way for them to actually get that implemented inside the company in a useful way. And they're literally just kind of interesting experiments that people are running like that.
• 19:14
  Daniel: Does that get partially to like the ninety five percent of AI pilot failing type of report from MIT?
• 19:23
  Rajiv: Absolutely. Now, the 95%, of course, is like a little bit of a hype number that they like to put out in this. And for those of us who've been in the space for a while, we remember the fact of 80% of data science projects fail, I think was something that we had. And to some extent, that's okay. You can't expect every initiative, every experiment, everything that you start to succeed.
• 19:47
  You want things to fail because partly is if something works, that means you have to maintain it. You have to monitor it. You have to put up a lot of guards around it. There's a cost for something that actually succeeds. But when we talk about AI, it's very easy to build a cool demo, but it's not only the value to the company.
• 20:05
  You have to figure out how to integrate this into people's everyday work life. And so you can build a very shiny widget that sits and can do something awesome. But if that's not inside somebody's regular workflow of how they work, the tools that they work in, if they're not properly trained on how to use it, if their leadership isn't supporting you to use it, there's lots of factors like that that go into why people might not actually adopt and use a technology. And it's really nothing about AI. It's really about organizational change and introducing technology into companies.
• 20:40
  Daniel: Yeah, I know from just a founder perspective, can be, know, just from a different side of this, it can be frustrating when you see like, Oh, there's this company over here that the technology side is fairly simple. Like, oh, it's just a simple model that does a simple thing, or a browser extension that does this. And you're like, wow, I could have vibe coded that in a weekend. How are they scaling to the moon? We have all this cool technology.
• 21:14
  And I think part of it is that side of it that you talked about, part of the hard problem is cracking what actually does provide value to your organization, what can be adopted, how you communicate that, how you tell that story, and how you deliver on your promises, how you provide customer support. A lot of that is not really related to the technology and that component. So maybe, I don't know if this would be accurate to say, but maybe the first step people could take is just getting something off the ground that's fairly simple and interacts with an AI model. Maybe it's just to do a very simple task, but really pushing that through, like you say, to be embedded in a product or be embedded in a process. That may be the best way for people to kind of start that journey is to really start from that simple side and deal with some of, in all honesty, the harder problems around the periphery of the technology.
• 22:19
  Rajiv: And I think a big part of that, like what you're saying, is just to get closer to those end users, the stakeholders, because I think once you often cut through that, sometimes you figure out that really they don't necessarily need a fancy kind of GPT-five model to solve their problems. Maybe you can solve it with almost a simple if then rule that you can just implement in And some so this is where kind of looking at the data, spending time talking to those end users that often gives you a much better result. It's going to give you the biggest bang for your buck than going out and reading some archive paper.
• 22:57
  Daniel: Yeah, yeah, that's true. And I guess now you should just have the AI model read the paper for you and give you some summary points. A big fan
• 23:08
  Rajiv: of that. On my walks, often just sit and I'll talk to Chad GPT and we'll talk through papers and what are the main technical points and stuff.
• 23:14
  Daniel: Yeah, I'm glad I'm not the only one. Thanks for validating that. I guess getting back more into the development, retrieval kind of reasoning stuff, what are some of the Now that we've been working with this technology for a while, we've got more cycles. Someone can spin up a rag pipeline in whatever, ten minutes I can spin it up and I have something going. But it's another thing to kind of, of course, scale that, maintain it over time, deal with some of the issues.
• 23:46
  I guess my question is what pitfalls are people falling into that maybe we didn't know about whatever it was a year ago when we were kind of just getting into these initial kind of naive rag sorts of things? What challenges or consistent challenges and pitfalls do you see people kind of falling into?
• 24:05
  Rajiv: Yeah, think the consistent thing I see with something like RAG is it's fairly easy to build a quick demo. You can grab an off the shelf embedding model to do that. You can combine that with a generation model like an OpenAI model and you can build yourself a quick kind of proof of concepts. I think the trouble that people get into with it is scaling it up. It's great on 100 documents, but now all of a sudden I have to go to 100,000 or 1,000,000 documents.
• 24:33
  How am I going to do that? Or when I first did my demo, I did a couple of very simple queries, text extraction queries. But now when I put it out in front of my users, I find out all of a sudden they're not giving nice one sentence queries. They're just asking two words and then I need to add a query reformulation step or something to do that. Or the accuracy is not kind of what I was looking for.
• 25:02
  And so I've added a bunch of pieces in there. I've added a re ranker. I've added other steps. But now my latency is suffering, right? There's all these kind of trade offs as you kind of get to production.
• 25:11
  And then you're like, oh, you know, do I go back and you go and you look online and you see that, oh, wait a minute, there's like 15 I think there's like 25, 30 flavors of rag. You're like, oh, did I set up my infrastructure wrong? Oh, do I go back and I change my chunking strategies? I can see there's 10 different chunking strategies people are doing that. And so I think this is the cycle of where it's very easy to get started, but getting to that final kind of production quality rag can kind of be a little worrisome.
• 25:40
  Daniel: Yeah. And do you think that that's where the real human engineering piece of this development still will be with us for some time? Because to some degree, you can describe let's say describe that sort of problem to my AI coding assistant. Is it reasonable for me to think that that kind of debugging and process could help me, or that kind of assistant type of thing could help me get to the bottom of this or update my retrieval pipeline and that sort of thing?
• 26:13
  Rajiv: So I'm kind of optimistic that the reasoning models that we have now are going to get us much farther towards helping you solve that problem. Now, of course, that reasoning model's got to understand how you're thinking about how to solve that problem. But already today, if you take the traditional Rag approach, for example, but you pair it with one of the reasoning models that can make tool calls, can kind of look at the results that come back, think about it, decide, hey, I want to re query it in a different way. You can improve the quality of your results in that way by using that reasoning to do that. So I'm pretty optimistic that we're going to keep finding new ways as long as there are workflows that we can train these models on that are fairly, let's say, logical.
• 26:57
  Daniel: Or typo. Yeah, exactly. Gotcha.
• 27:00
  Rajiv: A way we can connect the dots and teach the models to do that, that I think a lot of these things that if we give it the budget for spending time thinking, doing those tokens there, it's going to cost us more latency, but we're going to see better results. And I think some of us, we already see that in using some of these tools like deep research, where we can see that by spending more time on the task, it's able to give us a better result.
• 27:25
  Daniel: Yeah. Yeah. And one area that we haven't talked about yet is this sort of world of agents, which I know is loaded term, it's probably related to what you were just talking about in terms of time of compute and steps in the process and the reasoning models and all of those sorts of things. Could you help us parse apart from your perspective? It almost seems to me like it's one of those sure you remember when it seemed like every conversation I got on, the first part of the presentation was, what is the difference between data science and AI and machine learning, or the difference between machine learning and AI or whatever.
• 28:11
  And at a certain point, I was like, well, these terms all mean nothing essentially because people use them so interchangeably. I feel like that's sort of where we're getting with agents and assistants and all of these sorts of terms. But from your perspective, I guess if that word agent has a meaningful difference to you, what stands out in your mind there?
• 28:33
  Rajiv: Yeah, and I think we all like the idea of this agent, right? Like something I can give a problem to and it solves the problem. And now I think where the definitions break down to is how much autonomy is this agent, how structured is what we do like that. But if we just think back about the bigger picture of like, I have a problem, it's not a straightforward problem that I want to give to an agent. Now, there's at least two different ways that we can kind of tackle this.
• 29:01
  One is I can give them a step by step list of instructions. And this is what we call a workflow often, do this, do this, do this. And then I can check their work at every step and make sure that they're on the track to solve it. Or I can just be like, this is the difference maybe between my kind of five year old and a 13 year old, my 13 year old, like I'll give them the list and I'll cross my fingers and hope that they'll finish it. And, you know, usually they do, but not always.
• 29:26
  But I'm not involved in every step of the way. And I think one of the things is we're watching the agents evolve and this is one of the big trade offs that developers have today is how much structure, how much babysitting am I doing for this agent? How do I do versus kind of the hands off? Now, I think the trend is we're going to be able to do much more hands off. Just like we've seen these models be able to gain the reasoning ability over the last two years or so, I have no doubt that we're going to be able to train them to do more complex tasks, to be able to follow those steps.
• 30:01
  It's just a matter of kind of giving them the training data, having the experience to do that. So my bet is in the long term for many of these tasks, we'll be able to be much more hands off and the models themselves will strive to be able to solve them themselves.
• 30:17
  Daniel: I just thought it would be good to get your input on a theory I've been having, which is maybe related. I mean, maybe it's an offshoot from what we're talking about, which is really maybe the ability to use these Vibe coding tools or others to or assistance or agents to update retrieval processes or kind of architect our AI pipelines, if you will. I've had this sort of thought, and I'm curious about your opinion because you also have a background kind of pre generative AI in the data science world that previously I had in my mind this mental model of, on the one side, you have traditional software engineering, DevOps, infrastructure. On the other side, you have business and the product and marketing, all those things. And in the middle is data scientists because you translate the business problems and understand how to connect it to the data and the tech and produce your predictive model.
• 31:21
  You're living between those two worlds. It's almost like I see that middle zone shrinking and shrinking and shrinking because those domain experts on the business side are actually able to use very sophisticated tools now to kind of self serve themselves a lot of the kind of maybe stuff that would normally fit on the plate of a data scientist. So part of me is wondering, me personally, like Daniel Witenack as a data scientist, what is the future of that data science world when this middle is shrinking? I'm curious if you also see it that way or see it slightly differently, and what your thoughts are in terms of that view. Yeah.
• 32:07
  Rajiv: So I don't think the data science world is changing at all. First of all, I'm excited that the bar is kind of dropping in terms of people being able to use code to build solutions. Like my nine year old can literally like vibe code of a game that he can play as well as my 23 year old who has a who has a degree in computer science. And you couldn't tell the difference between like the games that they built like that. So there's a great ability in just allowing everybody to be able to kind of more participate and work kind of with code that we're being able to see now.
• 32:40
  Now, how does that change something like data science? Like data science, the original triangle was, part of it was coding. I think data scientists were never thought of as really great coders, which is why were 80 trying to put in
• 32:56
  Daniel: percent of those projects didn't make it past pilot too.
• 32:59
  Rajiv: Exactly. Right. Like they would not write production code and right. There was MLE engineers kind of became the offshoot to kind of do the production piece like that. Now, for me, it's a similar thing to like if you think of like journalists in media, right?
• 33:14
  Everybody says, oh, right, everything's going online. We're not going to have any journalists. Well, if you think at the end of the day, a journalist is a storyteller telling you about kind of the facts of what's going on. For me, the data science is a similar piece where it's still kind of a mission, a work that we're doing in terms of we're helping a business solve problems by looking at their data. The tools have changed, but the same problem still exists.
• 33:42
  And I think this is the most important thing is you still need a flexible mind as a data scientist to be able to look at data, to be able to talk to a stakeholder, to be able to go out and figure out what is the coding, what is the math, the algorithms to bring to solve that problem where you need a lot of this kind of left brain, right brain stuff. And so it's still a fairly unique role. And you can see this where you start talking to like AI engineers, where you have developers that are trying to kind of bridge this and solve the business problems. And we see one of the biggest problems they have is with evaluations. And for data scientists, they're trained on how to do evaluations coming up.
• 34:25
  Like you look at the data, you talk to people like error analysis, something that's built into kind of data scientists. But I always look over and see like how we have to kind of teach software developers that skill if they really want to be able to kind of do the same kind of work like that.
• 34:41
  Daniel: Yeah, that's super interesting. I kind of posed the question in maybe a little bit of a controversial way. I think I echo what you're saying. I mean, there's elements of this in certain cases, in certain industries where, hey, if you're using computer vision to analyze parts coming off of medicine coming off of a pharma manufacturing line and needing to do that 10,000 times a minute. This is not a problem that is like, hey, just prompt an LLM.
• 35:21
  On one side, there's very hard modeling problems that need to be solved there. I think on the other side, to what you're saying, I also see this gap around AI engineering where it's, Okay, we can architect the pipeline. And where I see a lot of people spinning their wheels is saying, Well, it seems like this is working. That's kind of where they end. And like, Well, we could also measure if it's working and construct a test set and maybe automate that.
• 35:52
  And as we update our pipeline, we could test the retrieval and those sorts of things. Love that perspective around the evaluation especially. I guess you see that side being stressed more and more as maybe software engineers see that the future is AI related and really want to push into that?
• 36:14
  Rajiv: So I see that there is kind of a growing emphasis for developers and engineers to understand evaluations and do that. My thing is that it's always going to be a little bit of attention for those folks because often the folks that are really good at software development have a very black and white way of looking at the world, that they focus on optimization, that there is a best solution, which necessarily isn't the same type of mindset that you need. And of course, this is a graduated spectrum. Everybody's a little different. So this is where I think there's always going to be that gap between just having kind of software developers fully step into it.
• 36:56
  But I want to take up one other thing that you say is a lot of times, like the hype we see around generative AI and NVIDIA and stuff kind of draws out and makes kind of the problems that we can solve with generative AI kind of much bigger than I think the actual usefulness of them. And what I mean is that there's a lot of problems inside an enterprise that can be solved without large language models. And my worry is the folks inside them that have been doing data science for ten years know that. They know that I could use operations or optimization to solve this problem, or this is a time series problem to do that. My biggest fear is the people coming into kind of AI and data science nowadays aren't seeing those types of problems and understanding that there's a whole set of tools to be able to solve those problems where often kind of everything comes.
• 37:47
  We're using generative AI as the hammer for solving every type of problem like that.
• 37:52
  Daniel: Yeah, maybe this exists out there. And so I'm already building something, so someone can totally steal my idea if they want. But I wonder if there exists out there this kind of idea of some sort of assistant that would live at the orchestration level above these kind of traditional data science tools and help you towards that analysis. So just by way of example, I'm assuming you could put Facebook's profit for time series forecasting behind an MCP server and be able to have that discovered by this orchestration level and maybe guide people to that. Now, that might not be the interface that you want to have for your time series modeling in production, but it could potentially guide folks to some of traditional data science tools and kind of help teach them maybe what they need to put in place that's not on the large language model side and actually have the large language model tell you that, Hey, I'm not the best for this and you should use Facebook profit or whatever.
• 39:04
  Rajiv: Yeah, I'm hoping that we'll be able to get to that. I think there's some element of These models are great for brainstorming, thinking through things, solutions through, too. But if the space is too large of possibilities of different ways to slice the problem, different ways to think about how you could set the predictions or what data to use, then even in LM, you're not going to be able to feed it all the relevant information to be able to actually kind of make that decision. This is where as humans, we have to kind of often be the piece that takes in a lot of that disparate information and figure out like, okay, this is what the business really focuses on. Let's zoom it down to this piece.
• 39:42
  And now I'm going use my LM to help me think about, hey, there's three different tools here and strategies, like tell me the trade offs, let's figure out which I was doing this earlier today, like which package should I spend my time learning how to use to solve this problem.
• 39:55
  Daniel: Yeah, yeah, that makes sense. Well, Rajeev, I'm sure we'll have many other great conversations at the Midwest AI Summit and at upcoming or future podcast episodes. But as we kind of get closer to the end here and you look out towards the future, what is it that kind of excites you about the next steps of our journey in this space?
• 40:21
  Rajiv: Yeah, no, I mean, it's just been a great time of innovation inside of data science, which is why I love it. I mean, everything from kind of going from XGBoost to CNNs to kind of where we are now. And so I'm looking forward to more innovation, especially in kind of the area of large language models. But I also want to remind people, like we were talking about, there's a great wake of tools that are out there that I still like to point people to that might not get the most attention, but there's a lot of times a more efficient way of solving your problem as well.
• 40:52
  Daniel: Yeah. And where would you Obviously, you produce a lot of content and that sort of thing, but as just a person that's more intimately familiar with that kind of ecosystem, if folks are like, Hey, you know, I heard, for example, I'm at a Rally Innovation Conference here in Indianapolis today, off in a corner, and I heard Kevin O'Leary from Shark Tank this morning, he was saying, Every day you should spend 30% of your mental capacity trying something new, like keep those juices flowing. So maybe it's our listeners today, they're taking away, hey, I should try one of these non Gen AI things. Like, where would I even go to to start that? Any any suggestions?
• 41:40
  Rajiv: Yeah, no, I love that idea of like spending a thirty minutes or an hour a day, like continual learning is the is the future like that. So I have my own content that I put out at logistics that tries to kind of inspire you to push you in different ways that kind of AI is doing, give people simple kind of nuggets like that. So I would of course kind of point to myself as well. I think the other area that I really like are newsletters. I think newsletters are a nice way to be able to take in all the information that's coming in, but in a little bit of a slower kind of meditative way rather than just kind of reacting to the latest trending post.
• 42:18
  Daniel: Yeah, that's awesome. And I'm sure we'll include a few links in our show notes to things that will be useful for people. But really appreciate you joining us again, Rajeev. Looking forward to seeing you in person. Yeah, keep up the great work.
• 42:38
  It's always good to hear your perspective and looking forward to having you on the show again.
• 42:43
  Rajiv: Thanks so much. I think this is one of the longest running data science podcasts out there, so it's been great to be part of it. Thanks so
• 42:49
  Daniel: much. Thanks.
• 42:57
  Rajiv: All
• 42:57
  Jerod: right, that's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com.
• 43:19
  Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.