Latest / The Adults In The AI Room / Optimizing Product Data and Harnessing Generative AI - Sanjay Mehta - The Earley AI Podcast with Seth Earley - Episode #035

Transcript

Optimizing Product Data and Harnessing Generative AI - Sanjay Mehta - The Earley AI Podcast with Seth Earley - Episode #035 · The Adults In The AI Room

• 0:00
  SPEAKER_01: This is the Early AI Podcast hosted by Seth Early First First Science.
• 0:05
  SPEAKER_01: Join us as we sell deep into the passions, expertise, and experiences of public practitioners to talk about what's possible with artificial intelligence.
• 0:13
  SPEAKER_01: The Early AI Podcast is sponsored by Early Information Science, the digital transformation journey with design and deployments of innovative technology solutions, as well as DMSY.
• 0:24
  SPEAKER_01: Now, enjoy the show.
• 0:28
  SPEAKER_04: Welcome to today's podcast.
• 0:30
  SPEAKER_04: My name is Seth Early.
• 0:31
  SPEAKER_04: And I'm Chris Featherstone.
• 0:33
  SPEAKER_04: Our guest is a seasoned technologist and solutions evangelist with rich experience in internet software engineering.
• 0:39
  SPEAKER_04: He's the head of industry commerce at LucidWorks, Sanjay Metto.
• 0:44
  SPEAKER_04: Welcome to the show.
• 0:46
  SPEAKER_02: Thank you for having me.
• 0:47
  SPEAKER_04: Yeah, it's great to see you.
• 0:48
  SPEAKER_04: And uh, you know, as we get started here, one of the things I like to ask is, you know, what is a common misconception about our world right now that is being kind of overrun with hype around things like generative AI.
• 1:03
  SPEAKER_04: Uh, and I know you you, if you have any other misconceptions about the things that you work with, that'd be great.
• 1:09
  SPEAKER_04: But let's let's jump in there.
• 1:12
  SPEAKER_02: Yeah, a couple things um that at least we've been observing um in the industry really is that um the myth that it's a panacea, right?
• 1:23
  SPEAKER_02: It's gonna solve everything, um, and that it is super easy to uh drive results and value is is uh most likely you know not really not true um as far as we're seeing, right?
• 1:37
  SPEAKER_02: There's a lot, a lot of complexity.
• 1:40
  SPEAKER_02: Um, you know, there's a lot of uh uh lot of nuances um as it relates to even different businesses and industries and how it's applied.
• 1:49
  SPEAKER_02: And and really um it's it's not turnkey.
• 1:52
  SPEAKER_02: I guess that you know, for the consumer side, uh definitely we're all seeing some aspects of it being kind of easy from like a chat GPT or BARD perspective for those who are using that.
• 2:03
  SPEAKER_02: Uh, but from more of a B2B perspective, there's quite a bit of complexity, especially.
• 2:10
  SPEAKER_04: And you know, we met at uh that conference, uh, I forget where it was now, but um, is it San Diego or something?
• 2:17
  SPEAKER_04: Right, San Diego.
• 2:18
  SPEAKER_04: And um, and I know you had a presentation, and I stole a couple of your slides for my presentation too, because it reinforced the message so well.
• 2:26
  SPEAKER_04: And uh you were going through a number of use cases.
• 2:28
  SPEAKER_04: And what do you think are the most compelling and important use cases?
• 2:32
  SPEAKER_04: And what do you think executives really need to know about generative analysis to support those types of use cases?
• 2:39
  SPEAKER_04: So you talked about consumer, yes, they can just you know use it to write papers and you know resumes and letters and you know, you know, do their homework and and all those things.
• 2:51
  SPEAKER_04: But you know, for uh for corporations, it's gonna be a little bit different in a in a number of different areas.
• 2:57
  SPEAKER_02: Yeah, absolutely.
• 2:58
  SPEAKER_02: I think the main uh area right now is folks are there, there's you know, probably technically millions of use cases now, right?
• 3:05
  SPEAKER_02: You have a lot of brains kind of up uh working on this problem area.
• 3:09
  SPEAKER_02: I think uh mostly for businesses identifying, you know, where uh the the the most value and and lowest risk, right, uh can be generated.
• 3:20
  SPEAKER_02: Um you know, and and I think uh they really have to realize the cost, and you know, really think about an ROI for these because these are very expensive, as uh you can imagine, right?
• 3:30
  SPEAKER_02: And determining where you can't just let it answer every type of request uh that you have or send all your data there.
• 3:37
  SPEAKER_02: You have to be uh you know very careful about um the use and adoption of it and making sure you're measuring that you're getting the ROI you want.
• 3:46
  SPEAKER_02: Otherwise, you're gonna end up um being a sinking ship in a way, right?
• 3:51
  SPEAKER_02: If you think about the costs that might might uh evolve.
• 3:56
  SPEAKER_04: Yeah, no, that makes a lot of sense.
• 3:58
  SPEAKER_04: And um, what are the things that you're seeing that organizations want to do with this?
• 4:01
  SPEAKER_04: I know there's a lot of uh um a lot of uh interest in customer support, um, whether you're supporting the customer directly or supporting the agents that custom that support the customer, and there's some in e-commerce.
• 4:12
  SPEAKER_04: Do you want to talk about some of those?
• 4:14
  SPEAKER_02: Yeah, that's a great use case.
• 4:16
  SPEAKER_02: Um, where MIT had cited some um recent research that's showing that um using large language models, for example, in chat, is uh improving resolution of inquiries problems by over 17%.
• 4:35
  SPEAKER_02: So that's obviously a lot of value.
• 4:37
  SPEAKER_02: And um that's a great one where you think about um the cost the cost that uh is involved in in getting a you know a human or a call center agent to respond versus the cost for um you know a chat bot, for example, using uh chat GPT to respond are uh you know, a big there's a big cost savings again um from that perspective.
• 5:05
  SPEAKER_04: And uh and again, best practices there, right?
• 5:08
  SPEAKER_04: The the chatbot and and the LLM needs to know how to solve those problems, right?
• 5:14
  SPEAKER_04: I mean, it's you know, it's it's just not going to, you don't want to just let the large language model run off and figure out your support questions, right?
• 5:22
  SPEAKER_04: So how do organizations deal with that?
• 5:25
  SPEAKER_02: Yeah, a lot of these are uh foundation foundational models as we call them.
• 5:29
  SPEAKER_02: Um, and uh there's a couple of things, you know, that you can do, of course.
• 5:34
  SPEAKER_02: Um we think about prompting, um, and you have a lot of control over the way you prompt and your prompt templates, right?
• 5:41
  SPEAKER_02: And uh the parameters you send in your prompt.
• 5:44
  SPEAKER_02: Um, and and then uh what we call like control generation too, right?
• 5:48
  SPEAKER_02: Which is uh, for example, what what uh we call it temperature or penalties or those sorts of things that you can send where you don't allow the model to improvise too much, right?
• 6:01
  SPEAKER_02: Kind of stay close to the source.
• 6:03
  SPEAKER_02: Um, citations also are you know another important thing, more from a quality perspective.
• 6:08
  SPEAKER_02: And and I think with all these, most of the models allow, of course, fine-tuning and training.
• 6:14
  SPEAKER_02: And so using historical data, for example, a historical QA, or what you can do is uh you have uh synthetic uh QA pairs or synthetic training data generated, even using these models uh to help refine it, uh refine the capabilities and like eliminate a lot of the potential risk and interact responses.
• 6:36
  SPEAKER_05: You mentioned something too centered around like um, you know, this notion that I mean, really realistically, I mean, no, there were people working on it, but since about December, nobody was really talking about it, right?
• 6:50
  SPEAKER_05: Until about December, and then it started to come on, you know, on strong.
• 6:54
  SPEAKER_05: And I see that especially, you know, like you were saying, there's nothing, you know, nothing that you know that is is just a one size fits all, especially in the consumer world.
• 7:02
  SPEAKER_05: But how has that consumer bias affected you in the in in business?
• 7:07
  SPEAKER_05: And have you seen that?
• 7:09
  SPEAKER_02: Yeah, that there's uh it you know, it does uh vary by industry, right?
• 7:14
  SPEAKER_02: Um, and uh, you know, particularly um in industries that are like if you think about retail, uh there's definitely um a lot of bias in in certain things when we think about the use cases there where uh the the use case might be more around um helping generate content, uh right, um and and content for campaigns, for sites, those sorts of things for SEO, right?
• 7:45
  SPEAKER_02: Versus uh if you look at another industry like financial services, uh the focus might be more on um driving productivity of analysts and research, where they're mining data, for example, you know, um retail for sure.
• 8:00
  SPEAKER_02: We might see uh more of a consumer side, but uh where it's more personalization, uh, where they're looking at you know personalized outcomes of these models versus uh like an internal use case where it's probably more focused on roles, right?
• 8:15
  SPEAKER_02: Um, and and uh their trade, for example.
• 8:19
  SPEAKER_02: Those are the kind of um kind of things.
• 8:22
  SPEAKER_02: But and then of course, um when we think about other uh things folk folk people focus on uh in different industries, customer facing again, privacy is a big focus and compliance, right?
• 8:34
  SPEAKER_02: Versus enterprise or workplace applications.
• 8:38
  SPEAKER_05: Yeah.
• 8:38
  SPEAKER_05: Have you seen anybody in your organization, you know, like either I think both ends of the spectrum missing the ball as well as just you know, way ahead of it?
• 8:49
  SPEAKER_05: And what you know, how are you helping pull that back?
• 8:52
  SPEAKER_05: Because we see it all the time.
• 8:54
  SPEAKER_02: Yeah, I think we're uh building really a concrete solution, um, so not just so focused on like the single technology, again, like the GPTs can do everything.
• 9:06
  SPEAKER_02: Um, it is, I think, uh a myth or a fallacy, right?
• 9:11
  SPEAKER_02: It actually has to be combined with um other capabilities to really create a compelling use case, right?
• 9:18
  SPEAKER_02: In our case, uh it might be combining it with search, uh, it might be combining it with recommenders, it might be combining it with data enrichment, uh, but having it uh, you know, just uh you know, plugging it into your site kind of as a chat bot, there's a lot more involved, a lot more required, and you might not get the results you expect, right?
• 9:38
  SPEAKER_05: So I'm I'm glad you brought that up.
• 9:41
  SPEAKER_05: That notion of of uh that there's not a one size fits all.
• 9:46
  SPEAKER_05: And I think people get a misconception that it's not only as a one size fits all for all of AI, and now we've pivoted to that is AI, and which is absolutely wrong.
• 9:57
  SPEAKER_05: And the same side of that is the ability to know how to do get discrete and discriminative type of approaches to figure out patterns and pattern recognition, cluster things and pull that into a scenario where you can utilize a generative model to create something really interesting, especially as a filter for human-related content, right?
• 10:18
  SPEAKER_05: So, anyway, so yeah, I'm glad you brought that up because it's I think it's super important.
• 10:22
  SPEAKER_05: No, plus the hundreds of models that are out there today and that exponentially keep growing and the size of those models, and you know, it's getting you know out of control, but it speaks to you know, we wouldn't have multiple models unless it was something very task-oriented specifically for some semantic tasks.
• 10:40
  SPEAKER_04: So specialization talk talk a little bit about um uh your space, search, and how it's impacting search, because I know uh you started to make that question, uh, make that point.
• 10:54
  SPEAKER_04: And uh my belief is you know it's it's search is an integral tool, especially if we uh have a target uh knowledge or content source and we want to turn the temperature down and then tell it to only use these sources, and then we can process the query, plus we can process the results.
• 11:10
  SPEAKER_04: But talk a little bit about how you think of it and how you see that in search.
• 11:15
  SPEAKER_04: And uh maybe you can also touch on product search, because I know that was a great uh example you provided at that conference.
• 11:21
  SPEAKER_02: Yeah, yeah.
• 11:22
  SPEAKER_02: Yeah, and it's it's simplest form.
• 11:24
  SPEAKER_02: Um, you know, I think it supercharges query understanding.
• 11:28
  SPEAKER_02: Um, you know, there's I think the the primary use case, right, is is conversational search, right?
• 11:35
  SPEAKER_02: Not just uh lexical or semantic now.
• 11:38
  SPEAKER_02: I mean, they're all kind of elements of it, but it's turning purely where you can use your dialogue, your natural language, and and and and speak to and and let the machine particularly understand uh what you mean.
• 11:50
  SPEAKER_02: And and I think that's particularly important when it comes to voice applications, right?
• 11:55
  SPEAKER_02: I think many of us have been using voice assistance for some time, and I think we're already getting used to the machine doing certain things for us, but now having more of a freeform natural conversation um and extracting intent, right?
• 12:11
  SPEAKER_04: Not just identities and results, but it's really extracting intent is yeah, I I say it's at the source on steroids or or or sentence completion on steroids, right?
• 12:22
  SPEAKER_04: Because anytime you, you know, what it's doing is it's trying to understand what how this phrasing and what this person asked is actually what's all ingested into a vector space, and then you look for well, what's the common terminology, what's the common intent for all these different ways of representing these ideas?
• 12:41
  SPEAKER_04: And then that becomes the intent, and that can be uh compared to other vectors which have the answers, right?
• 12:48
  SPEAKER_04: So it's it's an interesting way of thinking about it.
• 12:50
  SPEAKER_04: It's like people have lots, even when we're doing chatbots, you have to do uh intent classification, and this is kind of the same idea, but on steroids.
• 12:59
  SPEAKER_04: Would you agree with that?
• 13:00
  SPEAKER_04: And what else would you would you add to that?
• 13:02
  SPEAKER_02: Yeah, exactly.
• 13:03
  SPEAKER_02: And and you brought the product sort of catalog use case, and you know, a big one there, of course, is um most folks we've uh I've I've actually seen in my 20 years in the industry have always kind of struggled with product data.
• 13:16
  SPEAKER_02: I think you guys have seen it directly, whether it's taxonomy, classification, you know, uh simple labeling, right?
• 13:23
  SPEAKER_02: And and uh this really is where you actually get sort of the intelligence um that's available or insights that are available around kind of the entire web, right?
• 13:35
  SPEAKER_02: To actually inform your your products and enrich them and inform your categorization, right?
• 13:41
  SPEAKER_02: Based on really not just your own opinion of how uh the the you know the catalog or or or or navigation should be, but really what what is the uh what is kind of the the the universe of of content in that industry or product in that industry telling you, right?
• 13:58
  SPEAKER_04: Yeah, and I think there's a double that's a uh double-edged sword, in my opinion, because you don't want to be too standard and look like everybody else, you still want some differentiation, but there are times when you want, well, a standard description of a screw or a nut or a you know what a piece of equipment or right?
• 14:14
  SPEAKER_04: So so some things don't have to be uh that creative, and other places you do want some variability and some differentiation.
• 14:22
  SPEAKER_04: Um, but I remember when you showed me the example at the uh conference, you I asked the question did you ingest the product catalog into the vector space?
• 14:31
  SPEAKER_04: And you had said yes.
• 14:33
  SPEAKER_04: And in that case, we need to start with good product data, right?
• 14:37
  SPEAKER_04: So there's the enriching of the product data, which the gener generative value can help with.
• 14:41
  SPEAKER_04: And then once we enrich it and once we normalize it, then we ingest it into vector space.
• 14:46
  SPEAKER_04: Do you want to talk a little bit about that?
• 14:48
  SPEAKER_02: Yeah, and and and most folks probably have some level of you know description or title or variance.
• 14:56
  SPEAKER_02: If we're talking about apparel, for example, like color.
• 15:00
  SPEAKER_02: Um, there's some you know um basic data they may have.
• 15:04
  SPEAKER_02: Um, and and then things like maybe the activity or the actual um, you know, how to use it, how to maintain it, those sorts of things may uh they're very important in your decision making, but maybe coming from um the outside world, like you know, ratings, reviews, communities, things like that, which the LLM is like fantastic at.
• 15:28
  SPEAKER_02: If you think about vectors spaces, since they're can support any number of dimensions, there's the other one of context.
• 15:34
  SPEAKER_02: And um, if you think about uh context and personalization, adding that layer where you do tie in signals like behavioral data, other contexts, like there may be regional nuances.
• 15:45
  SPEAKER_02: We see that a lot, right?
• 15:47
  SPEAKER_02: There could be um, there could be even seasonal nuances depending on the product, right?
• 15:52
  SPEAKER_02: There could be all kinds of weather impact, those sorts of things.
• 15:56
  SPEAKER_02: And and you can add those sorts of uh dimensions, right, into your space and then drive much more powerful relevance if you have that user coming in, right, with that type of context.
• 16:09
  SPEAKER_04: Interesting, yeah.
• 16:10
  SPEAKER_04: Uh, and and do you want to talk a little bit more about you know how your technologies kind of facilitate that or what the and not to be a commercial, but just to get into the the tactics of you know, what are you what are you guys doing these days in terms of leveraging that stuff and what's kind of still on the horizon for you?
• 16:28
  SPEAKER_04: Like where how has it changed your business and then how is it being applied and when then what do you expect in the future?
• 16:34
  SPEAKER_04: I know it's a lot of questions in there.
• 16:36
  SPEAKER_02: No, definitely I would appreciate, you know, as a kind of a search and an AI-based search platform, right?
• 16:42
  SPEAKER_02: I think that there's a whole uh set of of uh of like a journey, let's call it, that data goes through, right?
• 16:48
  SPEAKER_02: And uh from the sourcing piece, right?
• 16:51
  SPEAKER_02: That's a big factor here where uh you may source from uh you know any number of structured, unstructured, semi-structured sources.
• 17:00
  SPEAKER_02: You can crawl again, uh, you know, data, and particularly when we're thinking about large language models and um data protection is a big concern, right?
• 17:09
  SPEAKER_02: Um protecting your data.
• 17:11
  SPEAKER_02: Uh, this is kind of getting into where you know you might be just sourcing internal protected data and the security, we call them like security trimming, which is not everyone can see everything, and depending on your role and those sorts of things.
• 17:23
  SPEAKER_02: So that's a fundamental piece of you know, one one aspect that the platform brings.
• 17:28
  SPEAKER_02: And then, of course, when we think about indexing and ingestion indexing enrichment, that's again a big opportunity where uh we have new models.
• 17:38
  SPEAKER_02: We've always used some sort of models, right?
• 17:40
  SPEAKER_02: Like uh uh knowledge knowledge bases and uh you know NLP libraries, right, to uh do some degree of enrichment analyzing this content.
• 17:49
  SPEAKER_02: But now we have these large angular models that we can invoke in that.
• 17:53
  SPEAKER_02: And one benefit of that is uh it reduces sort of the cost from a query standpoint, right?
• 18:00
  SPEAKER_02: Because the data is already kind of good to go.
• 18:02
  SPEAKER_02: Uh, you're not making these types of requests on every every query or every search or every page view, for example, right?
• 18:09
  SPEAKER_02: So there's you know a lot of benefit of having it in the index or a vector space or combined.
• 18:14
  SPEAKER_02: Um, and and then uh another big piece we support our graph databases or knowledge graphs, and and again, we talked about vector databases being a big piece to support neural hashing, for example, is one of the uh aspects we call it hybrid new neural hashing, which is kind of combining a few things together.
• 18:31
  SPEAKER_04: Do you want to talk more about that?
• 18:32
  SPEAKER_04: What that means in terms of neural hashing.
• 18:35
  SPEAKER_02: Yeah, it's a bit you know, the hashing's been around for a while, and and uh again, uh part of it, what you're what you're trying to get is, for example, nearest neighbors or approximate nearest neighbors in certain topics.
• 18:47
  SPEAKER_02: The way to get that is there's a number of ways.
• 18:50
  SPEAKER_02: One of the more common ones that are out there in the industry right now is neural hashing, right?
• 18:54
  SPEAKER_02: The kind of cosine similarity was another one uh that's been out there again, using some mathematics to find the proximity between certain embeddings, right?
• 19:04
  SPEAKER_02: In which case could be topics and things like that.
• 19:06
  SPEAKER_02: So uh we would be used to build those sorts of things.
• 19:10
  SPEAKER_02: And then finally, if you think about the query standpoint, right?
• 19:13
  SPEAKER_02: Um, so having that query engine where it can analyze queries, you might route queries to different models.
• 19:19
  SPEAKER_02: Um, you might break down um and and create create snippets uh both ways uh from you know query and pass it to different models, or vice versa, from ingest.
• 19:29
  SPEAKER_02: And then of course the filtering, faceting navigation, uh depending on your use case.
• 19:34
  SPEAKER_02: If it's chatbot, that's pretty straightforward.
• 19:37
  SPEAKER_02: But if you're embedding these into search, how do you generate you know the faceting and filtering and navigation over this sort of uh result?
• 19:45
  unknown: Right.
• 19:46
  SPEAKER_05: Oh, go ahead, Seth.
• 19:48
  SPEAKER_04: Go ahead, Chris.
• 19:48
  SPEAKER_04: You go ahead.
• 19:49
  SPEAKER_05: I was gonna say, you know, given um, you know, what you're working on with a lot of these things, I'd love to get a perspective too from you on one.
• 19:57
  SPEAKER_05: Um, and it's not necessarily a roadmap question, but more along the lines of, you know, with you guys testing, I'm sure you're testing a ton, but what does that look like in terms of from your perspective, how long it's going to take it to get into actual product to be used with customers, and how much time you're taking with testing and making sure that it's bulletproof before it goes, right?
• 20:21
  SPEAKER_05: Hardening stuff is not easy, but you know, this comes in the same vein.
• 20:25
  SPEAKER_05: I'd love to get your perspective on what that looks like without you don't have to share anything roadmap-wise, but what does that look like in terms of you know how long it's going to take to get to product?
• 20:35
  SPEAKER_02: Yeah, so we we actually uh have already uh built uh a product.
• 20:40
  SPEAKER_02: Uh we, you know, large language models, you know, you go back even like Bert, you know, these things have been around for a while.
• 20:46
  SPEAKER_02: Um, and and so uh, you know, uh there depends on like if you think about what Google or Amazon or or or Azure is offering to actually build something is is actually pretty quick now for a lot of folks.
• 21:01
  SPEAKER_02: To build a complete solution that's integrated with, for example, a search platform, you know, that's again uh depending on you know how your design or your architectures, if it can support that interchangeability of technology, right?
• 21:14
  SPEAKER_02: Or APIs, for example, I think that's a big one.
• 21:17
  SPEAKER_02: Um, so you know, I've seen uh when we think about customer implementations, you know, uh a lot of it is depending on how it's being used, if it's for uh embeddings and enriching data, which we've already been doing uh with a number of customers, uh, you know, and then again, a lot of that's human in the loop kind of the you know, the feedback piece of evaluating something like that could take like you know, three to six months.
• 21:43
  SPEAKER_02: Um but when you think about you know real like end to end, like changing your entire search and and including personalization and all that, you know, you're probably the north of that then.
• 21:55
  SPEAKER_05: Yeah, I mean it was just yeah, it was part of that you know, that notion.
• 21:59
  SPEAKER_05: I know it's semantic.
• 22:00
  SPEAKER_05: And it depends on the use case, depends on the product, all that kind of stuff.
• 22:03
  SPEAKER_05: You know, we see a lot of folks that are that are going into it from the perspective of very, very guarded, which you don't, I don't think you have to be.
• 22:12
  SPEAKER_05: I think it's just part of the same constructs that you have for hardening your stuff and going to production anyway.
• 22:18
  SPEAKER_05: Um, and yet we see a lot of folks that are pulling in legal folks around, you know, what this means and how much you know to put into the and it's it goes back to listen, you're using a foundation model.
• 22:29
  SPEAKER_05: If you're using a retrieval augmentation perspective, great, we're not doing anything except for filtering the results through a main a model to elicit the right you know response.
• 22:40
  SPEAKER_05: If we're you fine-tuning it, that's a different ball of wax, but we're fine-tuning it with our information still behind all of our guardrails.
• 22:46
  SPEAKER_05: And it's not, you know, it's not that different.
• 22:50
  SPEAKER_05: And yet, to find irony with a lot of these environments, all their sensitive data is sitting in cloud-based systems anyway.
• 22:58
  SPEAKER_05: Like they have a core of a, you know, of a not necessarily a threat vector that's open, you know, in having a cloud-based anything, but their sensitive data is sitting inside of this, and now they're worried and concerned about, let's say, a large language model and/or generative AI.
• 23:14
  SPEAKER_05: And I get it can hallucinate.
• 23:15
  SPEAKER_05: We can put guardrails around it, but at the same time, it's just part of the same damn stuff that we've been doing for you know decades.
• 23:22
  SPEAKER_05: So anyway, you get it.
• 23:25
  SPEAKER_02: Yeah, yeah.
• 23:26
  SPEAKER_02: That there's the one big one is that I've heard, and even at the conference, me and Seth were from some um organizations was that uh yeah, they don't want like specific queries that that their customers are asking, or their employees are more importantly, their employees are are asking to appear in the model, especially the responses, because they could be competitive, they could be put them at risk, um, depending on you know, a lot of industry.
• 23:52
  SPEAKER_02: So that's obviously the big, big one, right?
• 23:54
  SPEAKER_02: Like in and then and not making that data or signals available to others.
• 23:59
  SPEAKER_05: I mean, I I love that anthropic is doing some things, you know, with Claude, like you can't, sorry, you can't answer some things.
• 24:06
  SPEAKER_05: Great.
• 24:06
  SPEAKER_05: You know, you can't do anything that's nefarious and that those kind of things.
• 24:09
  SPEAKER_05: Now, you know, part of that people try to get away with, hey, simulate this or give me an example of this or whatever.
• 24:16
  SPEAKER_05: And but, you know, just like you said, it's it goes into the prompting, and the prompting is as much as what you want to respond with, as much as as well as you know, as much as what you don't want to respond with.
• 24:29
  SPEAKER_05: And I think people lose sight of that, right?
• 24:31
  SPEAKER_05: Around how to put guardrails on them.
• 24:34
  SPEAKER_05: But anyway, yeah, it's interesting.
• 24:36
  SPEAKER_04: Uh Sanjay, I wanted to kind of go back uh and talk a little bit more about search and retrieval because again, uh a large language model is a tool in a toolkit, and in order to use it in an enterprise uh effectively, you really need to specify where you're getting your data.
• 24:55
  SPEAKER_04: Not you're not relying on the large language model generally to answer the questions, right?
• 25:00
  SPEAKER_04: It's doing it based on its knowledge of language and terminology and you know, the stochastic parrot, right?
• 25:06
  SPEAKER_04: The thing that is saying repeating something back because it's learned what words go together.
• 25:12
  SPEAKER_04: So you mentioned a number of different things.
• 25:14
  SPEAKER_04: And I just wanted to revisit that list when you talked about, you know, processing content, processing data, you know, uh uh pulling data from various sources, you know, integrating it, security trimming.
• 25:26
  SPEAKER_04: I mean, there's a whole range of things that have to go, especially into retrieval augmented.
• 25:32
  SPEAKER_04: So I like the idea of saying, you know, it, you know, you have your your large language model is that that uh the source on steroids, then you pull your information, whether it's from a corporate data source knowledge base or a vector space.
• 25:46
  SPEAKER_04: And I do want to talk a little bit about the difference between doing it from a knowledge base that's that's uh structured versus a vector.
• 25:53
  SPEAKER_04: And I know there's a very different thinking there, and then you're processing the results.
• 25:57
  SPEAKER_04: But do you want to just kind of go through that laundry list of things that executives need to think about when they're looking at large language models and their corporate data and search and retrieval?
• 26:08
  SPEAKER_02: Yeah, so fortunately, um the what what one thing the models can do is uh cite uh their source or provide what sources uh they use.
• 26:16
  SPEAKER_02: And a lot of that is used for fact checking.
• 26:19
  SPEAKER_02: There's uh some models out there like RRAR that can uh do some fact checking, uh, for example.
• 26:25
  SPEAKER_02: And of course, retrieval augmented generation, you already know what you're giving to the model, and then you can use that.
• 26:30
  SPEAKER_02: Um, so those sorts of things I think are fundamental, but but but you'll go a little deeper, especially for folks who are uh very um sensitive data.
• 26:39
  SPEAKER_02: Uh it gets into even what snippets of a document you actually can send, right?
• 26:44
  SPEAKER_02: There might be breaking a document down into chunks and and determining kind of what chunks may you might allow and some that you won't, for example.
• 26:53
  SPEAKER_02: Like there's even that.
• 26:54
  SPEAKER_02: So it does allow you to get pretty granular uh from that perspective.
• 26:59
  SPEAKER_04: Um and and then I'm sorry, I don't mean to interrupt you.
• 27:02
  SPEAKER_04: Is that the content source?
• 27:03
  SPEAKER_04: Is that like chunking using data or chunking content and then applying security at that level?
• 27:08
  SPEAKER_04: Or is there another mechanism?
• 27:10
  SPEAKER_02: Yeah, it could be if it's a file, uh it could be a file source, it could be a you know CMS, uh so structured or in a CMS.
• 27:18
  SPEAKER_02: I mean, it could be a crawl, uh, but again, if we're we're able to break down the sections of a document, for example, right?
• 27:23
  SPEAKER_04: Gotcha.
• 27:24
  SPEAKER_04: Okay.
• 27:25
  SPEAKER_04: So I'm sorry, I took you off your train.
• 27:27
  SPEAKER_04: You you kept keep going when you were talking about uh that point.
• 27:31
  SPEAKER_02: Yeah, and and then of course, uh it's it's uh you know identifying the the right model models, or you know, and there's technologies like Langchain, you know, allow you these multiple models together, you know, those those are some things that are very interesting um as well.
• 27:45
  SPEAKER_02: Um, and then of course the training uh of that model.
• 27:49
  SPEAKER_02: So the training data can be historical data you have.
• 27:52
  SPEAKER_02: I think we mentioned your existing knowledge base you may have, or it could be um data you can source.
• 27:58
  SPEAKER_02: Um, if for example, in retail, uh if you're in a specific industry, you can see kind of like Google has uh like insights or trends, it can show you all the queries that occurred for like a certain region, and you can filter that and use that to you know pre-cede your models, a bunch of things.
• 28:14
  SPEAKER_02: But also, um, I think you know, executives need to consider uh measuring uh you know attribution models.
• 28:21
  SPEAKER_02: So for each one of these, uh measuring an outcome.
• 28:24
  SPEAKER_02: Um if we think about a conversion event or a revenue event or a deflection event, right?
• 28:31
  SPEAKER_02: Um, or productivity, like resolution time, those sorts of things, I think are are really key to executives.
• 28:38
  SPEAKER_04: Yeah, exactly.
• 28:39
  SPEAKER_04: And I think those use cases, we've talked in another podcast about building larger use cases, building baselines and then measuring the impact on those baselines.
• 28:48
  SPEAKER_04: And that's exactly what you're saying.
• 28:50
  SPEAKER_04: Uh, quick, quick uh question around the use of a vector space versus you know a knowledge base that you would retrieve from.
• 28:57
  SPEAKER_04: Um when when when is it meant more beneficial to say take knowledge from a knowledge base, ingest it into the vector store versus searching and retrieving and then processing that output with uh with an LLM?
• 29:11
  SPEAKER_02: Yeah, I think from what I've observed from um you know, from a search perspective, uh there's a lot of pre, I would call it pre-seeded knowledge grasp for different industries, as we know, right?
• 29:24
  SPEAKER_02: And it's think about it as uh when you're either building your index, uh that's another again great vehicle both to enrich but also understand your user's context, right?
• 29:37
  SPEAKER_02: And I think it gets interesting too because it it actually links all the topics in a very similar fashion.
• 29:42
  SPEAKER_02: There's definitely overlap with like a vector database, but it's already built and pre-trained.
• 29:47
  SPEAKER_02: And there's again a lot of data sets or knowledge graphs available that are pretty hyper focused on industries, where I think about a you know vector space.
• 29:55
  SPEAKER_02: Yes, there's a few that are there, and you can actually use a knowledge base to help build a vector or combine knowledge to build your own vector space as well, right?
• 30:05
  SPEAKER_02: And um, but I think there's multiple more dimensions to like a vector database.
• 30:10
  SPEAKER_02: When I mentioned like context uh as an example, or even user, or you keep going, there's endless layers.
• 30:18
  SPEAKER_02: That's something that you don't see kind of in a graph database or knowledge graph, right?
• 30:22
  SPEAKER_02: Um, so I think you kind of have to co-mingle in a way co-mingling where you can kind of traverse almost a graph and keep going deeper and deeper rather than looking at kind of the proximity of things.
• 30:32
  SPEAKER_03: Yeah.
• 30:33
  SPEAKER_04: And what are your thoughts about um the quality of the data, obviously, the any metadata usage of metadata?
• 30:41
  SPEAKER_04: Uh again, there's ways, different ways of chunking content.
• 30:45
  SPEAKER_04: Um, you know, some people are thinking that that's not necessary.
• 30:48
  SPEAKER_04: And I'm of the belief that uh that's all important, right?
• 30:52
  SPEAKER_04: And you're gonna get better performance, uh the better quality of your data, and the better, the more uh intelligently you you can uh chunk content.
• 30:59
  SPEAKER_04: That's right.
• 31:00
  SPEAKER_04: Yeah.
• 31:01
  SPEAKER_02: Yeah.
• 31:02
  SPEAKER_02: I mean, you from uh again going to back to executives, costs is a big one.
• 31:06
  SPEAKER_02: You you'd want to send like massive documents, um, you because you're paying by tokens in or input sizes, plus these the sizes are limited, as you know, the input uh sizes are limited, right?
• 31:18
  SPEAKER_02: Uh, you know, what 32k or 8k, 4k, depending on what you're using and what type.
• 31:22
  SPEAKER_02: So you've got to consider some of that's absolutely necessary um when you're when you are um you know taking it taking advantage of this.
• 31:31
  SPEAKER_02: Uh when you think about um training your model too, I think bias is a concern.
• 31:36
  SPEAKER_02: Um, especially if you're using external data.
• 31:39
  SPEAKER_02: Um, as you guys know, there's like fake news, actually, the the machine, this is a big problem, right?
• 31:45
  SPEAKER_02: If you're uh you know, or sort of trends that are are are not uh they're creating a lot of inaccurate and factual data, can actually uh the machine is uh does have a hard time discerning from it.
• 31:57
  SPEAKER_02: It's actually referring to a source.
• 31:59
  SPEAKER_02: It's it's not really improvising too much, but the source, you know, how does it know how great the source is, right?
• 32:06
  SPEAKER_02: So that's just some room there for sure.
• 32:08
  SPEAKER_02: Like the uh validity of the source, right?
• 32:12
  SPEAKER_02: Authenticity, all those sorts of things is another uh piece.
• 32:16
  SPEAKER_04: So what keeps you?
• 32:17
  SPEAKER_04: I'm sorry, go ahead, Chris.
• 32:18
  SPEAKER_05: You're good, yeah, you're good, you're good.
• 32:20
  SPEAKER_05: Go ahead now.
• 32:21
  SPEAKER_04: What keeps you up at night?
• 32:22
  SPEAKER_04: Are there any problems or you know, issues that and challenges that you're facing these days that uh are of major concern?
• 32:30
  SPEAKER_02: Yeah, I mean, some models are uh, you know, I think is the bias or and there's adversarial uh you know attacks that we we keep on hearing, you know, there's there's things that put certain models at risk.
• 32:42
  SPEAKER_02: So that does keep, I think, all of us a little bit awake where it might hurt us more, right?
• 32:47
  SPEAKER_02: Because this uh this uh these responses, right, that the machine's generating are are you know causing more harm than good.
• 32:56
  SPEAKER_02: So that of course that's that does keep everyone awake.
• 32:59
  SPEAKER_02: And there's fortunately a lot of research being done to detect those sorts of things and mitigate those things, right?
• 33:06
  SPEAKER_02: And I I mean uh again, guardrails are things we all uh try to offer to put in place to eliminate some of that as much as possible.
• 33:14
  SPEAKER_02: So that that would be that'd be kind of probably the big one that keeps yeah.
• 33:19
  SPEAKER_04: And and can you talk about a recent problem that you've solved with um your tools and technologies?
• 33:25
  SPEAKER_04: You know, obviously not mentioning a client name, but what's what's a nice class of an uh example problem that uh you've recently solved using any combination, whether it's straight-up search or yeah.
• 33:37
  SPEAKER_02: Yeah, um I think travel and hospitality seems like to be an interesting, you know, um uh market.
• 33:44
  SPEAKER_02: Uh you know, we've been working with a number of those folks in that space um around um people basically the way they're thinking about travel.
• 33:53
  SPEAKER_02: You want to go on a on a vacation, and you've got you know what you can kind of define like your family, something that's kid friendly or pet friendly.
• 34:02
  SPEAKER_02: Uh, these are the sorts of activities that you you like.
• 34:06
  SPEAKER_02: You know, you have certain requirements, right?
• 34:07
  SPEAKER_02: Has to have Wi-Fi or uh, you know, has to have an open bar, you know, whatever you may be, you know, and that's again a ton of rich data that's uh you know, more of a dialogue as if you were speaking to a travel agent, right?
• 34:21
  SPEAKER_02: So kind of like a virtual travel agent would be uh uh kind of that example, right?
• 34:26
  SPEAKER_02: Where um it does a great job of then you know suggesting um options for you.
• 34:32
  SPEAKER_02: So that's that's one.
• 34:34
  SPEAKER_02: Um as you saw in the conference, that virtual Sami A again for like food and beverage is very interesting, right?
• 34:42
  SPEAKER_02: Because it draws on that information.
• 34:44
  SPEAKER_02: Um, and it could be used for a number of use cases, right?
• 34:47
  SPEAKER_02: Um, you know, nutrition, dietitian, uh, those sorts of things.
• 34:52
  SPEAKER_02: Uh and then on the back end use cases that uh uh when we think about back end, that's not necessarily customer-facing, is really uh the workplace or employee productivity.
• 35:01
  SPEAKER_02: If you think about banking finances or healthcare, where you can have a co-pilot who uh is summarizing, you know, millions of documents for you, making them much easier to navigate through or discover.
• 35:12
  SPEAKER_02: We call it like information discovery, um, and then and then kind of interrogate uh rather than you having to read you know massive, uh, massive documents.
• 35:22
  SPEAKER_02: You think about like clinical trials or corporate filings, you know, analyst reports, who wants to read through like some 30, 40 page documents.
• 35:31
  SPEAKER_02: You just have certain questions you need answered.
• 35:33
  SPEAKER_02: So exactly.
• 35:35
  SPEAKER_05: Yeah, that's I'd love to know too if you've if you guys have experienced any ethical concerns or anything that keeps you like, uh guys, we maybe want to think about this, you know, because you alluded to some, but I would love to know like maybe any specifics or what you're thinking.
• 35:51
  SPEAKER_02: Yeah, ethics does come up quite a bit, um, you know, and and um when we think about like the the the the tone, especially when it's conversational, um, is it recognizing, you know, the the person's sort of um uh I wouldn't say take making sure it's responding where it's not responding in an offensive way, but also when we think about things, it's not using their information without their knowledge, too.
• 36:17
  SPEAKER_02: Like uh, you know, especially if uh some customer data is being used, for example, from their history to inform the model, like those sorts of things um are ethical concerns that need to come up.
• 36:29
  SPEAKER_02: Um, you know, again, the sentiment or tone of the end of the user as well, making sure you you almost route to human, um, where you're not forcing someone down a rat hole with the machine, especially when it gets to be complex or a heated, like emotional type of dialogue.
• 36:46
  SPEAKER_02: So those sorts of things, again, that's kind of where we think about guardrails or or workflows or orchestration of these sorts of things, right?
• 36:54
  SPEAKER_02: Um, I think is is uh the big ones for ethics.
• 36:58
  SPEAKER_04: I wanted to switch gears a little bit and ask you about Sanjay the person.
• 37:03
  SPEAKER_04: So, how did you kind of get to where you are and give us a little rundown of your background and you know what what you did and who were you uh as a young young man, uh always interested in technology, for example, or or uh uh what what was your uh genesis in this area?
• 37:20
  SPEAKER_02: Yeah, definitely always, you know, uh for not to totally date myself, but uh you know, back when uh you know the PCs were out there, I you know came up and and had those types of uh uh uh you know machines available to me to play with, and uh, you know, whether it's uh you know Atari or Commodores or you know, then apples and those sorts of things.
• 37:41
  SPEAKER_02: I think yeah, having exposure definitely uh you know helped me, especially kind of the gaming when you're young and trying to program games.
• 37:48
  SPEAKER_03: Where where are you from?
• 37:48
  SPEAKER_02: Which from I grew up in the Chicago area.
• 37:51
  SPEAKER_03: Okay, great.
• 37:52
  SPEAKER_02: Yeah, so uh that definitely uh you know the good academic uh background over there with uh getting us access to equipment and things like that at a young age, right?
• 38:01
  SPEAKER_02: And so that uh you know, gaming and kind of programming kind of was something I was getting very interested in at a young age.
• 38:09
  SPEAKER_02: Um, and then yeah, I went, you know, going through just academia with like data science and computer science and psychology and economics, it got you know, 40.
• 38:17
  SPEAKER_02: So a lot of convergence though, and things we're we're doing.
• 38:20
  SPEAKER_02: And then yeah, I moved into basically I saw the internet, you know, late, late, late 90s was a big opportunity.
• 38:26
  SPEAKER_02: We had that dot boom, but but uh fortunately survived a lot of that and worked in in uh more internet applications um in media and personalization, and that kind of moved into e-commerce and then uh you know moved into all kinds of like cloud computing and uh AI search over the years.
• 38:46
  SPEAKER_02: So uh yeah, been been uh been around the block, you know, been acquired by Oracle uh three times uh with different institutions.
• 38:54
  SPEAKER_02: But most of my focus has been in yeah, e-commerce.
• 38:58
  SPEAKER_02: I think the e-commerce arena.
• 38:59
  SPEAKER_04: Did you sit in the same desk and the company names changed?
• 39:03
  SPEAKER_04: Yeah, that's happened to some people I know.
• 39:05
  SPEAKER_04: They're like same desk, 10 years, three, four companies.
• 39:08
  SPEAKER_02: Yeah, that that yeah, definitely with Net Suite.
• 39:11
  SPEAKER_02: So yeah, that's been my my uh and I definitely build a passion for uh personalization and kind of the the art of the possible with AI.
• 39:21
  SPEAKER_04: And and if you could go back uh to after you graduated college, what what would you uh what advice would you give yourself?
• 39:29
  SPEAKER_02: You know, um I think taking bigger risks, maybe um I think would have been um, you know, especially at a younger age, I think you get a little comfortable.
• 39:41
  SPEAKER_02: Um, and um I think that they could have been taking you know some bigger risks at that point.
• 39:46
  SPEAKER_02: Um that that would be like mine, don't be afraid, I guess, uh to take bigger risks, be my idea.
• 39:52
  SPEAKER_04: Right, right.
• 39:52
  SPEAKER_04: That's great.
• 39:53
  SPEAKER_04: So where can people find you?
• 39:55
  SPEAKER_04: I know we'll put some stuff in the show notes, but what's the best way to get you?
• 39:59
  SPEAKER_02: Yeah, I'm on uh LinkedIn, uh Sanjay00 Mata is my handle.
• 40:04
  SPEAKER_02: Um that's a great way.
• 40:05
  SPEAKER_02: You know, Twitter, I'm mostly active, uh definitely on LinkedIn.
• 40:08
  SPEAKER_02: I have a lot of papers and webinars that I do, and of course, this will show up there as well.
• 40:13
  SPEAKER_02: Uh, you know, and then through Lucid Works, again, I'm headed industry there.
• 40:17
  SPEAKER_05: So uh yeah, some hey Sanjay, real quick too.
• 40:21
  SPEAKER_05: Um, because there's always some really, you know, like there's some unbelievable depth and interesting, you know, uh aspects to the folks we have on this.
• 40:28
  SPEAKER_05: And I know you live in Southern California, and you know, having you know been from there myself, you know, the best thing that I that we had growing up was the neighbor's avocado tree.
• 40:39
  SPEAKER_05: And I love the fact that it would grow straight over and it was like, hey, listen, you know, I mean, you know, possession is nine tenths of the law.
• 40:46
  SPEAKER_05: So uh we would just you know take it.
• 40:48
  SPEAKER_05: But you know, when we were speaking before, I'd you know, love to get, you know, let everybody know the stuff you do outside of all this, you know, boring work stuff.
• 40:57
  SPEAKER_05: What does it be for what do you do?
• 40:59
  SPEAKER_02: What do you do for uh yeah, yeah?
• 41:01
  SPEAKER_02: No, uh thanks.
• 41:02
  SPEAKER_02: That's fine.
• 41:02
  SPEAKER_02: Yeah, I mean, uh my my I have a passion for music and uh you know I had a music scholarship in school.
• 41:08
  SPEAKER_02: I play piano and drums, and um, I still play I play professionally do some tours.
• 41:12
  SPEAKER_02: Um, luckily this type of year allows you to do that.
• 41:15
  SPEAKER_02: You're on the road and you can kind of play around the world.
• 41:17
  SPEAKER_04: Uh do you what kind of gigs do you do?
• 41:21
  SPEAKER_02: Uh most when I when I was uh going through school, I was doing a lot more jazz and blues.
• 41:26
  SPEAKER_02: Um, and then you know, I think a lot of that stayed here in Los Angeles.
• 41:29
  SPEAKER_02: We have a kind of a good like classic rock, jazz and blues.
• 41:32
  SPEAKER_02: Really, but I consider kind of everything's game for me, you know, kind of the fusion piece.
• 41:38
  SPEAKER_02: So that's been fun.
• 41:39
  SPEAKER_02: And then uh yeah, I have sort of a big uh hobby in farming, and so I do uh keep uh uh an avocado grove, uh, you know, in Southern California and then a citrus grove uh in in Riverside.
• 41:53
  SPEAKER_02: So that keeps me busy, keeps my uh yeah, body in shape.
• 41:57
  SPEAKER_05: Yeah, that's awesome.
• 41:58
  SPEAKER_05: Yeah, when we when we were talking before and I heard that, I was like, oh man.
• 42:02
  SPEAKER_05: You know, uh two of the two of the best things, you know, in terms of citrus and avocados that you know.
• 42:07
  SPEAKER_05: Oh yeah, oh yeah, it's awesome.
• 42:10
  SPEAKER_05: Awesome.
• 42:11
  SPEAKER_04: Well, Sanjay, it's been wonderful to have you.
• 42:13
  SPEAKER_04: Thank you so much for your time and sharing your knowledge and your expertise and your experience.
• 42:18
  SPEAKER_04: It's really been a pleasure.
• 42:19
  SPEAKER_02: Cool.
• 42:19
  SPEAKER_02: Thank you again for having me.
• 42:21
  SPEAKER_02: Enjoyed it.
• 42:22
  SPEAKER_02: Yeah, it's great.
• 42:23
  SPEAKER_04: So um thanks.
• 42:25
  SPEAKER_04: Uh thanks to our audience for for listening, and uh, I hope you learned something today.
• 42:30
  SPEAKER_04: And again, Sanjay, uh Chris, thanks again for helping out here and uh being on the show.
• 42:36
  SPEAKER_01: Thank you for joining us on another deep dive into AI innovation.
• 42:40
  SPEAKER_01: Tune in next time when we introduce another industry expert and discuss how to maximize AI in the world.
• 42:46
  SPEAKER_01: The early AI podcast is sponsored by Early Information Science and CMSY.
• 42:51
  SPEAKER_01: To learn more, visit early.com.com.