Latest / The Adults In The AI Room / Ian Hook on Advancing Operational Excellence with AI and Knowledge Management - The Earley AI Podcast with Seth Earley - Episode #041

Transcript

Ian Hook on Advancing Operational Excellence with AI and Knowledge Management - The Earley AI Podcast with Seth Earley - Episode #041 · The Adults In The AI Room

• 0:00
  SPEAKER_05: This is the Early AI Podcast, hosted by Seth Early and Chris Fetterstone.
• 0:05
  SPEAKER_05: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_05: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as CMS Wire.
• 0:24
  SPEAKER_05: Now, enjoy the show.
• 0:36
  SPEAKER_03: I'm sorry?
• 0:37
  SPEAKER_01: I said new year, my friend.
• 0:39
  SPEAKER_01: This is crazy.
• 0:40
  SPEAKER_00: Absolutely.
• 0:41
  SPEAKER_00: Happy New Year.
• 0:42
  SPEAKER_00: Yeah, I hope you had a good holiday.
• 0:43
  SPEAKER_00: It's uh should be an interesting 2024.
• 0:46
  SPEAKER_01: Yeah.
• 0:48
  SPEAKER_01: Number one podcast of the of 2024.
• 0:54
  SPEAKER_01: Not number one podcast of 2024, which I hope it is, but the first podcast.
• 0:59
  SPEAKER_00: There you go.
• 1:00
  SPEAKER_00: Our first podcast.
• 1:01
  SPEAKER_00: That's right.
• 1:02
  SPEAKER_00: That's right.
• 1:03
  SPEAKER_00: So I'm really excited to introduce our guest, our uh guest today.
• 1:06
  SPEAKER_00: We're going to be discussing knowledge management applications and the importance of knowledge management for artificial intelligence.
• 1:14
  SPEAKER_00: Uh, we're going to be talking about the benefits uh and drawbacks of centralization versus decentralization around things like knowledge processes and artificial intelligence.
• 1:24
  SPEAKER_00: We'll talk about solving issues with generative AI, such as managing hallucinations and using uh corporate data to uh answer questions rather than the knowledge within the LLM.
• 1:37
  SPEAKER_00: And so our guest today is an AI and knowledge banks professional with experience leading multi-million dollar projects.
• 1:43
  SPEAKER_00: He's had patents in NLP recommendation engines and more.
• 1:48
  SPEAKER_00: Uh, he has experience leading multiple Gen AI projects and transforming uh what enterprise level companies are capable of.
• 1:55
  SPEAKER_00: Today's guest is also a keynote speaker at major events, including Digital Health Conference, Cam World, Mobile Monday, Mobile World Congress, and more.
• 2:03
  SPEAKER_00: Currently, he's the director of knowledge management insights and excellence at Novartis.
• 2:08
  SPEAKER_00: Ian Hook, welcome to the show.
• 2:10
  SPEAKER_02: Hi, everybody.
• 2:11
  SPEAKER_02: Thank you for having me.
• 2:13
  SPEAKER_00: So I'd like to start off with your perspective on common misconceptions about knowledge management and AI.
• 2:21
  SPEAKER_00: Uh, first of all, give us uh your your definition of knowledge management.
• 2:25
  SPEAKER_00: I know people kind of have lots of different ways of interpreting that.
• 2:28
  SPEAKER_00: And then um we'll talk a little bit about um technology, the role of technology and the role of uh supporting processes.
• 2:36
  SPEAKER_00: But why don't you just give your your perspective on knowledge management in the enterprise?
• 2:41
  SPEAKER_02: Yeah, so I think uh a really kind of interesting, provocative kind of in uh impulse and perspective on knowledge management is the way that I approach it, is really it's about kind of understanding what your business is and what it does, and and uh how we can better facilitate it in doing that, really driving it towards value.
• 3:03
  SPEAKER_02: And so when I talk about knowledge management in in the business, I talk about three pillars, right?
• 3:08
  SPEAKER_02: Understanding your content, understanding your people, and understanding your business.
• 3:13
  SPEAKER_02: And if you can understand those three things in relation to each other, I think you're doing very good knowledge management.
• 3:20
  SPEAKER_02: And I think until you understand those things, I think you are very limited in the knowledge management that you're able to do inside an organization.
• 3:27
  SPEAKER_02: And that's really where my interest in AI stuff comes in because AI is really good at helping us understand those stuff.
• 3:34
  SPEAKER_00: Yeah, so when you talk about understanding the business, you know, I think a lot of uh there's a lot of uh mistakes that are made around uh you know checking the box and procuring technology, whether it's knowledge management technology or AI technology or a combination of both.
• 3:52
  SPEAKER_00: Uh, do you want to talk a little bit about how organizations should be approaching the technology issue?
• 3:57
  SPEAKER_00: Because again, it's it's the backbone and it's the uh it's the uh mechanism by which we can capture and share knowledge.
• 4:05
  SPEAKER_00: Uh, but uh but what are the mistakes that people are making when they're looking at technology?
• 4:11
  SPEAKER_02: You're absolutely right.
• 4:12
  SPEAKER_02: Uh and I think you what you said there right is right.
• 4:16
  SPEAKER_02: It's the backbone, it's the structure around which we can build things.
• 4:20
  SPEAKER_02: But I think what we, uh especially in knowledge management, need to better understand is what is the value in which we're driving towards, and how do we need to leverage things like technology?
• 4:31
  SPEAKER_02: And I think technology is a part of the picture, but how do we need to leverage that towards driving more value?
• 4:37
  SPEAKER_02: And I think we don't typically spend enough time understanding what value means, right?
• 4:44
  SPEAKER_02: And what our part of that value picture is and can be and should be, right?
• 4:50
  SPEAKER_02: And and even when it gets down to actually putting a number figure on that, a lot of people talk about knowledge management.
• 4:55
  SPEAKER_02: Oh, it's this thing that we know that we should do, and you know, it's kind of like exercise, right?
• 5:00
  SPEAKER_02: I I I know that I should exercise and I'll get to it at some point, but I think we really need to get better at putting a number and a figure of of what value it is that we're driving in an organization.
• 5:10
  SPEAKER_02: And when we start to do that, um the conversations open up and people really start to understand, oh, yeah, this is tangible, right?
• 5:17
  SPEAKER_02: This is something that we need to do.
• 5:18
  SPEAKER_02: And now technology, we can bring technology in to facilitate that business value, right?
• 5:24
  SPEAKER_02: And that's where you get the right technology applied in the right way, and and that application of it in the right way is really critically important.
• 5:32
  SPEAKER_02: Um, and that's where you get funding and you know, all that kind of stuff kind of comes after.
• 5:36
  SPEAKER_02: But I think it's really about understanding the business value first when you do the job.
• 5:40
  SPEAKER_00: Right.
• 5:40
  SPEAKER_00: So starting off with, well, what are you trying to accomplish?
• 5:43
  SPEAKER_00: Right?
• 5:44
  SPEAKER_00: What's your business objective?
• 5:45
  SPEAKER_00: And then, you know, I always look at it, and people have heard me talk about this.
• 5:49
  SPEAKER_00: You know, you have the enterprise strategy at the top, and then you have kind of data, knowledge, content, AI initiatives, algorithm, search at the bottom.
• 5:58
  SPEAKER_00: And then what's in between that?
• 6:00
  SPEAKER_00: If a leader says, How are we going to get uh revenue out of this?
• 6:04
  SPEAKER_00: Well, if you're working at that in the weeds, it's hard to show that, but you can show how the content and the data supports a specific uh business process, right?
• 6:13
  SPEAKER_00: And you can you can measure the baselines and then measure what the what the improvement will be, and that process supports the business outcome, or multiple processes do, and then multiple business outcomes support the strategy.
• 6:25
  SPEAKER_00: So it's really doing that deconstruction and having that detailed knowledge of what those processes are.
• 6:31
  SPEAKER_00: I think too often you get businesses that are checking the box and saying, oh, we need this.
• 6:36
  SPEAKER_00: We need but you it just as you say, yeah, it's good, but we can't put our finger on what it should, uh how good it is or or how we justify it.
• 6:44
  SPEAKER_00: And I think that's a really critical piece.
• 6:46
  SPEAKER_00: How once you go ahead.
• 7:00
  SPEAKER_01: It brings into the abilities of being able to know and understand better what's there to make better insightful decisions.
• 7:06
  SPEAKER_01: So, therefore, not only can you have a quantitative measure on let's say the efficiencies gained because ease of access and ease of information and more relevant um information, but it's also qualitative because then it makes you know all those folks' jobs easier.
• 7:22
  SPEAKER_01: And that's that's really hard to measure, right?
• 7:24
  SPEAKER_01: The qualitative side of that.
• 7:26
  SPEAKER_01: It's just you know, ease of access.
• 7:28
  SPEAKER_01: Because putting a number on it, you know, and I'd love to get your take on on how you put an actual value number on that thing to drive it.
• 7:35
  SPEAKER_01: Because to me, it literally, you know, falls into the form of, you know, if it's for a customer-facing thing, then that's customer lifetime value.
• 7:44
  SPEAKER_01: If it's an operational thing, then that falls into an efficiency gain in some way, which it is, it is definitely, you know, if I give you a knowledge management and I can quantity, you know, quantitatively measure, hey, I'm gonna 500 people are gonna have this much more access and this quick, this much quicker.
• 8:00
  SPEAKER_01: So therefore, it's a quantitative measure.
• 8:02
  SPEAKER_01: And then I'd love to get your take on how you provide that value driver back to the organization for them to say business to say, oh yeah, that makes a hell of a lot of sense.
• 8:11
  SPEAKER_01: Let's do it, as opposed to, you know, listen, just throw up a transactional, you know, SQL environment and that they have no clear concept or anything about that, structured versus unstructured.
• 8:22
  SPEAKER_01: You know what I mean?
• 8:22
  SPEAKER_01: Yeah, so I'd love to absolutely go ahead.
• 8:25
  SPEAKER_01: Sorry.
• 8:26
  SPEAKER_02: There's two ways which I've done it uh at at the moment in in uh in my company.
• 8:31
  SPEAKER_02: Um, the first way, very, very tangibly, we picked knowledge, which is very, very expensive to the organization, and we're getting this, we're paying um, you know, uh companies outside of of Nevada's to give us this knowledge.
• 8:45
  SPEAKER_02: Um each piece of knowledge is about$200,000.
• 8:49
  SPEAKER_00: You're talking about paid subscriptions.
• 8:51
  SPEAKER_02: Uh no, so this is uh market research that we pay companies to give for us, right?
• 8:57
  SPEAKER_02: Gotcha.
• 8:57
  SPEAKER_02: Uh and so each piece of market research is about$200,000, right?
• 9:02
  SPEAKER_02: We said we currently have no way to know what market research we have, which means every time somebody goes and creates a piece of market research, maybe they're doing some duplication, right?
• 9:12
  SPEAKER_02: And we sat down with procurement, we understood, well, how much are we spending, what are we spending on, what does it look like, and we figured out, well, we can probably reduce our spend on primary market research stuff by about 30%, right?
• 9:26
  SPEAKER_02: And that 30% is about$18 million.
• 9:30
  SPEAKER_02: We said, okay, well, let's do it then, right?
• 9:32
  SPEAKER_02: So we reduced the spend and we found, yeah, if we put everything together in one place, make it easily accessible, use the right technology behind it, blah, blah, blah, blah, blah.
• 9:40
  SPEAKER_02: We can actually reduce that spend without having an impact on the product that we can deliver, right?
• 9:45
  SPEAKER_02: Because we've got it all centralized and we tell people, you know, we changed the process and made people go and look at it first, and all of this kind of stuff.
• 9:52
  SPEAKER_02: And yeah, we can put a very clear number on it, and we can say it works, right?
• 9:58
  SPEAKER_02: And then we can say because because we started there, which is a very easy, tangible use case, we can then explore, well, okay, what if we did that in other parts of the organization as well?
• 10:06
  SPEAKER_02: What else might we be able to reduce?
• 10:09
  SPEAKER_02: Or um, so that was one way, and then the other way, um so my company goes is trying to become what they call a launch machine, right?
• 10:19
  SPEAKER_02: Which is uh we launch drugs, and every time we launch a drug in pharma in general, it costs more, it takes longer, um, and we reach fewer patients every single time.
• 10:30
  SPEAKER_02: So they want to become a launch machine, they want to you know have this standardized process.
• 10:34
  SPEAKER_02: And I said, okay, guys, that's a nice idea, but if we don't understand our knowledge, how do you want to become a launch machine?
• 10:41
  SPEAKER_02: And I gave them this kind of scenario.
• 10:44
  SPEAKER_02: I work on a drug today, right?
• 10:46
  SPEAKER_02: And then I finish my project on that drug and I'm gonna go work on a different drug tomorrow.
• 10:50
  SPEAKER_02: Let's say I take half of the lessons learned that I could have, and I share half of the lessons learned that I took.
• 10:58
  SPEAKER_02: And of those lessons learned that I shared, well, you know, we we put half of those actually into practice.
• 11:03
  SPEAKER_02: And of those that end up in practice, well, half of those have that down, down, down, and it ended up, you know, is pretty anecdotally, about three percent of the value gets transferred from launch A back into launch B.
• 11:16
  SPEAKER_02: And I sat down with the business and hey, okay, business, do you actually take half of the lessons learned that you could?
• 11:22
  SPEAKER_02: And they said, Well, no, I mean, maybe we take like five, ten percent.
• 11:26
  SPEAKER_02: Yeah, right.
• 11:28
  SPEAKER_02: And if you know, I'm saying half and it ends up at three percent.
• 11:31
  SPEAKER_02: What if you take five, ten, five, ten percent here, five, ten percent there?
• 11:35
  SPEAKER_02: What is it actually, you know, like minds blown?
• 11:38
  SPEAKER_01: You know, so plus I think you have that notion too of the value chain, right?
• 11:44
  SPEAKER_01: The chain of custody and the value chain through the rest of the organization, because then you look at it from the perspective, I think this is where people miss it too, is within a knowledge graph is what's with you know, how do you quantity quantitate the um reusability of said data?
• 11:58
  SPEAKER_01: Whether it's a clean room perspective or whether it's internal data, you have that chain, that chain of custody, but you also have that chain of value of this is how many times this data's been used.
• 12:08
  SPEAKER_01: And most people have no clue or concept that a knowledge graph or a knowledge management system can give you those abilities and also give you versioning and you know, all of that kind of stuff, because you you know, you get into all these aspects.
• 12:19
  SPEAKER_01: So, yeah, that's awesome.
• 12:20
  SPEAKER_00: Absolutely.
• 12:21
  SPEAKER_00: Ian, are you using knowledge graphs in the organization?
• 12:25
  SPEAKER_02: Uh we so yes, in places, uh, not as much as I would like to, and not as much as I hopefully will in 2024, but yeah, we are.
• 12:35
  SPEAKER_00: How how are you defining knowledge graphs just for uh the audience?
• 12:40
  SPEAKER_02: Um, so the way that we are currently kind of using it, we we've got all of our knowledge in a central repository, it's tagged, and then we're using knowledge graphs to really deeply understand what the knowledge is, where it sits, who is in relation to that knowledge, right?
• 12:58
  SPEAKER_02: So um we're using that to start to surface things like expertise around knowledge.
• 13:02
  SPEAKER_02: If you've surfaced uh sorry, if you've submitted a bunch of content related to topic X, then you probably know something about topic X.
• 13:09
  SPEAKER_02: So those kinds of use cases, which I think are just touching the surface of what's possible.
• 13:15
  SPEAKER_02: Um, but yeah, I think I think there's a lot to do.
• 13:19
  SPEAKER_01: You know, I should I should back up.
• 13:20
  SPEAKER_01: Sorry, Seth.
• 13:21
  SPEAKER_01: My my apologies.
• 13:22
  SPEAKER_01: What I should, you know, for everybody on the that's listening to this, I I must I shouldn't use knowledge, uh knowledge graph and data catalog interchangeably because I you know I I love the focus you know the focus of and I do I'll get into and I'll conflate those two because I'm like, well, yeah, of course you should knowledge graph for your catalog, blah, blah, blah, blah.
• 13:41
  SPEAKER_01: I shouldn't do that because part of that too, you know, within a knowledge management system is the catalog of data that's there that's accessible, and that can be managed by a bunch of different systems.
• 13:51
  SPEAKER_01: So, anyway, so my my apologies to everybody listening, and then also just a clarification, you know, in terms of economies.
• 13:57
  SPEAKER_00: Yeah, and and I look at knowledge graphs.
• 14:00
  SPEAKER_00: So, you know, when we have all the different organizing principles, all the different metadata structures, the taxonomies, the ways you're classifying information, the ways you're looking at, say, mechanisms of action or diseases, indications, um, you know, symptoms, whatever those things may be all those different vocabularies, um uh, those are uh taxonomies, then there's relationships between them.
• 14:26
  SPEAKER_00: So there could be a mechanism of action for a particular compound.
• 14:30
  SPEAKER_00: Uh, and when you start putting all those things together, that that's the ontology.
• 14:34
  SPEAKER_00: That's all the taxonomies and the relationships between them mapped together.
• 14:39
  SPEAKER_00: And then when you add the data to that, that becomes a knowledge graph.
• 14:42
  SPEAKER_00: So it really is the knowledge scaffolding of the organization connected to the data.
• 14:47
  SPEAKER_00: It's a wonderful concept uh in in theory, and it's a difficult one to put into practice, but um, but uh organizations are being able to do that more effectively.
• 14:58
  SPEAKER_00: But it does help you have multiple access points for your data and to be able to navigate from one uh repository to another, just like you would on IMDB.
• 15:09
  SPEAKER_00: You look at you know directors of movies, and then who are the actors in those movies, and then what other movies are those actors in?
• 15:15
  SPEAKER_00: That's a knowledge graph, right?
• 15:17
  SPEAKER_00: That's what the way of explaining that is.
• 15:20
  SPEAKER_00: So when you think about um one of the things you mentioned was, you know, it's hard to get the information into a knowledge management system.
• 15:28
  SPEAKER_00: And uh, and and of course, it's motivating people to do that, to capture the stuff in the context of their process.
• 15:35
  SPEAKER_00: Um, but um, there's also you'd also mentioned the intent behind the knowledge, right?
• 15:40
  SPEAKER_00: Because knowledge is not something that is, you know, just it it requires context, right?
• 15:48
  SPEAKER_00: So, do you want to talk a little bit more about context and intent in the knowledge management?
• 15:53
  SPEAKER_02: Absolutely.
• 15:54
  SPEAKER_02: And I I think this is really the crossover between human and machine, right?
• 15:59
  SPEAKER_02: Which is for me the fascinating part that we're at, we're at right now and and will continue to get more interesting.
• 16:06
  SPEAKER_02: Uh we still rely uh in in in my company a lot on humans to understand intent of knowledge, right?
• 16:15
  SPEAKER_02: Um, because so so kind of very, very kind of dumb, let's put it that way.
• 16:21
  SPEAKER_02: Um a machine is very good if you tell it, well, hey, this piece of knowledge is gonna go out of date in you know December, great.
• 16:29
  SPEAKER_02: It's very good at being able to tell you, oh yeah, December is out of date.
• 16:33
  SPEAKER_02: It's not so fantastic.
• 16:34
  SPEAKER_02: If you've got two documents that are pretty much the same, you know, somebody's made a couple of tweaks over there.
• 16:39
  SPEAKER_02: Well, which one's the better document?
• 16:41
  SPEAKER_02: Which one's more valuable to it's not so great at that.
• 16:44
  SPEAKER_02: So we're still relying a lot more on humans to kind of understand the value of our knowledge, and I think that is really, really important.
• 16:53
  SPEAKER_02: Uh, we spent a lot of time and effort and political will in um in shaping the part of the organization which will do that for us here.
• 17:04
  SPEAKER_02: Um they they see the value in it, they know the value in it, and and that's that's great.
• 17:09
  SPEAKER_02: So we've managed to be able to do that.
• 17:10
  SPEAKER_02: I know a lot of people have a lot of challenges being able to get kind of management on board with saying like, yeah, we're gonna commit people and time and effort to be able to do that.
• 17:19
  SPEAKER_02: So yeah, I I think it's the intent of knowledge is critically important.
• 17:25
  SPEAKER_02: Um, and it comes I for me, I think, almost equally important as things like tagging and and and kind of understanding what knowledge is is there.
• 17:37
  SPEAKER_01: What about um I only assume though that you know obviously you you you put the humans in the right?
• 17:42
  SPEAKER_01: So you have human in the loop, and that's gonna provide things, and I'd love to get your ratios on what you that the you know the best you know balance is you got human in the loop that you're doing validation, right?
• 17:56
  SPEAKER_01: Doing validation, and then also then how much does that play into compliance for you?
• 18:03
  SPEAKER_01: Because I know you guys have huge compliance standards.
• 18:05
  SPEAKER_01: So where at what point is the balance between you're using the humans to to you know validate or you know, and or inadvertently, you know, validate what's going on in terms of the decisions and/or that's there, but also you have this you know whole compliance standard too.
• 18:22
  SPEAKER_01: And where do you put the balance in value or and in emphasis and investment?
• 18:26
  SPEAKER_01: I guess I guess it's more investment, not value, right?
• 18:29
  SPEAKER_02: Yeah, yeah, absolutely.
• 18:31
  SPEAKER_02: So a couple of things to unpack that.
• 18:33
  SPEAKER_02: The first thing is uh at the moment, I don't focus too much on external facing content, I focus much more on internal, and that gets me around the loop of you know a lot of the compliance stuff, right?
• 18:47
  SPEAKER_02: Um it's it there's also a lot of I mean, just beyond the compliance, there's a lot of complexity on external facing knowledge for a farmer organization, which is it's just a bit of a headache.
• 18:59
  SPEAKER_02: And uh interestingly, the legal perspective on that is to use uh AI less and use humans more because they think that humans are more trustworthy.
• 19:14
  SPEAKER_02: I don't I I don't know, right?
• 19:15
  SPEAKER_02: But I'm just that's that's kind of the perspective at the moment, fine.
• 19:19
  SPEAKER_01: Um it goes back to seven intent, right?
• 19:22
  SPEAKER_01: It goes back so you know intent, like human intent versus machine intent.
• 19:26
  SPEAKER_01: Once this arrived anyway, keep going.
• 19:30
  SPEAKER_02: Um and so when it comes back to like how are we kind of managing effort almost, right?
• 19:36
  SPEAKER_02: So as knowledge comes into our repository, yeah, it's it's automatically tagged, and then we have a review process to sit over that automatic tag and say, Well, did it get it right?
• 19:45
  SPEAKER_02: Right.
• 19:46
  SPEAKER_02: And then when it sits in the system, we have processes which say, Well, hey, that knowledge has been there for let's say a few months, nobody's looked at it, nobody's in it, seems nobody's interested in it.
• 19:58
  SPEAKER_02: We can go back to the business and say, hey, business owner of this knowledge.
• 20:02
  SPEAKER_02: Is this because it's out of date?
• 20:05
  SPEAKER_02: Is it because nobody cares?
• 20:08
  SPEAKER_02: Is it because you know, and and then we can say, well, hey, it's it is out of date, we need to replace it with this new thing, or hey, nobody cares about this thing.
• 20:17
  SPEAKER_02: Should it actually be on the platform or should we remove it?
• 20:19
  SPEAKER_02: And that for me is a critical thing, which I in my experience, knowledge management struggles with is actually the management of knowledge.
• 20:32
  SPEAKER_02: I think knowledge management is really, really good at kind of collecting it all, and we're really good at you know sticking it all together in one place, but I think uh not enough places are managing, intentionally managing the knowledge that they've got.
• 20:45
  SPEAKER_02: And that becomes even more important if you want to put AI over it.
• 20:48
  SPEAKER_00: Okay, yes.
• 20:49
  SPEAKER_00: So that was one of the things that I wanted to get to because at the end of the day, if you have poorly curated knowledge that is not tagged, that no one owns, that's not being reviewed, uh, what is AI going to do for that?
• 21:03
  SPEAKER_00: You're using AI in and you know in auto-classification and ingestion of the content and the knowledge.
• 21:10
  SPEAKER_00: But then if it's not managed with a life cycle, tell me what AI, what will AI do uh with these uncurated sources or this dark data that everybody, all the vendors are saying, oh, Gen AI is gonna fix this for you, you get all this value.
• 21:26
  SPEAKER_00: Tell me about that.
• 21:27
  SPEAKER_00: Dispel that that bit.
• 21:29
  SPEAKER_02: It it it's gonna cause a lot, a lot of headaches for a lot of organizations.
• 21:34
  SPEAKER_02: And I and I see it here um even uh even here.
• 21:39
  SPEAKER_02: So the there's a few different parts that I talk about, especially when we talk about generative AI.
• 21:44
  SPEAKER_02: Like, what do you need to do?
• 21:45
  SPEAKER_02: If you want to go and do generative AI in an organization, what do you actually need to do?
• 21:49
  SPEAKER_02: Well, firstly, you need to understand the knowledge you've got, and and if you're not doing that, you're in trouble, right?
• 21:56
  SPEAKER_02: Because it's not going to give you the right answers when it goes and finds something, right?
• 21:59
  SPEAKER_02: It's gonna Give you the the royal answer is going to hallucinate and and all this kind of stuff.
• 22:04
  SPEAKER_02: But then it also becomes in things around scalability and sustainability, right?
• 22:09
  SPEAKER_02: So um one of the things which which fascinates me and frustrates me in equal measure is very, very easy to stand up uh a Gen AI prototype, right?
• 22:21
  SPEAKER_02: I can go to somebody in the business and I can get 10 pieces of knowledge, I can stick them all together into one place, I can put you know open AI over the top of it, and it and it looks wonderful.
• 22:29
  SPEAKER_02: And I can take that to my leadership, I can say, hey, leadership, look at this one, and they'll give me a million dollars to go away and build it, right?
• 22:35
  SPEAKER_02: Where does the next hundred pieces of content come from and the next thousand pieces of content come from, right?
• 22:40
  SPEAKER_02: Because it's not going to be from me going around and asking a bunch of people, hey, can you give me those 10 documents?
• 22:45
  SPEAKER_02: Right?
• 22:45
  SPEAKER_02: You need to have a proper structure, proper infrastructure in place to be able to do that.
• 22:49
  SPEAKER_02: And then when you've got them there, what happens when those old documents change?
• 22:53
  SPEAKER_02: What do you do then?
• 22:54
  SPEAKER_02: Right?
• 22:55
  SPEAKER_02: Well, you know, that's a knowledge management problem, right?
• 22:57
  SPEAKER_02: That's the management of the knowledge because the Gen AI doesn't know that the answer it's just given you is from six months out of date and it's now non-compliant anymore.
• 23:07
  SPEAKER_02: It just doesn't know that kind of stuff, right?
• 23:09
  SPEAKER_02: It doesn't care, right?
• 23:10
  SPEAKER_02: There's a fascinating quote that somebody said, Um, which was that uh and it might have been you set, which is that uh Gen AI just all it does is hallucinate, right?
• 23:22
  SPEAKER_02: It's just it's us whether we judge those hallucinations to be true or not true, right?
• 23:28
  SPEAKER_02: But but it doesn't know.
• 23:30
  SPEAKER_02: It's there's there's no brain behind it.
• 23:32
  SPEAKER_02: There's nobody, you know, it's there's no sort of truth, right?
• 23:35
  SPEAKER_00: And if we if we're doing it, if we're using the large language, the way I look at it is you can use a large language model in two ways.
• 23:42
  SPEAKER_00: One, you can seek the answers from the uh terms and relationships and knowledge embedded in the large language model, right?
• 23:51
  SPEAKER_00: Because it has concept relationships and it's actually doing a very sophisticated mechanism of predicting what an answer should be.
• 23:58
  SPEAKER_00: And you're right, you know, it's we're the ones who judge whether it's right or not, because it's simply statistically, it's a stochastic parrot, right?
• 24:05
  SPEAKER_00: It's simply repeating things that it thinks you want to hear or should hear in terms of what's most likely to come in these answers.
• 24:13
  SPEAKER_00: Now, it's still incredibly powerful, and I've used it for uh my creative purposes when I was writing a survey, for example, that's going to be going out.
• 24:22
  SPEAKER_00: Send you a copy of it in um and uh and it helped me rewrite it just for clarity, right?
• 24:29
  SPEAKER_00: I provided, I didn't say write me a survey.
• 24:32
  SPEAKER_00: I said, here's my survey.
• 24:34
  SPEAKER_00: What do you recommend?
• 24:35
  SPEAKER_00: First, it gave me a bunch of recommendations, and then I said, okay, can you rewrite, can you edit the survey to improve it?
• 24:42
  SPEAKER_00: It gave me a version.
• 24:43
  SPEAKER_00: Then I said, can you rewrite the survey to improve it?
• 24:46
  SPEAKER_00: So there's all these stepwise ways of doing this, and then I used my judgment.
• 24:50
  SPEAKER_00: Now, that's one way to do that, asking the LLM for an answer or giving it something to respond to.
• 24:58
  SPEAKER_00: And of course, that's uh part of the prompts.
• 25:00
  SPEAKER_00: But that LLM does not have your knowledge and expertise to the organization and is likely going to make up stuff that is not necessarily accurate, right?
• 25:10
  SPEAKER_00: If we like the results, uh it's it's uh it's reasonable.
• 25:14
  SPEAKER_00: If we don't like the results, then it's a hallucination, right?
• 25:17
  SPEAKER_00: Or if it's actually incorrect.
• 25:19
  SPEAKER_00: But now it comes back to how do we reduce those hallucinations and how do we make sure it has the right ground truth?
• 25:26
  SPEAKER_00: And that's where the retrieval augmented generation, which says take the LLM and use it to process the query and then uh run that query against your database and then use the LLM again to process the results.
• 25:41
  SPEAKER_00: But it's that in-between part, right?
• 25:44
  SPEAKER_00: Data source.
• 25:46
  SPEAKER_00: And so when you look at organizations trying to do this, even when we've done really great POCs and proof of value projects, you know, there's a lot more nuance once you increase the volume of the content because you need additional context, right?
• 26:02
  SPEAKER_00: You can't ask simplistic questions of a very large corpus.
• 26:06
  SPEAKER_00: So tell me about your thinking there.
• 26:08
  SPEAKER_00: You know, what are the biggest misconceptions that people have about Gen AI?
• 26:12
  SPEAKER_00: And do they understand, you know, that distinction uh that I that I made?
• 26:18
  SPEAKER_02: Yeah, yeah.
• 26:19
  SPEAKER_02: And and right now they don't understand the distinction.
• 26:22
  SPEAKER_02: And I'll give you my favorite story.
• 26:24
  SPEAKER_02: Uh I won't name them, but a consulting company reached out to me uh in, I think it was June, July last year, and they said, hey Ian, we we've got a fantastic way to um replace all of the MSLs that you have in Novato's medical scientific liaisons, these are people that sit down with doctors and give them answers about our drugs.
• 26:44
  SPEAKER_02: I said, Okay, well, I don't think it's gonna work.
• 26:47
  SPEAKER_02: I went into this and I said, I don't think it's gonna work.
• 26:49
  SPEAKER_02: And they said, Oh no, no, don't worry about it, let's let's go and try it out.
• 26:52
  SPEAKER_02: So they trained it on one of our drugs, um, all of the available content on the internet, and I asked it the question, okay, tell me about the efficacy of this drug on patients age 20 to 40.
• 27:03
  SPEAKER_02: And it gave me a perfect answer, and they were all really happy with themselves.
• 27:06
  SPEAKER_02: It's okay, tell me about the efficacy of this drug on pregnant women.
• 27:10
  SPEAKER_02: And it gave me another perfect answer, and they were like super happy, they thought it sold me.
• 27:14
  SPEAKER_02: And I said, Okay, tell me about the efficacy of this drug on pregnant pineapples.
• 27:19
  SPEAKER_02: And it gave me a perfect answer for pregnant women.
• 27:22
  SPEAKER_02: I said, Okay, now tell me about the efficacy of this drug for boys aged 15 to 20, and it gave me a perfect answer for men aged 20 to 40.
• 27:32
  SPEAKER_02: I said, there's your problem.
• 27:34
  SPEAKER_02: It cannot replace an MSL, right?
• 27:36
  SPEAKER_02: Because in the pregnant pineapples, okay, nobody's gonna ask about pregnant pineapples, but somebody might ask about you know men aged 15 to 20.
• 27:43
  SPEAKER_02: And if you give them the wrong answer, it's gonna kill somebody, right?
• 27:46
  SPEAKER_01: Yeah, or or they may they you know have an accent, right?
• 27:50
  SPEAKER_01: And they don't get the ESR, right?
• 27:52
  SPEAKER_01: Or they don't type it in and they misspell it.
• 27:54
  SPEAKER_01: Yeah.
• 27:55
  SPEAKER_02: Right.
• 27:56
  SPEAKER_02: And so the thing for me is use case.
• 28:01
  SPEAKER_02: Are you applying this to the right use case?
• 28:03
  SPEAKER_02: And if you don't understand the the the model and you don't understand the use case, it's gonna fail.
• 28:10
  SPEAKER_02: But if you understand the model and you understand the use case really well, it can be fantastic.
• 28:14
  SPEAKER_02: Like exactly the use case that you talked about.
• 28:16
  SPEAKER_02: And there's use cases we're exploring right now, which are around uh how do we use this to create draft content, which then goes into somebody's hands who is an expert that can review it, right?
• 28:28
  SPEAKER_02: Fantastic use case, super powerful, will reduce our you know, our time by months in creating this kind of content.
• 28:37
  SPEAKER_02: You know, it's great.
• 28:39
  SPEAKER_02: But yeah.
• 28:41
  SPEAKER_01: I could see them, I could see, you know, uh, you know, what in in theory, you know, one of the things that could have come back was okay, you're right.
• 28:49
  SPEAKER_01: We can't replace you call them MSLs or MLSs, MLs.
• 28:53
  SPEAKER_01: However, we can give those folks a tool to be faster, more productive all day long, right?
• 28:58
  SPEAKER_01: And then give them like because you have the women in the loop and they know how to derive, because you you know, one of the things that we've seen across the industries that I work with and stuff is that you've got, as you know, we'll say we want to say an 80-20, but it's more like a 90-10 in terms of the top 10% are the real experts, and the 90% are just still learning, right?
• 29:18
  SPEAKER_01: In terms of service, you know, um, you know, service requests or you know, like your experts, whatever that is within an organization.
• 29:26
  SPEAKER_01: So you give them tools like these to make the experts even more effective and the learners more effective because you get them up to speed faster, right?
• 29:34
  SPEAKER_01: Because of these, the you know, still the human the loop.
• 29:36
  SPEAKER_01: And that's spawn into your use case comment.
• 29:39
  SPEAKER_02: You're absolutely right.
• 29:40
  SPEAKER_02: And this is where it gets to something I I really, really strongly believe.
• 29:45
  SPEAKER_02: As knowledge management professionals, we need to own the conversation on generative AI, right?
• 29:52
  SPEAKER_02: And so, in the way in which you know data scientists own the conversation around, let's call it traditional AI, because they own the data that comes into it and they understand that data, great.
• 30:05
  SPEAKER_02: Well, the data for generative AI is knowledge.
• 30:10
  SPEAKER_02: We own knowledge, and therefore we need to own the conversation around generative AI.
• 30:14
  SPEAKER_02: We need to own, you know, we need to understand it as professionals, and then we need to understand well, the use cases that we can apply it to, and we need to be, you know, the drivers of the conversation, not even a part of the conversation.
• 30:27
  SPEAKER_02: I think what typically happens is something cool comes in and IT take it over because it's an IT thing.
• 30:32
  SPEAKER_02: This is too dangerous, right?
• 30:34
  SPEAKER_02: We need to be this is our thing, this is our evolution.
• 30:38
  SPEAKER_01: I would add one thing to that comment, too, in terms of not only owning the generative AI conversations, but the outcomes.
• 30:45
  SPEAKER_01: What are the outcomes that you're trying to drive that map to the use cases that are defined well?
• 30:50
  SPEAKER_01: Because you're to your point, yeah.
• 30:52
  SPEAKER_01: We get into I love I love the fact that define irony, we're gonna put it in technologies and tools for people who are operationally focused.
• 30:59
  SPEAKER_01: So, guess what they figure out?
• 31:01
  SPEAKER_01: Tactical and operational measures and outcomes, which is bullshit because it doesn't map to the business needs of what needs to happen in terms of your core conversation.
• 31:09
  SPEAKER_00: Well, it can as long as you you have that linkage, right?
• 31:16
  SPEAKER_01: Yeah, yeah, because we definitely we do this a lot with operational teams centered around, hey, I want to look at the knock and I want to ask questions of the knock using a generative model to understand how to detect and remediate issues faster from a network operations perspective and pull that information.
• 31:33
  SPEAKER_01: That's an outcome, but that's still driving to an operational focused, you know, thing that I can then do something downstream with, right?
• 31:41
  SPEAKER_01: But it again, it goes back to Ian's comment around the knowledge management of well, what were the trouble tickets that were seeing?
• 31:47
  SPEAKER_01: What network information do we miss?
• 31:50
  SPEAKER_01: How do we actually do remediation paths and plans for recommendations?
• 31:53
  SPEAKER_01: All that stuff is in the knowledge management path that's not in an operationally focused path.
• 31:57
  SPEAKER_01: So yeah, I'm I'm I violently agree with you.
• 32:00
  SPEAKER_02: You know, absolutely go ahead, go ahead, Ian.
• 32:03
  SPEAKER_02: Well, absolutely.
• 32:05
  SPEAKER_02: And and like I say, I think if you've done any kind of data science work, you you understand the level in which data scientists have to understand the data and you know what it's coming from and where it's going.
• 32:16
  SPEAKER_02: That's the whole pathway which we need to own now.
• 32:19
  SPEAKER_02: We need to step up and own it because nobody else can, right?
• 32:22
  SPEAKER_02: Right.
• 32:23
  SPEAKER_00: Right.
• 32:24
  SPEAKER_00: I I see a lot of the data science organizations asking, well, what's your algorithm?
• 32:30
  SPEAKER_00: You know, what's the model you're using?
• 32:32
  SPEAKER_00: And you know, and tell me about how you're handling this with data.
• 32:35
  SPEAKER_00: They don't want to believe that knowledge curation is part of this.
• 32:39
  SPEAKER_00: You know, they don't want to believe that you know tagging is something that has to happen or that there has to be a reference uh architecture.
• 32:46
  SPEAKER_00: And that I think is really, you know, it's perpetuate, it's a fallacy that's perpetuated by many of the vendors and a lot of the press that's out there.
• 32:54
  SPEAKER_00: You know, and I've I've talked before in this podcast about talking to vendors who said, oh yeah, you don't need, you know, taxonomies, metadata, reference architect, you don't need any of that stuff.
• 33:04
  SPEAKER_00: And I'd say, really, what do you do with the data?
• 33:07
  SPEAKER_00: And they'd say, Oh, well, you do have to do some data labeling.
• 33:09
  SPEAKER_00: It's like, duh, data labeling is metadata.
• 33:12
  SPEAKER_00: Metadata requires a reference architecture.
• 33:14
  SPEAKER_00: So of course you're using those concepts, but to to stand there and tell me you're not and it's not important, uh, belies the the fundamental lack of understanding, you know, and the biggest piece of this has to do with operationalizing these things, putting in the right governance structures, and being able to uh uh deploy this throughout the organization, scale it and and make it um uh something that's in production rather than something that is, you know, a novelty or one-off, or as you say, you know, looks really good in the demo, but then when you start to scale it, it becomes much more difficult.
• 33:50
  SPEAKER_00: Tell me a little bit about what you think the governance uh pieces should be.
• 33:54
  SPEAKER_00: What are some of the elements uh to both generative AI and and knowledge management and data management?
• 34:00
  SPEAKER_00: How all those things kind of fit together from your perspective?
• 34:03
  SPEAKER_02: Yeah, so I think the uh and and the the other part I want to kind of put into this is the generative AI is a tool on a as part of a bigger picture, right?
• 34:16
  SPEAKER_02: And I think that's that's another part that knowledge management needs to, I think knowledge management needs to be more ambitious than it than it has been that this tool is is there.
• 34:24
  SPEAKER_02: The governance as a part of it absolutely it's critical.
• 34:28
  SPEAKER_02: It is fundamental, right?
• 34:29
  SPEAKER_02: And if you've ever played around with generative AI and it's giving you a weird kind of an answer, the re I would say 90% of the time it's giving you a weird answer is is lack of governance over the data which it was leveraging, right?
• 34:42
  SPEAKER_02: So it's critical.
• 34:44
  SPEAKER_02: I think we can't underestimate what needs to be done there.
• 34:47
  SPEAKER_02: I yes, there are AI tools which can help on that journey to to proper uh knowledge governance.
• 34:54
  SPEAKER_02: You know, knowledge graphs you talked about, one of them automated tagging, you know, these kinds of things can help, but I think still at the moment it it ultimately comes down to a human which understands what's going on here and is able to curate a system and an architecture which can build this stuff on top of it.
• 35:12
  SPEAKER_02: Um, and so what one of the things I I mentioned, I I think, like I say, there is a a high-level ambition as to what I think knowledge management is able to do within an organization with the tools that we have access to right now.
• 35:27
  SPEAKER_02: Generative AI is a is another tool which helps us along that journey.
• 35:31
  SPEAKER_02: It's an efficiency tool which helps us along that journey, but it's governance, which is I think much more than governance for generative AI, it's governance for the organization in terms of you know what we should be doing for a company.
• 35:45
  SPEAKER_01: Well, we get a lot of a lot of organizations, and I'm sure you've seen this too, where we've got these phenomenal, you know, classic AI or discrete generative AI models that are data generators, much driven from a specific outcome and a specific you know, designed um you know, accurate score, right?
• 36:07
  SPEAKER_01: And that our data science teams that become data generators for in the context of now using a generative model to understand that kind of stuff, right?
• 36:15
  SPEAKER_01: However, I see a ton of is people automatically a generative model to say, I'm gonna fine-tune this thing, I'm gonna fine tune my stuff, I'm gonna fine-tune it, which okay, it may make sense.
• 36:27
  SPEAKER_01: However, maybe a technique like maybe better prompting, right?
• 36:31
  SPEAKER_01: Again, I think a more of a knowledge management effort now, as opposed to what it was before, where we get deep, it's a goddamn question the data, right?
• 36:41
  SPEAKER_01: So, how do we, you know, do that and or like you know, Seth, you brought up using a rag technique.
• 36:47
  SPEAKER_01: All three of those are techniques to get us to a more accurate response.
• 36:52
  SPEAKER_01: However, the data going in is the most important thing.
• 36:55
  SPEAKER_01: So we understand what's there.
• 36:57
  SPEAKER_01: But I don't know if you've seen folks that are like, oh, we're gonna go fine-tune them all, we're gonna go do this.
• 37:02
  SPEAKER_01: What does that look like in your world?
• 37:05
  SPEAKER_02: Mostly I have uh the people that think that they can do that are people that don't really understand it.
• 37:11
  SPEAKER_02: Yeah, would say, which I I mean, not to put anybody in you know one bucket, but but I think that's where it's at, right?
• 37:20
  SPEAKER_02: I I think uh, yeah, you're absolutely right.
• 37:23
  SPEAKER_02: Maybe it'll help, right?
• 37:24
  SPEAKER_02: But I can give you a list of 10 things that'll help a lot better, right?
• 37:28
  SPEAKER_02: Before you get there.
• 37:29
  SPEAKER_02: And you want to get there, fine, let's get there.
• 37:31
  SPEAKER_02: But it's that's not where you start anymore.
• 37:33
  SPEAKER_02: Maybe it maybe it was where you used to start, but it's not where you start now.
• 37:37
  SPEAKER_02: Um the the other thing which is interesting thing that I'm playing around with right now, uh, a lot of people will know this, right?
• 37:43
  SPEAKER_02: If you go to like Chat GPT and you say, Hey, give me uh a marketing plan for uh, you know, Coca-Cola, it'll give you, you know, a marketing plan.
• 37:52
  SPEAKER_02: But if you say, hey, give me a marketing plan for millennials that are doing blah, blah, blah, blah, blah, but you know, give it a lot of data, better prompting you talked about, it'll give you a better answer.
• 38:00
  SPEAKER_02: So one of the things that I'm looking at right now is can we use expert location systems to understand who the people are in the organization to then pre-prompt, right?
• 38:12
  SPEAKER_02: To say, well, hey, this is Chris, this is what Chris knows about, this is what Chris is working on, and what he's interested in, and what he's an expert on.
• 38:20
  SPEAKER_02: So when Chris comes and asks you a question about this, give him a different answer from when Ian comes and asks you that question, right?
• 38:26
  SPEAKER_02: Because Chris is interested in that and Ian's interested in this.
• 38:29
  SPEAKER_02: I think that's for me gonna be a really powerful tool in enabling this in enterprise, which is a different perspective from enabling it for consumers out, you know, what open AI is doing.
• 38:41
  SPEAKER_02: I think there's a going to be a really interesting kind of divide as to how we implement this in enterprise.
• 38:46
  SPEAKER_00: I think that um the training and the guidelines around prompt engineering are going to be very important and uh specific to job tasks and and again to use cases, right?
• 38:58
  SPEAKER_00: Because you know, I really do see two kind of divergent aspects of this.
• 39:03
  SPEAKER_00: One is using the LLM for what it can do in and of itself, and using the LLM to access corporate data.
• 39:09
  SPEAKER_00: And in both cases, you still need to prompt it uh appropriately uh in order to get a specific uh answer, or there needs to be a dialogue.
• 39:17
  SPEAKER_00: And you know, we we found that if we uh enrich the embeddings, when we bring knowledge into the large language model, we're using component authoring, component architecture, which gives you more semantically meaningful chunks to write a question-answering type of an application, and then also applying metadata to that content in order to enrich the embeddings.
• 39:41
  SPEAKER_00: And that has provided you know significantly better results.
• 39:44
  SPEAKER_00: Have you found uh anything similar?
• 39:47
  SPEAKER_00: What is the role of metadata in your knowledge uh applications?
• 39:51
  SPEAKER_00: And have you found an appreciable difference between having good metadata within with using an LLM and retrieval augmented generation versus not having metadata?
• 40:02
  SPEAKER_02: Yeah, uh I think so.
• 40:04
  SPEAKER_02: We've not tested it as extensively as as you have.
• 40:07
  SPEAKER_02: Uh we have tested it a bit and we have found, yeah, it it does make a uh a difference.
• 40:12
  SPEAKER_02: Yeah.
• 40:12
  SPEAKER_00: Um especially for additional signals for the vector, right?
• 40:18
  SPEAKER_00: Exactly.
• 40:18
  SPEAKER_02: Exactly.
• 40:19
  SPEAKER_02: And and also I think uh this for me is gonna come a lot down to use case.
• 40:25
  SPEAKER_02: So where we uh where we're looking at vast you know stores of knowledge, uh I think we are able to segment the the knowledge by use cases and say, well, actually, don't don't even look at 80% of what we've got, right?
• 40:41
  SPEAKER_02: Just ignore that because it's just not useful for you, it's gonna send you off down the road.
• 40:44
  SPEAKER_02: Just look at this 20% over here and and the answer's somewhere there, right?
• 40:48
  SPEAKER_00: Right.
• 40:49
  SPEAKER_00: And that's filtering.
• 40:50
  SPEAKER_00: That's filtering.
• 40:50
  SPEAKER_00: Yeah.
• 40:51
  SPEAKER_00: Right.
• 40:52
  SPEAKER_00: Done with metadata.
• 40:53
  SPEAKER_00: Even if you're saying the source of this answer or the repository you should look at, that is metadata, right?
• 40:59
  SPEAKER_00: That's source, that's saying you look over here, don't look over there.
• 41:04
  SPEAKER_00: That is, in effect, directing the LLM using an attribute of that knowledge source, right?
• 41:10
  SPEAKER_00: Which which is additional context.
• 41:12
  SPEAKER_02: Absolutely.
• 41:13
  SPEAKER_02: And and then the question becomes what are the use cases and what is the knowledge in relation to the use cases, which is what I was saying before, I understand my user, my content, and my business.
• 41:23
  SPEAKER_02: So, what's the business use case that we need to apply it to and how do we direct it towards the right answer?
• 41:28
  SPEAKER_00: Yep.
• 41:30
  SPEAKER_00: And um, can you talk about have you been able to measure some of your outcomes to show that you've uh appreciably uh improved the efficiency of uh of an operation?
• 41:40
  SPEAKER_00: Or where have your successes been?
• 41:42
  SPEAKER_02: Not yet.
• 41:42
  SPEAKER_02: Not yet.
• 41:43
  SPEAKER_02: Um, so we're working on that stuff.
• 41:46
  SPEAKER_02: We should be pulling it out in in kind of hopefully the first quarter of this year.
• 41:50
  SPEAKER_02: Um but I I from what we've seen in the MVPs that we've built, it's massively uh impactful.
• 41:57
  SPEAKER_02: Uh, I think I've got a list of about 250 use cases that we can apply it to.
• 42:02
  SPEAKER_02: A lot of those are efficiency gains.
• 42:03
  SPEAKER_02: So there's a lot more to go at.
• 42:05
  SPEAKER_00: Yeah, and I believe building libraries of use cases and having a gold standard of answers to those questions because they have to be unambiguous.
• 42:16
  SPEAKER_00: Then you can compare different models, you compare the efficiency, the effectiveness, the recall, the precision across those different variables by knowing what that gold standard use case is and that gold standard answer is.
• 42:29
  SPEAKER_00: Even if the LLM produces a variation in that answer, you can still either have human judgment or you can actually use an LLM to compare two answers to see if they're a right.
• 42:40
  SPEAKER_01: Absolutely.
• 42:41
  SPEAKER_01: Absolutely.
• 42:41
  SPEAKER_01: That's one thing I was gonna ask too, is what that looks like in terms of you know going into it.
• 42:47
  SPEAKER_01: Because I think a lot of people miss what it looks like in terms of, you know, like we talked about the people miss and say just fine-tune a model or or create your own model or whatever, missing the cost optimization going into it, understanding what it looks like.
• 43:00
  SPEAKER_01: Because to your point, Seth, if you're gonna look at, you know, this model versus this model, one may be open source, one may be proprietary, there's cost you know, a cost um discrepancy between those two.
• 43:12
  SPEAKER_01: And it doesn't need to break the bank in order to get some of these things.
• 43:16
  SPEAKER_01: Because to your point, a lot of people are going in, eyes wide closed, testing and doing putting prototypes up and everything else, but have no idea that's gonna be, you know, an impacting to the bottom line as to what it's gonna cost in order to run those things, the inference, the training, you know, all of that kind of stuff.
• 43:34
  SPEAKER_01: And so I'm getting into now a lot of those discussions centered around okay, pull your head out of the clouds.
• 43:41
  SPEAKER_01: Now it's a matter of can we actually, you know, really cut, you know, justify the cost of actually running something like this.
• 43:46
  SPEAKER_01: Is it going to be the gains that we needed and designed for?
• 43:50
  SPEAKER_01: And we did we design up front?
• 43:51
  SPEAKER_01: So I'd love your take on that.
• 43:53
  SPEAKER_02: So I I am a really, really big proponent of really, really dirty.
• 44:00
  SPEAKER_02: Prototypes quick and dirty.
• 44:02
  SPEAKER_02: Get it done.
• 44:03
  SPEAKER_02: Right.
• 44:03
  SPEAKER_02: So one of the things that we did here, I've got a fantastic data scientist that I work with.
• 44:07
  SPEAKER_02: And what she did, she built me an environment where I've got um open AI, right?
• 44:13
  SPEAKER_02: So it's uh running Chat GPT 4.
• 44:17
  SPEAKER_02: Um and I can basically plug and play knowledge into that environment.
• 44:22
  SPEAKER_02: It's in its own sandbox, right?
• 44:24
  SPEAKER_02: And so what I can do then is I can take a hundred documents that I've got from the business, I can plug them in here, right?
• 44:31
  SPEAKER_02: I can run just the the you know the the straight model over the top of it, and I can I can get a feel for what's the output gonna look like, you know, is it gonna be valid?
• 44:40
  SPEAKER_02: Is it gonna meet my my use case?
• 44:42
  SPEAKER_02: Right, and I can do that in about half an hour, right?
• 44:45
  SPEAKER_02: I can run the whole thing in half an hour, and then the next half an hour run a completely different use case, completely different outcome, you know.
• 44:53
  SPEAKER_02: That cost me about$2,000, right?
• 44:57
  SPEAKER_02: And it has managed to de-risk literally hundreds of projects that that people were trying to, you know, oh, I've got an idea to do this.
• 45:05
  SPEAKER_02: Like, and normally what you would do is you would go and spin up a whole team and you would get thousands of, you know, tens of thousands of dollars of investment and blah blah blah.
• 45:12
  SPEAKER_02: No, no, no, we don't need to do that right now, right?
• 45:14
  SPEAKER_02: We just put it together, play around with it, I give you an output, we tell you directionally, is it worth doing or not?
• 45:21
  SPEAKER_02: And that the other part to that, which is really important to remember if anybody's doing this, is uh the legal and compliance aspect of it, right?
• 45:30
  SPEAKER_02: So legal and compliance are, in my experience, freaking out right now, right?
• 45:37
  SPEAKER_02: Because they there's this new tool which they don't really understand.
• 45:41
  SPEAKER_02: You know, a lot of people have got access to and they've been playing around with it, especially inside the enterprise.
• 45:45
  SPEAKER_02: And what do we do?
• 45:46
  SPEAKER_02: How do we kind of get a grip on this?
• 45:48
  SPEAKER_02: And so if you can give them an environment and invite them into the conversation, hey guys, let's build some stuff together.
• 45:54
  SPEAKER_02: This will help you educate yourself around what's going on here as well as us as we build it, and help you give us an understanding of is this something you would ever want to scale across the company before we've ever invested, you know,$100,000 into it.
• 46:09
  SPEAKER_02: They are super happy, right?
• 46:11
  SPEAKER_02: And that's, I think, also critical to remember is the reason why when I started this and why a lot of organizations are getting a lot of put back pushback from legal and compliance is because they're as scared as as you know, as anybody in this conversation.
• 46:26
  SPEAKER_02: We need to bring them along, right?
• 46:29
  SPEAKER_01: Define irony that most organizations now have most of their critical data sitting in a cloud-based environment.
• 46:36
  SPEAKER_01: So what in the hell is the difference between your most critical data sitting in a hyperscalers cloud and you're freaking out about what a large language model can and can't do?
• 46:47
  SPEAKER_01: Good lord.
• 46:48
  SPEAKER_01: Like, you know, like can we pull, can we please pull our heads out and and actually think about this like adults and say, okay, you know, I get it, the risk vector here.
• 46:57
  SPEAKER_01: However, it's we're you're past that risk vector.
• 47:01
  SPEAKER_01: Like you're you're already down the river, you know, through the hardest part of the rapids, and now you're worried about what's ahead?
• 47:08
  SPEAKER_01: Come on.
• 47:09
  SPEAKER_01: Yeah, you know what I mean?
• 47:10
  SPEAKER_00: Yeah, I mean, there are a lot of ramifications to to compliance.
• 47:14
  SPEAKER_00: Uh, we're right at the end of the uh our time here.
• 47:17
  SPEAKER_00: So, Ian, before we go, I wanted to uh just learn a little bit about you, how you got to where you are, and you know, what's your career path been?
• 47:25
  SPEAKER_00: And then what do you what do you do for uh fun?
• 47:27
  SPEAKER_00: What do you do for outside of work?
• 47:29
  SPEAKER_02: Yeah, I I am I like to say I'm the the least qualified person to be in this position.
• 47:34
  SPEAKER_02: And so I uh I have a degree in biblical studies uh and I was a preschool teacher and then went into startup and then I uh building apps for preschool kids.
• 47:44
  SPEAKER_02: Uh and then I ended up uh I ended up in Accenture doing some gamification stuff there, and that's where we ended up building a bunch of the AI stuff uh with a couple of colleagues there.
• 47:52
  SPEAKER_02: It was great fun, and so that's why I came here is basically to do similar kind of stuff uh over in Novartis with the the AI and the KM background, and it really was a very lucky point.
• 48:03
  SPEAKER_00: Yeah.
• 48:03
  SPEAKER_00: How did you get to Accenture from your background?
• 48:05
  SPEAKER_00: I I missed that part.
• 48:07
  SPEAKER_02: So I was building games for preschool kids.
• 48:10
  SPEAKER_02: Oh, and Accenture at the time had a gamification program, and um, and so I jumped over to to help them run the gamification program.
• 48:17
  SPEAKER_00: So preschool and consultants and client and customers are pretty much similar kind of uh pretty much the same.
• 48:26
  SPEAKER_00: Hey, hey, we still react to the same way and uh to to games and and uh pleasure from that.
• 48:33
  SPEAKER_00: And what do you do for fun?
• 48:34
  SPEAKER_00: What do you do in your spare time?
• 48:36
  SPEAKER_02: I uh so I just recently moved to to Basel in Switzerland.
• 48:40
  SPEAKER_02: Um, so I I had to sell all my stuff, but I was a a blacksmith and uh I used to make kitchen knives.
• 48:46
  SPEAKER_02: That was kind of what got me got me going.
• 48:48
  SPEAKER_02: So my my whole garage used to be a you know whole workshop with an anvil and a forge in there and stuff.
• 48:54
  SPEAKER_00: So where was that before you moved to Basel?
• 48:57
  SPEAKER_02: In Prague.
• 48:58
  SPEAKER_00: In Prague, okay.
• 48:59
  SPEAKER_00: Wow, that's quite an interesting hobby.
• 49:01
  SPEAKER_00: Yeah.
• 49:02
  SPEAKER_01: I got one question for you, Ian.
• 49:04
  SPEAKER_01: If um was there a question that we sh didn't ask you that we should have?
• 49:08
  SPEAKER_01: Was there something that you know is is like, you know, if you think about all the things we talked about, was there something that maybe that you know we missed or that we should have asked before we go?
• 49:19
  SPEAKER_02: Yeah, I think we covered it.
• 49:22
  SPEAKER_02: I think we covered a good chunk of it.
• 49:23
  SPEAKER_02: The the only other thing which which I'm doing right now, which I think is important, is education of leadership teams.
• 49:31
  SPEAKER_02: So I spend a lot of time uh talking to leadership teams actually before Christmas went away to to China to educate the leadership team over there over just what it is and you know how it works and how we should be a part of it.
• 49:44
  SPEAKER_02: And that's really I think part of this conversation of um owning the conversation around Chat GPT.
• 49:49
  SPEAKER_00: Absolutely, absolutely.
• 49:50
  SPEAKER_00: Well, Ian, it's been a pleasure having you on the show.
• 49:53
  SPEAKER_00: I really appreciate your time.
• 49:55
  SPEAKER_00: Um, thank you so much for for being with us.
• 49:59
  SPEAKER_02: Thank you very much.
• 50:00
  SPEAKER_02: I really appreciate it.
• 50:01
  SPEAKER_01: And if you uh if you're at MWC, you know, I I think I'll be there too.
• 50:05
  SPEAKER_01: Then we're definitely um uh you know get you a beer andor you know get you something so that we don't have any gallbladder tax, but I'll take you after you, right?
• 50:15
  SPEAKER_02: Yeah, my wife's put me on a diet now, so there you go.
• 50:19
  SPEAKER_00: That's been another episode of the Early AI Podcast.
• 50:23
  SPEAKER_00: Uh, thank you to our audience for listening, and again, thank you to uh Ian, of course, Chris, and then uh Liam and Carolyn behind the scenes.
• 50:31
  SPEAKER_00: So thanks everybody and we'll see you next time.
• 50:34
  SPEAKER_05: Thank you for joining us on another deep dive into AI innovation.
• 50:38
  SPEAKER_05: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 50:44
  SPEAKER_05: The Early AI podcast is sponsored by Early Information Science at CMSY.
• 50:50
  SPEAKER_05: To learn more, visit early.com.
• 50:52
  SPEAKER_05: That is E A R L E Y dot com.
• 50:57
  SPEAKER_05: Thanks for today.