Latest / The Adults In The AI Room / Enterprise A.I. Strategy, Knowledge Management and more - Rachad Najjar - The Earley AI Podcast with Seth Earley - Episode #037

Transcript

Enterprise A.I. Strategy, Knowledge Management and more - Rachad Najjar - The Earley AI Podcast with Seth Earley - Episode #037 · The Adults In The AI Room

• 0:00
  SPEAKER_00: This is the Early AI podcast hosted by Seth Early and Chris Edison.
• 0:05
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of publicers and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI podcast is sponsored by Early Information Science, the digital transformation journey with design and deployment of innovative technology solutions, as well as CMS wire.
• 0:24
  SPEAKER_00: Now, enjoy the show.
• 0:30
  SPEAKER_02: All right.
• 0:30
  SPEAKER_02: Well, welcome to the Early AI Podcast.
• 0:34
  SPEAKER_02: My name is Seth Early, and I'm really excited to introduce our guest today.
• 0:39
  SPEAKER_02: We're going to be talking about some of the common misconceptions about implementing artificial intelligence in the enterprise.
• 0:46
  SPEAKER_02: A lot of the increase in use and misuse of AI tools and technologies.
• 0:52
  SPEAKER_02: We're going to talk about how generative AI is accelerating and impacting knowledge use cases and especially retrieval augmented generation, which is my favorite topic.
• 1:05
  SPEAKER_02: And we're going to talk about ways to maximize knowledge sharing and learning opportunities.
• 1:10
  SPEAKER_02: Our guest has been at the forefront of innovation in the fields of organizational learning and knowledge management for nearly a decade.
• 1:18
  SPEAKER_02: Since 2013, he's been a driving force at GE Renewable Energy, where his role involves crafting and executing an integrated learning strategy that encompasses experiential, social, and formal learning.
• 1:31
  SPEAKER_02: His responsibilities extend to defining the enterprise knowledge architecture and fostering knowledge sharing communities.
• 1:38
  SPEAKER_02: He's also co-author of a recent book on knowledge management and research innovation alongside numerous scientific publications.
• 1:46
  SPEAKER_02: Please welcome uh Rishad Najar.
• 1:50
  SPEAKER_02: Welcome to the show.
• 1:52
  SPEAKER_01: Thank you.
• 1:53
  SPEAKER_01: My great pleasure to be on your show, to have this conversation, and I'm looking forward.
• 1:58
  SPEAKER_01: There's many, many topics that we can cover.
• 2:01
  SPEAKER_01: We'll try to be really uh convergent and toward the really the main messages.
• 2:08
  SPEAKER_02: Yeah, I looked over your work and it's pretty amazing.
• 2:10
  SPEAKER_02: Uh, I mean, the stuff that you shared with us is just fantastic.
• 2:13
  SPEAKER_02: So we'll try to dig into some of that.
• 2:15
  SPEAKER_02: What I'd like to start with are some of the misconceptions.
• 2:19
  SPEAKER_02: What are you seeing in terms of misconceptions around uh AI and um and its deployment?
• 2:26
  SPEAKER_02: What's uh going on it from your perspective?
• 2:28
  SPEAKER_01: Yes, there's um multiple misconceptions.
• 2:32
  SPEAKER_01: Maybe let's focus more on the integration strategy.
• 2:35
  SPEAKER_01: And this is where exactly uh I was curious to know to and I asked myself, okay, so now the AI has gained the mainstream attention, but and we see a lot of use cases, uh, some of them are text generation, text summarization, um, and text translation.
• 2:54
  SPEAKER_01: But what are those specific use cases uh that we can implement and how we can avoid some of those misconceptions uh and uh and for some uh reasons, those will lead us to uh to harmful results.
• 3:11
  SPEAKER_01: And there was uh recently a Cisco study that stated 40 percent of uh AI projects are abandoned because they couldn't really um uh deploy some impactful uh AI projects.
• 3:28
  SPEAKER_02: So back to your business, they didn't have bottom line business values, which you're saying.
• 3:32
  SPEAKER_02: They couldn't show perfectly, exactly.
• 3:35
  SPEAKER_01: So that's exactly uh back to your initial question, some of the misconceptions.
• 3:40
  SPEAKER_01: I'd like to start with uh AI or generative AI being a culture uh being a digital transformation.
• 3:47
  SPEAKER_01: So I hear many of times uh AI is a digital transformation and it's a new technology and we need to deploy it.
• 3:56
  SPEAKER_01: On the contrary, AI is a cultural transformation, and why this is really important to uh to understand, because when we say AI is a digital transformation, uh we are inherently um uh deploying or or uh communicating a workforce risk.
• 4:16
  SPEAKER_01: So employees, our employees will think AI will replace their job.
• 4:22
  SPEAKER_01: On contrary, AI, people who embrace AI, those will replace people who don't uh embrace AI.
• 4:32
  SPEAKER_03: Right.
• 4:32
  SPEAKER_01: Uh yes, and and by also uh considering AI as cultural transformation, which means AI is a skills booster, is a capacity uh amplifier.
• 4:45
  SPEAKER_01: So we should understand and focus on our people, uh understand those touch points and their workflows and their daily tasks, see where we can inject AI, where we can integrate AI so we can avoid uh the repetitive task, free them up their time, and um assign them some impactful task where they can add more value.
• 5:09
  SPEAKER_01: So it's extension, amplification of our human capital in terms of skills and capabilities.
• 5:16
  SPEAKER_02: I love your point about uh finding that intervention in the workflow, really understanding that process.
• 5:23
  SPEAKER_02: I like to say you can't automate what you don't understand, right?
• 5:27
  SPEAKER_02: And you can't automate a mess, right?
• 5:30
  SPEAKER_02: I guess you can automate a mess, but you just can't automate a mess.
• 5:33
  SPEAKER_02: But but the point that you're making is understand what that workflow process is, understanding that day-to-day uh activity, and try to remove the things that can be redundant or you know, um manual or uh repetitive, etc., and be able to have a specific intervention.
• 5:51
  SPEAKER_02: I also look for what I call information leverage points, where can you have a big impact downstream when you can either get information more quickly, get more accurate information, or get information you weren't able to get.
• 6:03
  SPEAKER_02: But carry on, keep keep uh keep going because this is a great point.
• 6:07
  SPEAKER_02: Your your point being it's not just it's not a not necessarily a digital transformation in terms of technology, it's a it's really a people transformation and a cultural transformation, cultural transformation, exactly.
• 6:20
  SPEAKER_01: And uh, you mentioned something very important we need to understand the mess or organize it, and this is one of the false expectations uh that we put on generative AI that we expect gen AI to organize our mess.
• 6:36
  SPEAKER_01: Right to not do it, it's our job as practitioners, as domain experts, as specialists, to do our homework uh at first place and then say, okay, how we can increase productivity, how we can uh increase our uh understanding of the situation, and then we can introduce the AI.
• 6:56
  SPEAKER_01: There's also some kind of uh operational shift and paradigm shift.
• 7:01
  SPEAKER_01: So we we need to look at our processes and say we are not going to deploy another incident management system, or we are not going to deploy another learning management system.
• 7:12
  SPEAKER_01: It is what we have already in a place and search for those processes.
• 7:18
  SPEAKER_01: For example, if we are deploying some kind of uh uh career development uh and we are looking specifically into some kind of insights and coaching, so here we should uh evaluate how AI or those capabilities can help us to offer more personalized insights and more tailored insights in the coaching activity instead of saying we we are going to deploy a new LMS system and we are going to revolutionize our operational processes.
• 7:52
  SPEAKER_02: No, that makes a lot of sense, and uh what we don't need is another system, right?
• 7:56
  SPEAKER_02: And I love when you know when people say all we need is one place where everyone can go for all of their information, and I and it's like saying the solution to application proliferation is another application, yes, right?
• 8:11
  SPEAKER_02: And unless one kill at least two or three or four of them, it's not gonna get you there, right?
• 8:16
  SPEAKER_02: Because everybody's looking for that.
• 8:17
  SPEAKER_02: But that's not what you need.
• 8:19
  SPEAKER_02: You need to optimize the process and optimize your existing systems or existing tools, your existing workflows, and integrate into the fabric of the organization, as you mentioned in some of your writing.
• 8:32
  SPEAKER_02: Those are great points.
• 8:34
  SPEAKER_01: And also uh what you have mentioned in your book, we need the information architecture before we need before AI.
• 8:43
  SPEAKER_01: So uh connecting or having this kind of ontology and this kind of information architecture where we understand our data schema, our data sources, how they are connected together, what is the context, and now we can say, okay, let's uh implement AI.
• 8:59
  SPEAKER_01: So, this is also one of uh the points that I have learned from your book.
• 9:03
  SPEAKER_02: Thank you.
• 9:04
  SPEAKER_02: Well, I'll tell you, you know, it's back in the day when I when I first wrote that article, There's no AI without IA.
• 9:11
  SPEAKER_02: Um, it there's a so much uh noise and nonsense, just like there is today about generative AI.
• 9:16
  SPEAKER_02: And you know, it was going to magically do all these things, and you don't need ontologies, you don't need taxonomies, you don't need metadata, which is the first furthest thing from the truth, right?
• 9:25
  SPEAKER_02: You really do need that core architecture.
• 9:28
  SPEAKER_02: You know, generative AI and other forms of machine learning can accelerate the uh expansion and evolution and upkeep of an ontology, but it can't build, it can give you building blocks, but it can't build it because machine generated taxonomies and ontologies all are very problematic, right?
• 9:46
  SPEAKER_02: They they look like the machine generated, you know, they don't have make common sense, uh they're not in common sense English, right?
• 9:52
  SPEAKER_02: Or or whatever the language is.
• 9:54
  SPEAKER_02: Uh, but yeah, so you so and that requires human judgment, at least to get the reference architecture down.
• 10:00
  SPEAKER_02: That's what we we designed recently for global knowledge architecture for an organization, and that prepares them for using generative AI.
• 10:08
  SPEAKER_02: Um, maybe you could talk a little bit.
• 10:10
  SPEAKER_02: You you did this amazing study comparing a hundred generative AI technologies, and it's like, wow, you know, and I looked at it and it's just fantastic.
• 10:21
  SPEAKER_02: Um, do you want to talk a little bit about um where generative AI is accelerating and impacting knowledge use cases, areas, and processes?
• 10:31
  SPEAKER_02: Do you want to talk a little bit more about that?
• 10:33
  SPEAKER_02: I was at a Forrester conference and one of the um speakers, and I'm I have to I have to I have to have him on the podcast.
• 10:40
  SPEAKER_02: Actually, I can't remember his name off the top of my head, but uh he said that generative AI retrieval augmented generation, right?
• 10:47
  SPEAKER_02: Which is where we're using our corporate knowledge base, yes, is going to have the biggest impact on the organization of any uh AI technology.
• 10:56
  SPEAKER_02: Do you want to talk about these knowledge use cases and how your research kind of uh informed some of those use cases and applications?
• 11:07
  SPEAKER_01: Yes, exactly.
• 11:08
  SPEAKER_01: So uh this research was really initiated from a basic question.
• 11:13
  SPEAKER_01: Uh and I was curious to better understand where practically and in real life Gen AI is adding value.
• 11:21
  SPEAKER_01: As I mentioned earlier, uh in the media, in the mainstream, and social networks, every single uh company is uh rushing to deploy AI and especially those on the floor forefront uh customer services like conversational search, ticket management system, and the chatbot or virtual assistance.
• 11:42
  SPEAKER_01: And then I asked myself, okay, so now uh I am a knowledge manager and organizational learning leader.
• 11:48
  SPEAKER_01: So, where exactly uh Gen AI will impact my work and where uh what challenges Gen AI will help me to overcome those challenges?
• 11:59
  SPEAKER_01: So, my reasoning was first I said, let me uh first lay down the knowledge lifecycle, starting from the discovery of knowledge, the co-creation, the exchange, until the organization, formalization, reuse, and then uh all those analytics and intelligence.
• 12:18
  SPEAKER_01: Once I have set the lifecycle of our knowledge, then I went into every single phase and I said what are the key key processes or the challenges where we face as knowledge managers.
• 12:30
  SPEAKER_01: So once I built this grid, evaluation grid, I said, okay, now let me go to the industry, to the vendors, to the developers, and see every tool in the knowledge management and learning field, how they are implementing uh those AI capabilities.
• 12:48
  SPEAKER_01: And to give some data and statistics, 45% of those tools are related directly to customer support services.
• 12:58
  SPEAKER_01: Uh, less are uh related to uh to the search, so around uh 11%, another 10% related to learning, and another 10% related to uh marketing, sales and marketing.
• 13:12
  SPEAKER_01: So, really uh, and that's where we can understand why why today there's uh there's really uh a division or two groups of companies.
• 13:22
  SPEAKER_01: Uh some companies are really banning uh AI because they are afraid of the consequences, they are not aware uh what uh risk can handle.
• 13:33
  SPEAKER_01: So they are sending emails to their employees telling them don't use ChatGPT, don't use uh this and that.
• 13:39
  SPEAKER_01: And another group of people uh of companies who are rushing to uh to implement uh AI.
• 13:45
  SPEAKER_01: Uh but specifically, what my interest was exactly uh those KM processes and learning processes where they can help.
• 13:53
  SPEAKER_01: I can give three examples.
• 13:54
  SPEAKER_01: Uh the first one is related to ideation.
• 13:57
  SPEAKER_01: Uh, knowledge management is about tacit uh knowledge, uh how we can externalize our tacit knowledge, how we can enable.
• 14:04
  SPEAKER_02: Tacit knowledge is the knowledge you walk around with in your head.
• 14:07
  SPEAKER_02: The example I used to use is uh when I when I uh had my office uh renovated, I had uh plasterers, and you can write down the step to plastering a wall, you know, in about five steps, right?
• 14:19
  SPEAKER_02: You you know, you fill a barrel with water, you put the plaster in, you stir it around, you put it on the motorboard, you smush it on the ceiling.
• 14:25
  SPEAKER_02: Takes five years to become a proficient plasterer, right?
• 14:28
  SPEAKER_02: So the explicit knowledge is the instructions, the the five steps, the tacit knowledge is the ability to do it, and we have more tacit knowledge and experience than we have uh the ability to make explicit.
• 14:40
  SPEAKER_02: I'm sorry for that diversion.
• 14:43
  SPEAKER_01: A great point because this is the challenge.
• 14:45
  SPEAKER_01: Today, all the knowledge that we are working on Gen AI is explicit, so it's already there, and the challenge is to go further, the explicit knowledge into the tacit, how we can uh externalize it and share it broadly.
• 14:58
  SPEAKER_01: So, one of the use cases uh or two uh use cases that I could found the first one is related to uh to ideation and creativity, how when uh we are we have some ideas in in the head, but they are not fully formed or fully baked ideas.
• 15:18
  SPEAKER_01: So here Gen AI uh can idea autocomplete the same way our uh when we send a message word autocomplete, it pop up.
• 15:28
  SPEAKER_01: Now Gen AI can do idea autocomplete and can expand that idea with more arguments and examples.
• 15:36
  SPEAKER_01: So it's helping us to do some kind of ideation and brainstorming activities.
• 15:41
  SPEAKER_01: Another use case related to tacit externalization, and I find it very impactful, is related to storytelling.
• 15:49
  SPEAKER_01: We normally, one of the techniques we record videos, uh, we interview experts and we ask questions uh with the objective to extract uh its uh tacit knowledge.
• 16:01
  SPEAKER_01: Uh, through uh through video capturing and through ingesting AI capabilities, those videos can be automatically translated, speech-to-speech translation to and to further languages, so make them available at large scale at or global scale in Portuguese, French, Spanish, and so on.
• 16:21
  SPEAKER_01: And then what is more important, uh we can search within the video.
• 16:27
  SPEAKER_01: So now a piece of segment, 30 seconds from that video, when I ask my questions, generally I can go and extract 30 seconds from that video and then give me the transcript and give me the uh video sections.
• 16:42
  SPEAKER_01: So now we are facilitating the discovery uh of those uh tacit or unstructured data into more operational data that can be searched and integrated into our corpus.
• 16:56
  SPEAKER_02: Right, right.
• 16:57
  SPEAKER_02: No, that's a great that's a great example.
• 16:59
  SPEAKER_02: And of course, uh, in terms of search and integration into our corpus, again, we still need that reference architecture.
• 17:06
  SPEAKER_02: Um, in in your writings, you talked a little bit about um the misuse of AI.
• 17:12
  SPEAKER_02: What what what do you consider the misuse of AI and talk more about that?
• 17:17
  SPEAKER_01: Yes, uh, to give some numbers before going into some examples, uh the number of misuse cases in 2019, there were around 123 uh incident or misuse cases.
• 17:30
  SPEAKER_01: And 2023 is the in the first half of 203, more than 1000, so 10 times more in just six months uh in 2023.
• 17:41
  SPEAKER_01: And some example uh of those misuse uh, and this is really uh critical and risky uh face facial analysis.
• 17:52
  SPEAKER_01: For example, some application that they do pseudo science, so they claim uh they can do some facial analysis to analyze the criminality tendency of some person using that the shape of the head, uh the skin tone, uh some kind of uh I don't know the shape of the head.
• 18:16
  SPEAKER_01: And what they do, they repackage their solutions with AI capabilities and some marketing stuff, and they try to convince the end user that through the power of AI, you can detect the criminality or you can detect some harmful behavior of your uh citizen or clients.
• 18:35
  SPEAKER_01: Uh and now they give it as a as a claim it for granted.
• 18:40
  SPEAKER_01: This is one of the very uh misuse uh cases uh that can be harmful, and we need really some kind of ethical AI to prevent this kind of applications.
• 18:52
  SPEAKER_02: I mean, I know there's a there's bias in training sets.
• 18:55
  SPEAKER_02: So when you're using it for things like employment screening, or you're using it for credit risk analysis, or even you know, projected recidivism in terms of uh of people being let out of um correctional facilities, uh, there's a lot of bias in those training data.
• 19:10
  SPEAKER_02: Can you give us other examples of the misuse?
• 19:12
  SPEAKER_02: I mean, that clearly was that's that's crazy in my mind.
• 19:16
  SPEAKER_02: Yes, for someone to say something like that.
• 19:18
  SPEAKER_02: That is pseudoscience backed up by more, you know, um uh term, you know, uh uh by by jargon and terminology and yes, uh yeah.
• 19:28
  SPEAKER_01: Uh and any other uh good example for that another example uh that I saw as it was really eye-opening, uh using deep fake.
• 19:38
  SPEAKER_01: So uh um Morgan Freeman was um impersonalized, so there was someone uh ex person who was who composed a speech and through deep fake techniques he he put the speech and then he sensed the lips uh talking, so not only the face and uh the image, but also lip syncing, and he uh he impersonated Morgan Freeman uh character and made him say something he didn't uh say, and that's very uh and the propagation effect is a very serious because you know how on social media things proliferate and things go crazy.
• 20:23
  SPEAKER_02: Wow, yeah, that's a great example.
• 20:25
  SPEAKER_02: It's like you can't trust what you see and hear, and and it's a very scary uh you know, uh uh perspective or premise.
• 20:35
  SPEAKER_02: Um, now you also had uh written uh a uh guide and you had done some work uh around guiding principles, seven guiding principles for successful enterprise AI strategy.
• 20:50
  SPEAKER_02: Do you want to talk a little bit about that framework and how you kind of arrived at that and uh maybe talk about some of those principles?
• 20:57
  SPEAKER_01: Yes, great.
• 20:58
  SPEAKER_01: Uh so there were seven principles during my uh evaluation for those hundred uh vendors.
• 21:05
  SPEAKER_01: I asked myself, okay, so now we know what to do, uh, and we have some kind of uh use cases and business case defined, but how to make it successful, how we can really uh go to the uh C level, to the top management, convince them uh with investment and being part of the transformation.
• 21:27
  SPEAKER_01: So I found myself um in a questioning what are the principles that will uh guide our strategy so we can successfully integrate AI.
• 21:39
  SPEAKER_01: And that's how I arrived to seven principles.
• 21:42
  SPEAKER_01: Uh the first one is really fundamental.
• 21:44
  SPEAKER_01: We need to have a strong business case.
• 21:46
  SPEAKER_02: Business case, yes, measurable business case, absolutely.
• 21:49
  SPEAKER_02: Measurable business case.
• 21:50
  SPEAKER_02: There's a lot of experimentation going on, which is okay as long as you know it's experimentation.
• 21:55
  SPEAKER_02: But if you're trying to deploy something, you really do need to understand that process.
• 22:00
  SPEAKER_02: Those baseline metrics.
• 22:01
  SPEAKER_02: Yeah.
• 22:01
  SPEAKER_02: Sorry, Karen.
• 22:03
  SPEAKER_01: Yes, exactly.
• 22:04
  SPEAKER_01: And that strategy should involve the end user, the customer, and that strategy.
• 22:10
  SPEAKER_01: So we want to make some impactful uh work to the customers.
• 22:14
  SPEAKER_01: On that point, specifically, this is where comes the ethical AI.
• 22:18
  SPEAKER_01: And this is one of the guiding principles to develop responsible AI.
• 22:23
  SPEAKER_01: And by responsible AI, this is also some of the misconceptions.
• 22:27
  SPEAKER_01: We transform uh a responsible AI or the ethical guidelines into a checklist.
• 22:33
  SPEAKER_01: And we say our application should be transparent, it should be beneficial, it should be, it should protect our data customer, and then we check the box.
• 22:42
  SPEAKER_01: No, it will not work like this because this ethical AI is a process, it's a case study.
• 22:49
  SPEAKER_01: Every single time we develop an application, we should go through the re-evaluation and through really assessing our principles for that specific application.
• 23:02
  SPEAKER_01: And that's where the customer should be involved not only in the deployment phase, but also in the development phase.
• 23:10
  SPEAKER_01: And we should be very transparent in how the machine is making decisions.
• 23:15
  SPEAKER_01: For example, if we are developing some image recognition for medical field, uh, we should not consider uh the doctor or the medicine medical staff as an end user and impose on them the application.
• 23:31
  SPEAKER_01: Instead, we should be transparent how the machine is making the decision, and then the final words the the doctor should have the final decision where this evaluation or this diagnosis is really ethical or not.
• 23:45
  SPEAKER_01: And we should also involve the customer telling him we are taking your personal data uh and embedding those personal data into our model.
• 23:55
  SPEAKER_01: Are you okay with that?
• 23:57
  SPEAKER_01: Do we have do we have our your approval?
• 23:59
  SPEAKER_01: This is very, very uh sensitive topic, and we should not take it uh at light.
• 24:06
  SPEAKER_02: Absolutely.
• 24:08
  SPEAKER_02: And um, and so you know, one of the things that you were talking about, and by the way, the article is seven principles, seven guiding principles of success of a successful enterprise AI strategy, and that's in Medium.
• 24:20
  SPEAKER_02: So folks can pull that up.
• 24:22
  SPEAKER_02: So you talk about business value, you talk about uh process integration, uh quality uh training, continuous supervision, um, power computing infrastructure, uh, and and uh uh AI and ML skills.
• 24:39
  SPEAKER_02: Are there any of those topics that you want to call out specifically?
• 24:43
  SPEAKER_02: Again, you know, when you start thinking about um, you know, trying to to you know, that quality training set, I mean, even when we think about retrieval augmented generation, we're using our own knowledge bases, right, for essentially fine-tuning um uh the uh the retrieval and fine-tuning the machine learning.
• 25:02
  SPEAKER_02: Do you want to talk what what which of those principles do you want to expand on?
• 25:06
  SPEAKER_02: And what do you think is most important beyond uh you know, after the uh business value and integration with process?
• 25:14
  SPEAKER_02: What else is is going to be really uh first and foremost?
• 25:17
  SPEAKER_01: Yes, we mentioned the ethical uh guideline.
• 25:21
  SPEAKER_01: Uh let us um uh mention the quality training set and couple it or associated with continuous supervision, those are interrelated.
• 25:29
  SPEAKER_01: Yeah, so uh the quality training set I said, the success of our AI models heavily depends on the quality of the training data.
• 25:38
  SPEAKER_01: So quality in, quality out.
• 25:40
  SPEAKER_01: And it's also we should not confuse that AI will improve the quality of my data, right?
• 25:46
  SPEAKER_01: Will not improve it.
• 25:48
  SPEAKER_01: It's what what we train the model, what we get.
• 25:51
  SPEAKER_01: This is as easy as this is a simple equation.
• 25:54
  SPEAKER_01: And we want to make sure also the training data set is diverse and representative and free from biases.
• 26:00
  SPEAKER_01: This is what you you mentioned uh already, and it could not lead to any uh discriminatory or unfair results.
• 26:07
  SPEAKER_01: So, for example, if we are in a financial uh institute, a banking industry, and uh we want to use AI for credit risk assessment to make some lending decisions, uh, we want to ensure fair and unbiased uh outcomes by collecting diverse data sets represented different demographics and financial backgrounds.
• 26:27
  SPEAKER_01: The data set should carefully curate and eliminate any potential biases such as gender or ethnicity, ethnicity.
• 26:36
  SPEAKER_01: We do not want to influence uh our credit risk assessment, and also the model should be trained on high uh quality data sets, and this is where the story doesn't end, because it's not enough to say we trained our model, we have our data, and now we'll the model will continue to work on its own because there's inherent uh uh issue with the neural networks.
• 27:05
  SPEAKER_01: Over time they they degrade in performance, and over time the error function, there's some kind of degradation, and we need to continuously uh retrain the model with the new data and the new representative data so we continue to get accurate and relevant input, and we want to avoid the inherited issue with the neural networks.
• 27:29
  SPEAKER_01: So it's not enough to say uh deploy it and leave it right continuously.
• 27:34
  SPEAKER_01: We should uh update and include the new use cases and new data to our model, right?
• 27:40
  SPEAKER_02: Right, absolutely.
• 27:41
  SPEAKER_02: So this kind of is uh leads uh me uh in my thinking to to the topic of governance, right?
• 27:49
  SPEAKER_02: We have machine learning uh operations, ML ops, right, to look at the development part of this, but but there's a bigger picture.
• 27:58
  SPEAKER_02: When you mentioned customer service being the largest application, well, customer service is many times owned by the call center, right?
• 28:05
  SPEAKER_02: Because it's when people are calling.
• 28:06
  SPEAKER_02: And then the vendors that they use are either outsourced providers, you know, business process outsourcers or you know, or uh or um uh call call center software uh vendors.
• 28:18
  SPEAKER_02: And many times they don't necessarily have the expertise to, you know, to uh model the knowledge and to fix the knowledge processes and to go upstream, right?
• 28:29
  SPEAKER_02: They're they're tasked with call deflection or handling calls or you know, maybe uh having a chat bot to answer a question.
• 28:36
  SPEAKER_02: But really, why are people calling the call center in the first place?
• 28:39
  SPEAKER_02: Because something is broken, right?
• 28:40
  SPEAKER_02: So you have to go upstream.
• 28:42
  SPEAKER_02: So you really have to look across not only the entire customer lifecycle, but you also have to look at the internal supporting processes for the people who are enabling that customer, right?
• 28:53
  SPEAKER_02: So I think when we look at customer service, it's a very narrow lens and it misses a lot of fundamental pieces.
• 29:01
  SPEAKER_02: So my sense uh of this is it really has to be, we really have to think about governance.
• 29:06
  SPEAKER_02: And that's what we're going to do our next webinar on, by the way, uh, in uh November 1st.
• 29:10
  SPEAKER_02: Uh, we're going to have a webinar, we're going to talk about um AI governance, right?
• 29:15
  SPEAKER_02: Uh machines, you know, and look at look at all of those factors, including what you mentioned in terms of business value.
• 29:21
  SPEAKER_02: But thinking about this, you know, from that customer support lens, again, and looking at the entrenched, you know, vendors, not saying they're doing a bad thing, but many times they're looking to other organizations to support that.
• 29:35
  SPEAKER_02: But really, they're not necessarily tasked or equipped for solving broader enterprise knowledge uh initiatives and organizational knowledge initiatives.
• 29:45
  SPEAKER_02: And so the question becomes you know, how do you structure governance and decision making?
• 29:50
  SPEAKER_02: I know governance can be a bad word in a lot of organizations, right?
• 29:54
  SPEAKER_02: Uh we're gonna have a meeting about governance.
• 29:56
  SPEAKER_02: Uh what works for you?
• 29:57
  SPEAKER_02: How about never?
• 29:57
  SPEAKER_02: Never works for me.
• 29:58
  SPEAKER_02: Does never work for you, right?
• 30:00
  SPEAKER_02: People, you know, so you have to couch it in other terms of, you know, a metrics uh-driven decision making or, you know, uh overarching, you know, business value, right?
• 30:11
  SPEAKER_02: But but we need to have representation from across departments and we need to have a mechanism where we have business ownership and not just a department.
• 30:20
  SPEAKER_02: I mean, it's so siloed, you know, when we would do product data projects and we uh ask about the call center or about uh customer support, they'd be like, oh yeah, we know nothing about that, right?
• 30:30
  SPEAKER_02: Well, what are you gonna do once you have perfect product data?
• 30:33
  SPEAKER_02: You're gonna have to look at the rest of the customer lifecycle, right?
• 30:35
  SPEAKER_02: So, what are your thoughts about how organizations need to govern and manage AI resource allocation, generative AI uh project decisions?
• 30:46
  SPEAKER_02: It seems to me that this is a wide open area with many, many needs.
• 30:51
  SPEAKER_02: And it's not just, you know, it's I think it can be centered around knowledge management and content management, uh, but but again, it's looking at those upstream internal processes for supporting that customer as well as the people who are on the front lines.
• 31:05
  SPEAKER_02: So, what has your research shown you in in those areas of overarching uh decision making, governance, resource allocation, process work, and so on?
• 31:16
  SPEAKER_01: Yes, exactly.
• 31:17
  SPEAKER_01: And this is also one of the let's say uh misconceptions that we create AI team, we form a team from uh machine learning skills or data scientists and some uh IT ops uh skills, and now we say this is the AI team that is responsible for our um uh our uh applications, customer service, for example.
• 31:41
  SPEAKER_01: Uh this is really uh unhealthy because AI, uh, as we stated earlier, uh it is diverse, and we should form a community including diverse profiles: legal, uh supply chain, uh, quality, project management, all of them having different perspectives on the same topic and the same issue.
• 32:07
  SPEAKER_01: So if we want to increase our customer satisfaction by, let's say, 30 percent, it's not the customer service function that is responsible, but the back end, the whole value chain leading to the design of the products and services.
• 32:24
  SPEAKER_01: Uh, and we should tackle that uh that uh value chain from different perspectives.
• 32:30
  SPEAKER_01: That's why uh we need to have multiple expertise and every specialist in his domain, like a quality product manager, working with a knowledge manager, and that's where I see knowledge managers play a critical role.
• 32:45
  SPEAKER_01: Those are the facilitators, and those have the responsibility to create the space, uh a safe space where we can include all those visions and opinions uh for uh having a better quality decisions.
• 33:01
  SPEAKER_01: Then Gen AI can have multiple capabilities, for example, uh giving us more insights on on where we need to improve in terms of product features and go back to product development and do the implementation.
• 33:15
  SPEAKER_01: So uh so, in terms of resources, it is a community-based uh structure rather than a centralized team where we delegate the responsibility to the centralized team.
• 33:27
  SPEAKER_02: That's a great point.
• 33:28
  SPEAKER_02: You know, it's it's almost like when when organizations were trying to uh incorporate more business intelligence and analytics into their organizations, you know, they started with you know centers of excellence or centralized teams, but then found that distributing that knowledge and distributing that expertise into the various teams, business units, departments, organizations was really much more effective, right?
• 33:52
  SPEAKER_02: Uh it's great to centralize your knowledge and to build some best practices that then can be shared, you know, kind of shampoo, rinse, repeat, right?
• 34:00
  SPEAKER_02: Once we know what that structure is, but you really need to embed it at the business level and at the um at the business unit and operating unit and uh departmental level.
• 34:10
  SPEAKER_02: Um, so those are great points.
• 34:12
  SPEAKER_02: And tell me that you know, just to kind of go back to the um idea of uh generative AI and some of the problems with hallucinations, with you know, misalignment with the brand, with you know, potential loss of IP, you know, and and we look at retrieval augmented generation as the mechanism to mitigate some of that.
• 34:32
  SPEAKER_02: For folks that are not as familiar with that, do you want to kind of define that or talk a little bit about you know what you're trying to mitigate with retrieval augmented generation and in and a little bit about how it works and how it solves some of those problems?
• 34:46
  SPEAKER_01: Yes, perfect.
• 34:47
  SPEAKER_01: So uh first uh let me lay down the foundations uh before the RAG uh retrieval augmented.
• 34:54
  SPEAKER_01: Uh so there was two types of uh technologies, the extractive um AI and the generative AI.
• 35:02
  SPEAKER_01: Uh with extractive AI, uh uh it works on a set of our corpus, so a set of documents we already have in the company, and extractive AI, what uh it does, they go and search for the relevant documents and extract exactly the sentence that answers my questions.
• 35:23
  SPEAKER_01: And if we don't have any kind of uh answer, the the outcome will be uh zero, or we don't we won't have an outcome.
• 35:31
  SPEAKER_01: So that's extractive AI.
• 35:33
  SPEAKER_01: Uh and then with generator of generative AI, uh with some like our question with context, now the model can create or generate new contents that is similar to the trained data, and now there's a question related how true or how false is the outcome.
• 35:53
  SPEAKER_01: That's another issue we can discuss it later.
• 35:56
  SPEAKER_01: Uh, but what they do with generative AI uh now is there's something called temperature.
• 36:03
  SPEAKER_01: So either temperature is like how much you want your model to go uh uh like creative, yes.
• 36:10
  SPEAKER_01: So how much you want your content to uh to resemble some kind of abstract uh uh art, some kind of surrealism uh or some kind of fantasy.
• 36:21
  SPEAKER_01: And this is where the temperature is equal to one, then you will get uh content that's totally uh imaginative.
• 36:29
  SPEAKER_01: Uh and uh if you diminish or you optimize your temperature like 0.4, 0.5, you will get somehow relevant content.
• 36:37
  SPEAKER_01: So in in RAG, retrieval augmented uh generation, uh there's kind of some kind of combination.
• 36:44
  SPEAKER_01: Getting exactly the sentence where answers my questions, but put some additional information and additional content and generate a new uh content that will help me to better understand or expand on that specific answer.
• 37:00
  SPEAKER_01: So a reg is very powerful, provided that we already find an answer in our corpus.
• 37:07
  SPEAKER_02: One of the things that um that we found was that by specifying to the model, we turn the temperature down to zero, right?
• 37:17
  SPEAKER_02: And we said, if you don't have the answer from this data source, say I don't know.
• 37:23
  SPEAKER_02: So that is really not giving it any room to be creative, right?
• 37:26
  SPEAKER_02: Uh, because not things factually prepared.
• 37:28
  SPEAKER_02: No, they still the answer would still vary depending upon how you ask the question.
• 37:33
  SPEAKER_02: And what's really interesting is audit trails, when we showed where the the data that was like comprising the answer, it didn't look very friendly, right?
• 37:42
  SPEAKER_02: It didn't look it, you couldn't really make sense of it.
• 37:45
  SPEAKER_02: It had snippets and pieces, and it's like, wow.
• 37:48
  SPEAKER_02: So the model really took those data sources and those pieces that it extracted from the retrieval piece, and then it made them more conversational.
• 37:58
  SPEAKER_02: So there's so there's processing at the front end of the the utterance of the question that people have, right?
• 38:04
  SPEAKER_02: And and normalizing that, and then using that to retrieve information, you know, maybe from a you know uh parametric search from a knowledge base or or uh or uh uh uh a uh or a uh uh using a um uh blanking at the moment, uh a vector space and vector similarity search, uh, and then and then processing the result uh to make it more conversational.
• 38:29
  SPEAKER_02: But uh but it was very interesting to see what it had done.
• 38:32
  SPEAKER_02: And and when we also enriched the content with additional metadata, with the knowledge mark, with the knowledge architecture, sometimes it would pull from content and then augment that with the metadata, sometimes it would pull from the metadata and augment that with the content, but it was always making it more conversational.
• 38:51
  SPEAKER_02: So that was those are very interesting findings.
• 38:54
  SPEAKER_01: Yes, and uh one way also back to the guiding principles, continuous supervision.
• 39:00
  SPEAKER_01: When we uh when we find lack in the outcome or the model is not giving us uh some kind of uh uh uh satisfying satisfying results, this is where the continuous supervision.
• 39:12
  SPEAKER_01: So we need to go back to our model, retrain the model, fine-tune it with additional information, and then before going to uh to the deployment, uh do some testing.
• 39:24
  SPEAKER_01: So get the real experts, uh, get human in the loop and do some kind of testing, new questions, and validate the answers.
• 39:34
  SPEAKER_01: So this is a continuous process.
• 39:36
  SPEAKER_01: Uh, we will not get it correct or right from the first time, and we need to reiterate on it.
• 39:41
  SPEAKER_01: Yeah, and that's come with a price, it's not uh for free.
• 39:45
  SPEAKER_01: There's a price related either to the infrastructure or to the time uh invested from our experts.
• 39:53
  SPEAKER_02: Yeah, that's a really great point.
• 39:54
  SPEAKER_02: I mean, when people are are talking about, you know, just point the LLM to your content and are expecting quality results.
• 40:03
  SPEAKER_02: That's not really that's not realistic, right?
• 40:05
  SPEAKER_02: Yes.
• 40:06
  SPEAKER_02: Uh it really depends on the use cases.
• 40:08
  SPEAKER_02: Those use cases have to be defined, and we need a bank of them.
• 40:12
  SPEAKER_02: You know, we use we test 60 use cases for this application uh we recently built, and then you can you can see if they're factually correct, but it's not easy sometimes because they will phrase the answer very differently.
• 40:26
  SPEAKER_02: And so you need some mechanism to uh test that result.
• 40:31
  SPEAKER_02: Um uh I have uh a colleague who has been using large language models to test the similarity of the result.
• 40:38
  SPEAKER_02: Do you have you been doing that?
• 40:39
  SPEAKER_02: Have you uh used that type of approach in terms of um model testing?
• 40:44
  SPEAKER_01: Yes, uh in fact, uh I'm uh I'm doing some personal project, project-based learning for my uh just uh personal satisfaction.
• 40:54
  SPEAKER_01: Uh and I'm trying to build a model using open source uh uh foundational model, haystack.
• 41:01
  SPEAKER_01: Uh and I have like many of us, I have so many uh so many documents related to knowledge management or the books that I have wrote or any kind of publication.
• 41:11
  SPEAKER_01: And I said, okay, I wanted to get a sense, what does it take to build from A to Z an application where I go ask a question what are the roles and responsibilities for a knowledge manager?
• 41:23
  SPEAKER_01: The answer is already in my documents, but uh it will save me some time.
• 41:27
  SPEAKER_01: Uh so I took the haystack model, which is the foundational model, general training and general purpose model, uh, and then I retrained the model on those documents.
• 41:40
  SPEAKER_01: Uh, the more I gave him documents, the more I retrained the model, the more accurate information I uh I get.
• 41:48
  SPEAKER_01: And there's some technique I used, it's it's called lost in the middle technique, and it's essentially called what is it called lost lost in the middle.
• 41:57
  SPEAKER_02: Lost in the middle, okay.
• 41:59
  SPEAKER_01: Yes, technique.
• 41:59
  SPEAKER_01: And it is this technique is used when uh deploying or developing rag applications.
• 42:06
  SPEAKER_01: What's this technique is about?
• 42:08
  SPEAKER_01: So I have a set of documents, uh and uh I have some documents are relevant or some documents are not relevant.
• 42:15
  SPEAKER_01: Lost in the middle will give me the most relevant in the top, the least relevant in the middle, and the most relevant at the end.
• 42:26
  SPEAKER_01: So, in this way, if I'm navigating 10 responses, uh I will get the most relevant in the beginning and at the end, and in the middle it will be lost because I will not pay attention, some kind of grabbing the attention of the end user to display the most relevant at the top and at the end of the results.
• 42:46
  SPEAKER_02: Interesting.
• 42:47
  SPEAKER_02: That's very interesting.
• 42:49
  SPEAKER_02: So when you think about um where this is going for organizations, I mean, you know, many organizations are not terribly mature in their knowledge processes, right?
• 43:00
  SPEAKER_02: And they've kind of looked to technology to solve the problems, right?
• 43:04
  SPEAKER_02: I need a better search engine, I need search semantic search, now I need neural search and all these things, right?
• 43:10
  SPEAKER_02: So what are you what's your advice to executives who are going down this path and you know are are trying to you know solve these problems from a technological perspective, and you know, perhaps do not have the the most um meaningful, robust or mature uh knowledge processes.
• 43:31
  SPEAKER_02: Where do they start?
• 43:33
  SPEAKER_02: Where do they begin?
• 43:34
  SPEAKER_02: And is it going to be a matter of learning the hard way, like trial and you know, failure before they as understand these principles?
• 43:43
  SPEAKER_02: What are your thoughts?
• 43:44
  SPEAKER_02: Wait, and you must talk to other colleagues and peers in the industry.
• 43:48
  SPEAKER_01: So what what what we we we must learn from the past, as you mentioned, already we saw uh the enterprise search engine uh like in mid and twenty.
• 44:00
  SPEAKER_01: 2015 or 2013, uh this uh this tendency or this uh let's say uh uh trend to deploy everyone to deploy uh enterprise search engine and they were frustrated their results were not as good as what Google um uh uh provided, and that's because internally they didn't uh do the governance or they didn't model their processes or design their knowledge architecture, and they didn't really understand how their the knowledge flows and their network is connected together.
• 44:35
  SPEAKER_01: So we don't want to repeat the history.
• 44:38
  SPEAKER_01: Uh if we really we wanna uh get the best out of AI, we first go back to our uh our operations, uh model it, design it, understand how they are connected, and then go into AI.
• 44:52
  SPEAKER_01: And not uh and I don't also uh uh recommend to fully deploy AI.
• 44:57
  SPEAKER_01: Search for those quick wins, easy to implement, high impact on the customer, because there's a cost, underlying cost, which is very uh very heavy cost.
• 45:07
  SPEAKER_01: Uh, today the big companies Facebook, Amazon, OpenAI, they are giving us the foundation model and they are charging a lot in terms of their revenue model and business model.
• 45:19
  SPEAKER_01: For example, uh uh Google Palm model, they are charging uh two cents for 15 seconds of interaction.
• 45:28
  SPEAKER_01: So if we do the maths math for one hour, they are charging$12.
• 45:33
  SPEAKER_01: This is very uh uh a rolling bill, right?
• 45:36
  SPEAKER_01: And it will go exponential.
• 45:38
  SPEAKER_01: So we don't want to go to say let's deploy everywhere AI.
• 45:41
  SPEAKER_01: Identify those quick wins, small applications, really small, where they have the most impact.
• 45:47
  SPEAKER_02: Yeah, no, that's a that's a really great point.
• 45:50
  SPEAKER_02: Um, so we have a few minutes left, and I wanted to um, you know, I I understand you're a big fan of of skiing and uh skiing in the uh Grenoble uh in the Alps.
• 46:01
  SPEAKER_02: So tell me where else have you skied and tell me a little bit about what else you do for for fun and uh outside of work.
• 46:08
  SPEAKER_01: Yeah, so I'm located in Grenoble on the Alps Mountains or 360 degree.
• 46:14
  SPEAKER_01: I have the Alps Mountains on the Swiss uh borders.
• 46:18
  SPEAKER_01: Uh and ski is uh it was part of uh my university uh curriculum.
• 46:24
  SPEAKER_01: So we had a course and we had some activities where we go out and we learn skiing.
• 46:31
  SPEAKER_01: Uh I came from Lebanon on uh the north coast, so I I was raised uh uh on the beach coast and I never did the uh skiing.
• 46:42
  SPEAKER_01: So when I came to France and uh went with my friends, I was the only person who was falling and falling again.
• 46:52
  SPEAKER_01: And the trainer or uh the the mentor advised me to enroll in a training course, uh and I went and enrolled uh and I was put uh with the pew pew.
• 47:03
  SPEAKER_01: Pew pew are those who are have three years old who are now learning learning ski.
• 47:09
  SPEAKER_01: So it doesn't seem adequate.
• 47:13
  SPEAKER_01: I took then personalized and individualized trainings, but then uh then I I got my little star and I was so proud of it.
• 47:23
  SPEAKER_02: I I learned to ski uh a little later in life as well in in my uh early 30s, and I took a 10-day learn to ski course, and at the end of it, I had some private lessons in the afternoon.
• 47:34
  SPEAKER_02: I had group lessons morning, private in the afternoon, and my ski instructor took me down a black diamond at the end of that uh 10 days, and he was like uh high-five high-fiving the other instructors because I guess it was a big deal to take someone who's never been on skis and then get them to a black.
• 47:51
  SPEAKER_02: I can't ski black diamonds very well now.
• 47:53
  SPEAKER_02: So I wouldn't say I skied the black diamond, I think I kind of stumbled down it and made it.
• 48:02
  SPEAKER_01: Well, you're welcome anytime.
• 48:03
  SPEAKER_01: Let's go and do some kind for fun skiing.
• 48:06
  SPEAKER_02: I would love to.
• 48:07
  SPEAKER_02: Oh, I would definitely take you up on that.
• 48:09
  SPEAKER_02: Um, so I want to ask you a quick so if you could ask uh everyone this question, if you could go back in time and give yourself some advice, you know, when you were getting out of college, uh what what might you tell yourself?
• 48:21
  SPEAKER_02: What kind of uh advice would you give yourself?
• 48:24
  SPEAKER_01: Oh, that's uh a good question.
• 48:26
  SPEAKER_01: I'm trying to buffer to gain some time.
• 48:31
  SPEAKER_02: Anything that uh that you look back and say, geez, you know, I I would tell myself uh this or give myself advice.
• 48:39
  SPEAKER_01: Um yes, so maybe what I will tell myself to be uh more uh aggressive in terms of uh jumping more ahead and have uh forward uh leap face uh and try not really to analyze a lot of my decisions, just do it, uh jump ahead and see what will happen.
• 49:03
  SPEAKER_02: Take more risks, yes, more risk, yes.
• 49:05
  SPEAKER_02: Yeah, that's great.
• 49:07
  SPEAKER_02: Well, listen, I want to thank you, Rashad, very much for my pleasure.
• 49:11
  SPEAKER_02: This has just been tremendous, and um, maybe we can have you on our webinar on governance, that would be great where we could do some collaboration.
• 49:19
  SPEAKER_02: So I'm looking forward to continuing the discussion.
• 49:21
  SPEAKER_02: So thank you so much for your time today.
• 49:24
  SPEAKER_01: All my pleasure, Tess, and thank you for uh having me today.
• 49:28
  SPEAKER_02: And I want to thank our audience.
• 49:29
  SPEAKER_02: Uh, we'll see you next time, uh, and also thank uh Carolyn and Liam for uh doing the production behind the scenes.
• 49:36
  SPEAKER_02: Uh so thanks again, uh everyone, and we'll we'll see you next time.
• 49:41
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 49:44
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 49:51
  SPEAKER_00: The Early AI podcast is sponsored by Early Information Science at CMSY.
• 49:56
  SPEAKER_00: To learn more, visit early.com.
• 49:58
  SPEAKER_00: That is e-a-r-y.com.
• 50:03
  SPEAKER_00: Thanks for tuning in.