Latest / The Adults In The AI Room / Erdem Özcan on the Future of Neurosymbolic AI - The Earley AI Podcast with Seth Earley - Episode #046

Transcript

Erdem Özcan on the Future of Neurosymbolic AI - The Earley AI Podcast with Seth Earley - Episode #046 · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast, hosted by Seth Early and Chris Featherstone.
• 0:06
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:15
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as CMS Wire.
• 0:25
  SPEAKER_00: Now, enjoy the show.
• 0:31
  SPEAKER_02: Welcome to the Early AI Podcast.
• 0:34
  SPEAKER_02: My name is Seth Early.
• 0:36
  SPEAKER_02: And I'm Chris Featherstone.
• 0:37
  SPEAKER_02: It's good to be with you, Seth.
• 0:38
  SPEAKER_02: It's been a while.
• 0:39
  SPEAKER_02: It's been a little while since we've we've been able to do a show together.
• 0:43
  SPEAKER_02: Today, I'm really excited to introduce our guest.
• 0:46
  SPEAKER_02: We're going to talk about knowledge management applications and the importance of knowledge and reasoning, especially with regard to generative AI.
• 0:55
  SPEAKER_02: Generative AI applications and how they depend on knowledge, the benefits of centralization and of AI capabilities within an organization, looking at how to solve problems with generative AI, again, especially around reasoning.
• 1:10
  SPEAKER_02: A lot of generative AI initiatives don't handle reasoning very well.
• 1:14
  SPEAKER_02: So our guest today is the VP of Intelligent Applications at Elemental Cognition.
• 1:19
  SPEAKER_02: He's leading the development of a platform that combines generative AI with the formal reasoning methods to deliver reliable and explainable outputs.
• 1:28
  SPEAKER_02: So, Erdem Asken, welcome to the show.
• 1:31
  SPEAKER_02: Hi, nice meeting you all.
• 1:33
  SPEAKER_02: Thanks for this nice introduction.
• 1:35
  SPEAKER_02: Absolutely.
• 1:36
  SPEAKER_02: So, you know, we'd like to start, we'd like to start the show with, you know, talking about misconceptions and things that you're running into in the marketplace.
• 1:44
  SPEAKER_02: And what are you seeing in terms of major misconceptions around AI, LLMs, knowledge, and reasoning?
• 1:52
  SPEAKER_02: But take those however you'd like to.
• 1:54
  SPEAKER_02: But what do what are you hearing when you go into organizations and start talking about solutions?
• 1:59
  SPEAKER_01: Sure.
• 1:59
  SPEAKER_01: I think one of the things that are actually really exciting about AI is that over the last five years to a decade, you know, AI has been in like a very compelling uh you know technology for a lot of enterprises.
• 2:13
  SPEAKER_01: A lot of enterprises are trying to figure out what their AI strategy should be.
• 2:18
  SPEAKER_01: And we like over the past decade, a lot of companies have transformed their data to make that happen.
• 2:24
  SPEAKER_01: And then so there's a lot of investment and attention going in that direction.
• 2:28
  SPEAKER_01: And then that has only yet got that only got amplified with you know the LLMs, the large language models, which have become more and more available to all, like you know, to all of us.
• 2:40
  SPEAKER_01: And uh, and so this has increased the amount of attention and investment that the companies want to want to make.
• 2:47
  SPEAKER_01: So this is all good news for the field.
• 2:49
  SPEAKER_01: And I think there is a lot of uh great things that are gonna come out of that.
• 2:54
  SPEAKER_01: On the other hand, like speaking of misconceptions, I think there is sort of a you know, the LLMs have created a demand which has not been really met by LLMs, and and I think that it's gonna be very challenging for LLM to meet this alone.
• 3:09
  SPEAKER_01: And then that challenge is to be a reliable assistant to people so that humans can uh automate some workflows or make some decisions and you know make high-stake decisions.
• 3:21
  SPEAKER_01: What is happening with the LLMs is that they make it incredibly easy to start a project, create a prototype, and see some of the lights that could come out of the AI, but then they actually fail at delivering on the reliability, accountability, and expandability levels.
• 3:38
  SPEAKER_01: And so I think that misconception is leading a lot of companies to start prototypes and failing at deployment.
• 3:46
  SPEAKER_01: And so this is basically the main challenge that we are focusing on at elemental cognition, uh, how to uh you know combine deep reasoning and formal reasoning with the LLMs to deliver on that kind of uh promise.
• 4:02
  SPEAKER_02: So, so you know, we had talked a little bit about prompt engineering being a bit of a misnomer, that it's not an engineering discipline, though people use it as such.
• 4:11
  SPEAKER_02: And I'm actually, that's a topic of uh one of our upcoming webinars.
• 4:15
  SPEAKER_02: But why don't you talk a little bit about your thoughts around when people talk about prompt engineering, what they're what they're considering, and what the value to organizing and managing prompts, or at least coming up with some mechanism to develop prompts.
• 4:30
  SPEAKER_02: Uh, what are your thoughts there?
• 4:33
  SPEAKER_01: Okay, great question.
• 4:34
  SPEAKER_01: I think like maybe let's first focus on what is good about the prompts and then and then talk about like you know what what is uh challenging with them.
• 4:42
  SPEAKER_01: I think what is good about the prompts is that over the last 10 to 15 years, deep learning has been like the main technology leading AI uh for different enterprises.
• 4:52
  SPEAKER_01: And the difficulty with deep learning in general is that it is very data hungry.
• 4:57
  SPEAKER_01: You need to have a lot of data to tune systems to behave as you wish.
• 5:01
  SPEAKER_01: So this has been a very big challenge.
• 5:03
  SPEAKER_01: Like, how do I collect this data to start with?
• 5:06
  SPEAKER_01: And then how do I evaluate my systems?
• 5:08
  SPEAKER_01: And once I evaluate my systems, how do I actually fine-tune them to uh get them to exactly to the behavior that I want?
• 5:16
  SPEAKER_01: So, what prompts have opened as a new possibility is that when you don't have as much data, you can actually instruct the system in natural language.
• 5:26
  SPEAKER_01: You can uh say what the instructions and the rules that you want the system to uh consider when making its decisions and you know, subsequent um, you know, follow-up conversation.
• 5:37
  SPEAKER_01: And so this is great in that regard.
• 5:39
  SPEAKER_01: Like, you know, it drastically reduced the amount of data that you know applications need to be deployed.
• 5:45
  SPEAKER_01: Now, the challenge that comes with that is that the prompts are you know described in natural language, they are interpreted by the underlying deep learning systems as tokens of words, and uh, and some tokens of words are produced on the other side.
• 6:02
  SPEAKER_01: So, what happens in between is sort of the magic that uh the new LLM tools are uh are enabling us with.
• 6:09
  SPEAKER_01: And and then the challenge there is that when you express something in natural language, first of all, you do not know if the other part has understood it exactly the way you meant.
• 6:19
  SPEAKER_01: Like, you know, within a conversation like this, I am constructing words, sentences, paragraphs, and my speech around how I see the world, but now you may perceive it or the audience may perceive it in a different way based on their context, their knowledge, et cetera.
• 6:34
  SPEAKER_01: So, like there is this gap of like this missing knowledge which is not said in the prompt.
• 6:39
  SPEAKER_01: And obviously, people can think that they can put that more and more into the prompt.
• 6:43
  SPEAKER_01: Then the next challenge becomes that the prompt is statistically interpreted by this, like, you know, token engine.
• 6:49
  SPEAKER_01: It is not really understood in its meanings.
• 6:51
  SPEAKER_01: There is no semantic backplane to map to to really understand how the system is going to behave.
• 6:57
  SPEAKER_01: So then what happens is that people create prompts, they evaluate the systems, the system may or may not really work the way they they wished.
• 7:06
  SPEAKER_01: They go back and then change the prompt and in an effort to make it better.
• 7:11
  SPEAKER_01: But then what happens is that again, because that we don't know exactly how this is understood by the underlying system, the the updates that we're making may not have the consequences that we were wishing for.
• 7:22
  SPEAKER_01: And as a result, this field that like you know has been called the prompt engineering is more of a like you know a process, an undeterministic process, in which you know you you you try to to uh introduce new things or adjust the behavior of the system, but you don't know how how the underlying LLM is going to behave with that.
• 7:43
  SPEAKER_01: And this is why like I don't like calling it a prompt engineering, because engineering is a well-known methodology discipline.
• 7:50
  SPEAKER_01: And uh with prompts, it is impossible to follow such a discipline and to get to a predictable outcome.
• 7:56
  SPEAKER_04: So do you think that Erdom that prompt engineering is basically just you know, like we used to do with short form-based utterance design, that it's just a version 2.0 of that?
• 8:08
  SPEAKER_04: Does that make sense?
• 8:09
  SPEAKER_04: You know, you have bots, things like that.
• 8:11
  SPEAKER_04: Anyway, go ahead.
• 8:12
  SPEAKER_01: Sorry, I didn't mean to Yeah, I I think this is a good analogy.
• 8:15
  SPEAKER_01: I I think the the the the the uh what we should understand about prompts is that again, like these are like tokens uh that are like statistically interpreted by an engine which has a very big knowledge of things that we don't even know how they are represented in its own knowledge, and then as a result of that, like you know the the systems are behaving in the right way in the sense that like you know, there are people building applications using props.
• 8:41
  SPEAKER_01: The difficulty is how to align the behavior of the application that you're building exactly to what you need, and this is really critical for businesses because a business cannot deploy a system which is not complying with the rules that need to govern the decision making.
• 8:58
  SPEAKER_01: And so this is uh, I mean, LLMs have other challenges too, but like I think the the the um the nice thing about the prompts is that it makes it easy to enter the world of AI and it makes it extremely easy to prototype things with that.
• 9:13
  SPEAKER_01: On the other hand, it makes it as hard to actually create a system, and and this is mainly because of this lack of you know reliable understanding of what is in the prompt and uh and the absence of a semantic backplane or representation uh underlying the decision making in the system, which makes it impossible to actually predict how the system is going to behave.
• 9:36
  SPEAKER_04: I go I go at this, sorry, Seth.
• 9:38
  SPEAKER_04: I I usually go at this in terms of because you know the questioning comes at us around, well, how are you going to define accuracy and you know, so that it doesn't hallucinate to again to derive the outcomes that you're trying to drive?
• 9:51
  SPEAKER_04: So I usually hit it from the perspective of, well, you have an opportunity to build your own model.
• 9:56
  SPEAKER_04: You can take an existing foundation model and fine-tune it.
• 10:00
  SPEAKER_04: You can put together a prompt engineering type of perspective, or you can use RAG.
• 10:04
  SPEAKER_04: And each one of them will use a prompting technique, of course, but that's generally the crescendo.
• 10:10
  SPEAKER_04: Would you agree with that?
• 10:10
  SPEAKER_04: Would you disagree with that?
• 10:11
  SPEAKER_04: You know, or what would you say as a plus plus to that?
• 10:15
  SPEAKER_01: I think you know, there's a lot of methods to control the behavior of an LLM.
• 10:20
  SPEAKER_01: Like you mentioned the most prominent ones like fine-tuning, better prompting, drag, and all that stuff is mainly aiming at you know, fine-tuning the parameters in some way of a model so that it behaves uh the why the way we want it to behave.
• 10:38
  SPEAKER_01: At the end, it is an empirical process, you know, like it is uh it is a statistical process that you're going through.
• 10:44
  SPEAKER_01: The difficulty with that is you're not really controlling the behavior, you're just trying to put statistical boundaries around them, right?
• 10:53
  SPEAKER_01: And so this is something that we are doing very differently uh in my company at Elemental Cognition, where we are actually using formal reasoning.
• 11:02
  SPEAKER_01: And in some ways, you can actually think that this is similar to RAG.
• 11:05
  SPEAKER_01: The difference that I would I would actually draw is that RAG is basically retrieving knowledge from uh text or information or maybe databases, right?
• 11:14
  SPEAKER_01: And then and then prompting the system with that so that you get a more reliable outcome.
• 11:19
  SPEAKER_01: Now, what RAG misses in terms of its own architecture is that when the when answering the question or or making the decision that you expect out of your model is not any is not based on in information which is readily available in searchable data or text, then your system cannot help you.
• 11:40
  SPEAKER_01: Like to give you an example, you know, uh we work with companies uh to help them build workforce planning tools.
• 11:47
  SPEAKER_01: Now, every company has some documentation around how people should take vacation, how they should be scheduled into projects, etc.
• 11:56
  SPEAKER_01: So with RAG, you can take this information and present those rules accurately to someone who is asking, uh, you know, should I let uh Andy go on vacation next week?
• 12:08
  SPEAKER_01: The system with a REG can tell you what the policy is.
• 12:11
  SPEAKER_01: It can tell you that if Andy didn't take like three weeks this year, which is the all out thing, and then they uh they agreed with their manager, then they can actually like take vacation.
• 12:21
  SPEAKER_01: Otherwise, they cannot take vacation.
• 12:22
  SPEAKER_01: So this is what you can expect from a RAC system.
• 12:25
  SPEAKER_01: But then you can bring it to the next level where the system can tell you yes, Andy can take vacation because they he already took like only 10 days this year, he has that much of available, and then there's no upcoming project for them.
• 12:40
  SPEAKER_01: In order for the system to behave like this, it needs to put on top of this like you know base rule set that it understands, it needs to apply it with the data and reason through to give you this synthetic answer at the end.
• 12:53
  SPEAKER_01: Like uh, and and so this is the kind of applications that we are developing for.
• 12:56
  SPEAKER_01: So, like at the basis, you can query the system about the rules, but beyond that, you can expect the system to apply those rules to a particular situation and make the decision and explain why that decision is the right one to you.
• 13:11
  SPEAKER_01: Nice and this is a very powerful, uh, you know, it's it's a very powerful um, you know, um paradigm in which you can actually rely on these systems to make decisions, you can rely on these systems to understand the trade-offs that you might need to make as a human being.
• 13:26
  SPEAKER_01: And then you can do that in a conversational setup, which is actually enabled, like this fluidity being enabled by the LLMs, but not the decision making itself.
• 13:35
  SPEAKER_02: Nice.
• 13:36
  SPEAKER_02: So when you uh when you were saying that you don't believe that prompt engineering is a valid uh discipline or it's hard to engineer, it's more of an art than it is a science.
• 13:48
  SPEAKER_02: I guess you could call it prompt design, right?
• 13:50
  SPEAKER_02: Um, and put a methodology behind it because I do think there are certain characteristics in prompts that you can use to both guide your user and to get better results.
• 13:59
  SPEAKER_02: And especially, I I don't think there's any doubt in many people's minds that you know, retrieval augmented generation, and there's a lot of different ways to do that, is really the way you're going to get the best results based on your data, right?
• 14:12
  SPEAKER_02: That's the source of ground truth in the organization.
• 14:16
  SPEAKER_02: So that does require a reference architecture.
• 14:19
  SPEAKER_02: It does require some uh content conversion or transformation, right?
• 14:25
  SPEAKER_02: Because many people don't, many organizations don't have the knowledge in the first place, right, to support the questions that are being asked.
• 14:32
  SPEAKER_02: So developing that library of use cases, developing the metadata that those use cases represent.
• 14:38
  SPEAKER_02: When people ask questions, they're asking questions about something, you know, a contract or, you know, an article or a procedure or whatever it is.
• 14:47
  SPEAKER_02: And then again, having some of those signals as part of the enriched ingestion and the enriched embeddings in the LLM and the vector space can help retrieve more precisely that content.
• 14:59
  SPEAKER_02: So I agree that is it is part art, but then there's lots of different ways of retrieving that information.
• 15:05
  SPEAKER_02: And then where does the logic and the reasoning layer in?
• 15:09
  SPEAKER_02: Uh, I know we talked a little bit about ontologies and how in ontologies you can do some inference and some reasoning, but where what is your reasoning engine or how is that developed?
• 15:20
  SPEAKER_02: You know, does it sit on top of the LLM?
• 15:23
  SPEAKER_02: Is it where where is it and and how is it configured?
• 15:27
  SPEAKER_01: So uh at the topmost level of the architecture, we actually call our architecture an LLM sandwich.
• 15:34
  SPEAKER_01: So LLM on one side to acquire knowledge, which is one of the breads, you know, it turns your documentation or the knowledge of your subject matter experts into a very well structured document that would govern the decision making.
• 15:50
  SPEAKER_01: And then in the middle, what we call the meat is a formal reasoning engine that is uh developed by our company, Elemental Cognition.
• 15:59
  SPEAKER_01: And um, and then this reasoning engine is able to do you know formal reasoning in terms of logical reasoning, logical inference, abductive reasoning, uh, abductive inference, and also it is able to do optimization, like you know, finite domain optimization, integer linear optimization, etc.
• 16:18
  SPEAKER_01: So it has these base capabilities, which then is driven by a document in English, which tells it what are the objectives of optimization, what are the rules and constraints that the decisions need to be made.
• 16:35
  SPEAKER_01: And so that English is not the full extent of the English language, but instead it is a subset of the English language which is allowing us to capture these rules and objectives in a very precise and unambiguous way, and then that English gets compiled into reasoning code and then gets executed formally.
• 16:55
  SPEAKER_02: That's interesting.
• 16:56
  SPEAKER_01: Yeah, and so this is a very big invention, you know, uh, and and like how to compile, how to define a subset of English which allows you to capture this kind of business rules and objectives, and then compile that into code that can execute um, you know, um on a formal basis.
• 17:13
  SPEAKER_01: And then on the other side, like this is the meat, and then on the other side of the LLM sandwich, there's another brand which is again an LLM, which brings this reasoning, formal reasoning into life through a fluid conversation with the end users, where the reasoning engine can make decisions and then get it can communicate it with end users using the natural language again.
• 17:36
  SPEAKER_01: So, you know, LLMs on one side to acquire knowledge from your documentation and subject matter expert to capture it in a precise form of English, which then gets executed by a reasoning engine in a formal basis, which allows you to be 100% reliable, explainable, and all these like you know, features that are that matter to businesses.
• 17:57
  SPEAKER_01: And then it gets used with by an LLM on the other side of the sandwich to deliver that to humans through conversation.
• 18:04
  SPEAKER_02: Right, right, making the response or the answer conversational because it it'll just pull the data from the source after it reasons, but then it needs to communicate that to a human.
• 18:15
  SPEAKER_02: Yeah, that makes a lot of sense.
• 18:17
  SPEAKER_02: And then you mentioned this is a it's you you're able to compile in a subset of English.
• 18:22
  SPEAKER_02: So this is one of your tools, right?
• 18:24
  SPEAKER_02: Or one of your your languages.
• 18:26
  SPEAKER_02: What is it called again?
• 18:27
  SPEAKER_02: Is it the cogent language?
• 18:28
  SPEAKER_01: Yeah, we call this the cogent language.
• 18:30
  SPEAKER_01: Um, we also call the platform that is using it as as cogent.
• 18:34
  SPEAKER_01: So cogent is a, you know, as a language, it is a form of English which allows you to define, you know, the concepts that needs to be taken into account, their relationships, the rules that govern the interactions between them.
• 18:49
  SPEAKER_01: And then in case it is like the decision making is not just about logical inference, but it needs to take some form of optimization into account, which also allows you to define optimization objectives.
• 19:00
  SPEAKER_02: So for your example with the scheduling, scheduling nurses or coverage, you know, you can't, you know, some of the rules are you can't work more than two shifts in a row, and you can't work more, you know, you can't have too many back-to-backs or whatever it might be, and along with vacation policies and the amount of vacation people have, and you know, what the predicted load uh would be.
• 19:20
  SPEAKER_02: So that's that's taking into consideration all of those different variables, and then using the reasoning language cogent, which is again a subset.
• 19:30
  SPEAKER_02: So can you give us some examples of what would be a cogent term or uh variable?
• 19:36
  SPEAKER_01: So uh for instance, yeah, I mean, if we take the example of nurse scheduling, for instance, in cogent you would say nurse is a type, a concept, a shift is a concept.
• 19:47
  SPEAKER_01: And then you can you would start defining that a nurse may be working on a shift.
• 19:52
  SPEAKER_01: And then you can say the system should decide which shift a nurse should be working on.
• 19:57
  SPEAKER_01: So that's the optimization piece.
• 19:59
  SPEAKER_01: And then for then you can start constraining the system in the decision making that it needs to operate with.
• 20:05
  SPEAKER_01: So, for instance, you can say things like a nurse cannot work more than two shifts on a day.
• 20:10
  SPEAKER_01: Then you can you can introduce the concept of departments.
• 20:13
  SPEAKER_01: You can say, like, department is a new concept, a nurse may be working for a department, and then you can say each department needs to be filled by at least a nurse or like you know, at least three nurses, uh, depending on the situation for every shift on every day.
• 20:27
  SPEAKER_01: And then you can layer in the vacations, like you know, you can say each nurse must take at least one day of a vacation per week.
• 20:33
  SPEAKER_01: And like, you know, you can define whatever rules that you need in that language, and then it leads the at least the you know, reasoning.
• 20:40
  SPEAKER_01: And maybe I should also say that like, you know, we like when we talk about optimization, we very quickly focus on scheduling and planning applications because this is a place where you know optimization has been very valuable for decades.
• 20:54
  SPEAKER_01: What we see is that as we are democratizing access to optimization applications using cogent, there's a lot of different fields where like people want to use it.
• 21:03
  SPEAKER_01: So uh a couple of interesting examples is that you know we're we're applying cogent in uh space planning, like generating a kitchen layout or a wardrop layout.
• 21:15
  SPEAKER_01: You know, like again, you can express your governing rules, like you know, uh, and then you can actually express the compatibility between the different products, and then you can express the user user wishes, and then you can find the most optimal way of generating these you know room layouts and kitchen layouts, etc.
• 21:33
  SPEAKER_01: We're using it with you know some universities to help students make their degree plannings.
• 21:38
  SPEAKER_01: You know, people can I more and more students want to take their own pathway into university studies.
• 21:47
  SPEAKER_01: They want to combine a major with a minor, they want to take some classes that are to interest them.
• 21:52
  SPEAKER_01: And it is fairly easy for them to miss some prerequisites.
• 21:54
  SPEAKER_01: And like when they think they can graduate, they discover that they didn't take some of the classes to graduate.
• 22:00
  SPEAKER_01: And then this can induce additional expenses they need to drop, et cetera.
• 22:04
  SPEAKER_01: So, like, you know, we we built a tool using cogent to to solve that problem.
• 22:08
  SPEAKER_01: So there's a lot of interesting problems that can be you know just like expressed in cogent and use that you know general purpose tool to uh build solutions, uh AI solutions for them.
• 22:18
  SPEAKER_02: You had you had uh mentioned a quote where that AI large AI vendors have written checks that they can't cache.
• 22:25
  SPEAKER_02: And that is in terms of the limitations.
• 22:27
  SPEAKER_02: But how do you think uh the big AI vendors are looking at that or thinking about this?
• 22:32
  SPEAKER_02: And you know, is is the inherent limitation of what an LLM does is just it's not going to be able to do reasoning, or do you see that evolving as these models evolve?
• 22:46
  SPEAKER_04: Well, even define what you know what what do you mean by the uh you know, checks they can't cash, right?
• 22:52
  SPEAKER_04: I think that would be Yes, yes, yes.
• 22:53
  SPEAKER_02: That's okay.
• 22:54
  SPEAKER_02: Yeah, why don't you explain that again?
• 22:56
  SPEAKER_02: It's part of this idea of uh reasoning, but how are they messaging and how is that messaging falling short of reality?
• 23:04
  SPEAKER_01: I think what is happening is that there is a lot of like you know, in in a lot of enterprises, there is some departments that are responsible of creating their AI strategy.
• 23:16
  SPEAKER_01: And uh, and then so these people look at the uh latest technology that is available, and then they build proof of concepts.
• 23:24
  SPEAKER_01: And the proof of concept level, you have happy paths, and then and then you you tune the systems to deliver that happy path, and then you create a proof of concept at demo.
• 23:36
  SPEAKER_01: Then it comes the question of like how do we deploy this and how do we make sure that it is accurate all the time because it's gonna face our customers or or uh people at the company to make decisions, high-stake decisions, right?
• 23:48
  SPEAKER_01: And this is where like the reality confronts the the proof of uh of the concept.
• 23:53
  SPEAKER_01: And what we're seeing happening is that when when it comes to that level, some of the projects have to drop because they actually uh discover that they cannot deploy it in these circumstances.
• 24:05
  SPEAKER_01: Some other projects very quickly become a hybrid solution, you know, like you know, some code gets written to handle some of the situations, some uh if and else gating functions get written to control the behavior of the LLM, some precise procedural code gets inserted into some head situations, which we know this is gonna happen, and it's not actually using the AI.
• 24:26
  SPEAKER_01: And so this the actual deploy solution very quickly becomes a hybrid uh solution, which is using AI plus some procedural code in various different places, right?
• 24:36
  SPEAKER_01: And I think you know, obviously the tech companies are seeing that as a problem, and then there are some methodologies which are being developed.
• 24:44
  SPEAKER_01: And and I would say that like the like one of the most promising approaches uh that I'm seeing these days is that LLMs get combined with tools, and then LLMs can can invoke a tool to perform a certain operation, and uh and then that tool is actually like formal code, it doesn't need to be reasoning all the time, but like one of the examples is search for RAG.
• 25:06
  SPEAKER_01: So that these tools deliver more reliable outcome that the LLM can revelish to leverage to output the end result, and so but this is done in a very, very ad hoc way.
• 25:17
  SPEAKER_01: You can see what we're doing at elemental cognition as one way of architecting this whole thing.
• 25:23
  SPEAKER_01: You can actually think about the reasoning engine as a very capable tool, you know, sitting at the core of the decision making, and the LLMs uh using that tool to help them like deliver uh high-stake decisions to the end users.
• 25:37
  SPEAKER_01: So I think you know, the reality right now is a lot of hybrid codes deliver whatever it is needed.
• 25:44
  SPEAKER_01: I see a lot of academic research now being called like neuro symbolic approaches to AI, which also we used to call it hybrid systems in the past, like combining symbolic and deep learning systems.
• 25:56
  SPEAKER_01: And uh a lot of companies are working on that.
• 25:59
  SPEAKER_01: I like you know, a lot of big companies like Microsoft, DeepMind have like recently published papers uh illustrating that they're they're they're working on these architectures.
• 26:08
  SPEAKER_01: What we do is like falls exactly into that category.
• 26:11
  SPEAKER_01: Like we're delivering an industrial solution to build neurosymbolic AI that can you know uh deliver these reliability and explainability as features of its output.
• 26:22
  SPEAKER_04: I want to dive into that neurosymbolic AI, but first, which you know, maybe as a follow-on to this.
• 26:29
  SPEAKER_04: I know you guys did a benchmark study, right?
• 26:31
  SPEAKER_04: In terms of of you know what types of of reasoning and inference are still challenging for LLMs.
• 26:37
  SPEAKER_04: That's a it's a it's a super important topic that to your point you were alluding to it with some of your you know other comments that I just don't think people grasp the the gravity of, of you know, because I feel like they still think that it can bend light, it can solve world hunger, and it can predict you know who the president of the United States is going to be, right?
• 26:58
  SPEAKER_04: That kind of stuff, which is all fictional.
• 27:01
  SPEAKER_04: So you know, love give some shed some light on the benchmark you guys did and what that looks like in terms of where they're still missing.
• 27:07
  SPEAKER_01: Yeah, thanks for bringing this up because I think this is a very good uh segue to like this deployment issue.
• 27:13
  SPEAKER_01: How do we trust the systems that are working?
• 27:16
  SPEAKER_01: So, what we've done is uh because we see a lot of companies trying to use LLMs to solve these uh challenges, which require reasoning, we took a couple of illustrative examples and then ran our system against GPT-4 to see like where both systems stand.
• 27:33
  SPEAKER_01: So uh we we we took three different uh use cases.
• 27:37
  SPEAKER_01: One of them is workforce and workforce management, the one that I just mentioned earlier, you know, how to schedule people to projects taking into account their availabilities and skills, etc.
• 27:48
  SPEAKER_01: Another one was uh, you know, an application, as I mentioned, that we built.
• 27:53
  SPEAKER_01: How do how do we help students make uh you know uh their plans uh for their uh university pathways?
• 28:00
  SPEAKER_01: And another one was around complex uh travel planning.
• 28:04
  SPEAKER_01: Like how do I how do I create a a travel plan to travel the world, for instance?
• 28:10
  SPEAKER_01: And this is also was this was also motivated by one of the applications that we deployed with with a partner of ours.
• 28:16
  SPEAKER_01: So so we we took these uh three systems and then we created like these cogent descriptions leading to like actually prescribing their behaviors, and then we ran this with our system and we ran this with GPT-4, giving them uh scenarios at different levels of complexity.
• 28:36
  SPEAKER_01: So, for instance, if I picked the the workforce management to illustrate the complexity, like you know, the number of people that are available and the number of projects that need to be run is a level of complexity.
• 28:48
  SPEAKER_01: And then we started with very simple things where which would be a trivial decision to make for a human, and then we brought it to a much more complicated amounts where like you know, decision making suddenly becomes very, very difficult.
• 29:00
  SPEAKER_01: And like bear in mind that all these uh all these optimization issues are NP complex optimization issues, so like they're like non-trivial things to be solved uh by humans.
• 29:10
  SPEAKER_01: And so what we saw there is that GPT can, and like I think this would be representative for all the state-of-the-art LLMs.
• 29:17
  SPEAKER_01: So the LLMs are able to actually answer some of these questions when especially like in the travel space, when there is some world knowledge which gets into the place play, like you know, where the cities are located through the embeddings, etc., it is doing a slightly better job than you know, a you know, degree planning, which may not be as common problem that like you know everybody knows.
• 29:38
  SPEAKER_01: And so what we saw is that like when the complexity is really like you know small, then LLMs were like usually able to uh to deliver uh the good outcome.
• 29:49
  SPEAKER_01: But then as the complexity grows and gets to levels which would actually motivate for using a technology and not relying on human decision making, LLMs start failing.
• 29:59
  SPEAKER_01: And and and then this happens in a gradual way.
• 30:02
  SPEAKER_01: And um, and whereas with our system, it it was 100% correct across the board of complexity.
• 30:09
  SPEAKER_01: And the reason for that again is is is that we're doing formal code, like you know, it is like we we are like taking this English description, we're compiling into a very efficient reasoning code, then this which gets executed in in the reasoning engine.
• 30:25
  SPEAKER_01: So it is like you know, expected that like you know, we can actually get this right across the board, which is like why we're developing this technology.
• 30:32
  SPEAKER_01: So this was one aspect of that.
• 30:35
  SPEAKER_01: I think the the next aspect, which is maybe even more critical than uh being right all the time, is that when you do not have a system which is reliably correct all the time, you need to know when it is right versus when it is wrong, right?
• 30:50
  SPEAKER_01: It is really critical to be able to make that distinction because otherwise you can actually serve outputs which are going to mislead uh users.
• 30:59
  SPEAKER_01: And so this is the second test that we have done in our study is to ask our system, like give it an output and ask our system versus the LLM do you think this is valid given these uh this uh you know this rule set that are uh prompted to you?
• 31:15
  SPEAKER_01: And again, like our system was getting it 100% right for all the reasons that I mentioned.
• 31:20
  SPEAKER_01: What was really interesting is that when we did that with the LLM, we tried different ways of asking this question.
• 31:27
  SPEAKER_01: When we asked, as another person has created this plan, can you validate that it is correct?
• 31:35
  SPEAKER_01: It was actually finding the invalidities more consistently than if we said, you have returned this plan, can you please double check that you you were correct?
• 31:45
  SPEAKER_01: Then it was actually overconfident that it that it would have done it in the correct way.
• 31:50
  SPEAKER_01: And as a result, it was you know doing the operations in in such a way that it is it is actually misleading the humans.
• 31:58
  SPEAKER_01: So this is again like a place where we see that this reliability comes to play in into play a lot, and then the the the actual ability to to classify whether an outcome is correct versus not becomes very important.
• 32:14
  SPEAKER_01: And so again, you know, formal reasoning is able to cope with that uh LLMs or not.
• 32:20
  SPEAKER_02: So so just a quick question.
• 32:22
  SPEAKER_02: When you say your system versus an LLM, I mean the optimization, the traveling salesperson problem is a very difficult problem to solve, right?
• 32:30
  SPEAKER_02: That's the optimization with multi-ultivariant analysis.
• 32:34
  SPEAKER_02: And uh so your and your engine, I imagine, would use some machine learning in doing that optimization, correct?
• 32:42
  SPEAKER_01: It actually doesn't use machine learning.
• 32:44
  SPEAKER_01: Instead, it uses you know some operations research tools to do that.
• 32:48
  SPEAKER_01: So again, the execution is formal, not statistical.
• 32:52
  SPEAKER_02: Okay, and then you know, you when you say your system versus an LLM, you're still using an LLM in some way, but you're saying using your reasoning engine, you're still using the the uh the sandwich, right?
• 33:06
  SPEAKER_02: But you're saying that your reasoning engine versus the LLM by itself is what's yeah, correct.
• 33:12
  SPEAKER_02: And then as it increases in complexity, the LLM does not uh keep up.
• 33:17
  SPEAKER_01: Correct.
• 33:18
  SPEAKER_01: But the the the interesting thing to see here is that the input is in English, it is not you're not comparing an English prompt versus uh code that is written in Python or whatever.
• 33:28
  SPEAKER_01: The the input is in English, and then that English input is compiled into reasoning code on one end or statistically managed on the other end.
• 33:39
  SPEAKER_01: So that's the main difference.
• 33:40
  SPEAKER_01: Otherwise, if we were comparing like code written in Python versus like a specification written in English, that wouldn't be a fair comparison.
• 33:49
  SPEAKER_02: Now, when you mentioned uh symbolic, symbolic is is well, why don't you give your definition of symbolic AI versus versus uh the statistical approaches?
• 33:58
  SPEAKER_02: But it's really representation of knowledge, right?
• 34:00
  SPEAKER_02: It's really representation of the the you know, the is-ness and aboutness, I like to call it the metadata structures, the metadata models, the the uh uh content architecture, the knowledge architecture.
• 34:11
  SPEAKER_02: Do you want to talk a little bit more about because it always seems to me it's always had to be a combination of both, but do you want to talk a little bit more about you started to mention this before, the uh the neurosymbolic um approach?
• 34:24
  SPEAKER_02: Sure.
• 34:24
  SPEAKER_01: Uh I mean my definition of the symbolic AI would be a a process in which we encode the knowledge in some formalism using symbols, and then we can translate these symbols into something that can be executed on a computer, right?
• 34:42
  SPEAKER_01: And and then we would have a defined process of getting the symbolic knowledge into something that can be executed by the compiler.
• 34:50
  SPEAKER_02: Now you can and again, symbol symbol is is metadata essentially, right?
• 34:56
  SPEAKER_04: Yeah, yeah, what's an example of a symbol, you know, that that kind of thing.
• 34:59
  SPEAKER_01: Like one of the one of the things that I think a a popular view of symbolic AI, as you said, like ontologies, knowledge knowledge representation.
• 35:08
  SPEAKER_01: We typically start indexing on the hierarchies between things, like as defined in an ontology.
• 35:16
  SPEAKER_01: We typically start indexing on what subsumes the other thing, what is the like subtype of another thing, how one thing relates to another thing, like you know, in terms of singular or multiple relationship, etc.
• 35:31
  SPEAKER_01: So um I would say that it it doesn't need to be limited to that.
• 35:36
  SPEAKER_01: It can start getting into rules, like you know, uh again, like a a nurse cannot work more than two shifts a day.
• 35:43
  SPEAKER_01: There is, yes, an ontological relationship between um you know the nurse and the shift, nurse working a shift from an ontological perspective, but on top of that, there is a constraint that comes on that, like they cannot work more than two shifts.
• 35:59
  SPEAKER_01: In the same way, optimization objectives can be represented in that way.
• 36:03
  SPEAKER_01: You know, I want you know at least that much people to be scheduled, and maybe the cost of missing a like, you know, the cost of not fulfilling a department on a day is much higher than not allowing a person to go on vacation.
• 36:19
  SPEAKER_01: So you can start also introducing this kind of concepts in symbolic ways.
• 36:24
  SPEAKER_01: So I think the the main difference there is that there is a challenge of representing things that can be said in natural language in these symbolic forms.
• 36:34
  SPEAKER_01: And this is all like the all what the research around symbolic AI is like how can we represent different things that are happening in the world in symbolic ways that that can then can be uh you know executed in a reliable fashion by a computer.
• 36:49
  SPEAKER_01: And the the difference there is that because this is a very big challenge, statistical systems come in and and then they actually introduce a way of making that mapping, not maybe in such a reliable way, but in more flexible ways that can uh cover a larger basis.
• 37:09
  SPEAKER_01: And I think that uh yeah, and then this is where the fluidity comes from.
• 37:12
  SPEAKER_01: You know, like how do I actually translate language into this symbolic knowledge?
• 37:17
  SPEAKER_01: And I think there are tools of thought, I would say, in AI.
• 37:20
  SPEAKER_01: Like some people are like only believing into statistical systems because they are thinking that there needs to be the learn, the learning part needs to be the only part of the equation, and everything can be learned, which is like true up to a certain degree, which we all humans uh learn in that way.
• 37:38
  SPEAKER_01: Like we start almost from scratch and then we learn things along the way.
• 37:42
  SPEAKER_01: But yet at the same time, you know, some of the things that can be learned do not necessarily result in very efficient representations, which is where all these mistakes are coming in, etc.
• 37:52
  SPEAKER_01: And uh and then humans have developed symbolisms to deal with this kind of stuff.
• 37:57
  SPEAKER_01: We invented mathematics to to, you know, like rather than talking about equations or like you know, mathematical things between each other to make sure that we can solve this in a reliable way.
• 38:08
  SPEAKER_01: We invented maths, right?
• 38:09
  SPEAKER_01: We invented mathematical representations and methodologies.
• 38:13
  SPEAKER_01: So I guess the symbolic AI camp in this is wants to leverage all these methodologies which are like known to be very efficient.
• 38:21
  SPEAKER_01: And neurosymbolic AI is all about combining these two.
• 38:24
  SPEAKER_01: It is not belonging to one of these extremities, but saying that probably the better outcome is gonna come from come from the combination of these two tools.
• 38:34
  SPEAKER_02: Right.
• 38:34
  SPEAKER_02: And if you're using REG at any point of that equation, then you need to have some curated and quality knowledge, as you had mentioned.
• 38:43
  SPEAKER_02: You know, you need to structure that in the right way.
• 38:45
  SPEAKER_02: What is it, what does it take to become become proficient or fluent in cogent?
• 38:51
  SPEAKER_02: So because it sounds like that would be the key to an organization being able to kind of leverage this fully.
• 38:58
  SPEAKER_02: Is that that a business analyst role?
• 39:01
  SPEAKER_02: Um, uh, and what is it?
• 39:02
  SPEAKER_02: What's the learning curve on that?
• 39:04
  SPEAKER_04: I would I would add on to that too, Seth, like in just in terms of you know, the the fortunate or unfortunate perspective is that everybody, if they've got one, you know, they define AI by you know by GPT three and four, right?
• 39:20
  SPEAKER_04: Which you know is is not the case.
• 39:22
  SPEAKER_04: But at the same time, what does that look like in terms of you know it's it's set the precedents, right?
• 39:28
  SPEAKER_04: And the unconscious bias.
• 39:29
  SPEAKER_04: So what does that look like comparatively with those, maybe as a as a predecessor?
• 39:34
  SPEAKER_04: So that's a question, you know, what does it look like within your platform?
• 39:38
  SPEAKER_04: And two, what does that look like as to why your platform over something else?
• 39:44
  unknown: Right?
• 39:44
  SPEAKER_04: Maybe like a GPT that people have experience with, you know what I mean?
• 39:47
  SPEAKER_04: Well, GPT for I shouldn't say just GPT because they're all large languages, but you know what I'm saying?
• 39:52
  SPEAKER_01: I I'll maybe answer the second question first and then go to the uh the first one.
• 39:56
  SPEAKER_01: So if you if you take the LLM world, the main reason would be the requirement for reliability and explainability, basically accountability, you know, like you don't want to be if you don't want to be in a situation where the outcome of your AI system is misleading people, and then this may have a an impact on your uh brand, your uh credibility, your business finances, then you probably don't want to use an LLM at this stage, like where they're at.
• 40:24
  SPEAKER_01: So this is where, like, you know, I think against an LLM, this is what people would do.
• 40:29
  SPEAKER_01: On the other side, there's a lot of operational research tools that are available out there.
• 40:33
  SPEAKER_01: And like, you know, uh the like optimization and operational research has been around for like longer than statistical AI.
• 40:40
  SPEAKER_01: And they're very well-known tools.
• 40:42
  SPEAKER_01: I think there the problem becomes that using these operations research tools require a certain amount of skill, which is very scarce.
• 40:51
  SPEAKER_01: You know, you need to understand these concepts, you need to understand maths, you need to know these libraries and know how to use them efficiently.
• 40:58
  SPEAKER_01: And I think this is why, like, you know, uh, there is, you know, when you speak to an airline, for instance, they have their own operations research uh teams uh that are developing systems for them because this is so valuable for their business.
• 41:11
  SPEAKER_01: But then there is a lot of other core of businesses which cannot afford to have these experts.
• 41:17
  SPEAKER_01: So they are either making calls to consultancy firms which have these expertise, or they don't do it.
• 41:23
  SPEAKER_01: And I think this is where like Cogent comes in as a force for like you know, democratizing this kind of access.
• 41:29
  SPEAKER_01: One of our partners recently told me that like you're basically taking this PhD level skill and then and then bringing it to business analysts.
• 41:37
  SPEAKER_01: Wow.
• 41:37
  SPEAKER_01: So this is this is uh this is one part of it.
• 41:40
  SPEAKER_01: The the for to answer the first question, like what is it to become like you know, fluent, incogent?
• 41:46
  SPEAKER_01: Is um so there is this is a platform which comes like first of all, it's not just a language, it's a platform.
• 41:51
  SPEAKER_01: The platform helps you capture your business knowledge, it generates APIs for for being able to connect into your application and generates the conversational interface so that you can power conversational experiences with user users.
• 42:03
  SPEAKER_01: So it is it is coming as a whole.
• 42:05
  SPEAKER_01: Um, so by you know adopting this platform, you're not like you know, you're also shortcutting a bunch of operations that you would do uh elsewhere.
• 42:13
  SPEAKER_01: And then you need to learn this language uh and be fluent in writing that language.
• 42:17
  SPEAKER_01: One of the things that we are doing is uh is that we actually, like as I said, left part of the LLM sandwich in my mental model is acquiring cogent from natural language.
• 42:29
  SPEAKER_01: So uh we have a chat-based tool where you can say what you would like to do in free form English, and it would translate it into this more precise form of English.
• 42:40
  SPEAKER_01: And this is very interesting, like it is somewhat similar to copilot chat, where you can say, like, I want to write uh this code, and then it would return the Python code.
• 42:48
  SPEAKER_01: But it is really interesting when you experience this is that the output is also English, like it is almost like a translation tool, and it is very easy to look at what you said and then look at what the outcome is and then apply it or not and configure it.
• 43:02
  SPEAKER_01: So we are building these facilitator tools for people to uh easily translate their business knowledge into uh into uh into cogent English.
• 43:13
  SPEAKER_02: Right.
• 43:13
  SPEAKER_02: I wanted to switch gears a little bit and just uh ask you about your journey and uh you know who you are as a person.
• 43:20
  SPEAKER_02: I see a musical instrument in the background, uh your musician.
• 43:24
  SPEAKER_02: And uh, how did you kind of uh arrive at what you're doing and where you are?
• 43:29
  SPEAKER_01: Yeah, I I think like my journey was a lot like you know, a lot of like right on time decision making.
• 43:35
  SPEAKER_01: I actually, as my name indicates, I'm originally from Turkey.
• 43:38
  SPEAKER_01: I grew up in Turkey and then uh you know went to France for my university studies uh with a little push from my family, which I'm very uh you know uh grateful about uh uh after the fact.
• 43:50
  SPEAKER_01: And um so I studied in France.
• 43:52
  SPEAKER_01: Um I studied computer science and uh and then did a PhD in computer science.
• 43:58
  SPEAKER_01: And um I actually had an opportunity.
• 44:00
  SPEAKER_01: To do AI at undergrad and wasn't attracted as much at the time.
• 44:05
  SPEAKER_01: And I prefer to lean into programming languages.
• 44:08
  SPEAKER_01: It was really fascinating.
• 44:09
  SPEAKER_01: And it's my symbolic part in some sense.
• 44:12
  SPEAKER_01: Like I found that the programming languages was one of the biggest inventions, which allow all the computer programs that we have today, and like this level of abstraction and productivity that they delivered.
• 44:24
  SPEAKER_01: So I actually did a PhD in computer science, focused on that for about 10 years as a researcher in the industry.
• 44:30
  SPEAKER_01: And then came around 2010 all these big inventions around like Siri, Watson, you know, like all these NLP-powered applications and challenges.
• 44:41
  SPEAKER_01: And so I increasingly became more and more interested and figured that everything that I did so far was very much applicable into those fields.
• 44:49
  SPEAKER_01: And sort of decided to change my expertise.
• 44:52
  SPEAKER_01: Joined a large company where it was actually working on Nuance, not IBM.
• 44:57
  SPEAKER_01: Sorry, it was working on Watson.
• 44:59
  SPEAKER_01: And uh and then started like we I've been very privileged to be working on this land park landmark uh you know invention, which beat a GeoPardy, and then we we we wanted to apply it to the help of medical doctors.
• 45:12
  SPEAKER_01: I worked on that, I was like very lucky to work with a RoadStar team on that, and they taught me a lot.
• 45:19
  SPEAKER_01: And then I at some point I like I was like something in me was always interested in product and application and not deep research.
• 45:26
  SPEAKER_01: And so I started my own uh company with a few co-founders.
• 45:30
  SPEAKER_01: We built actually a neurosymbolic AI solution for marketing.
• 45:33
  SPEAKER_01: You know, we built chatbots that help like people express their needs and find the best products for them, and uh, you know, grew that company and we got acquired, and then you know, I I took some break and then uh decided to join elemental cognition because, like, as you may see in the in that narrative, like I've been always in between these symbolic and deep learning uh aspects.
• 45:53
  SPEAKER_01: And like in this company, I I am able to create stuff that is like really falling into this architecture that I deeply believe in.
• 46:00
  SPEAKER_02: That's great.
• 46:01
  SPEAKER_04: Oh, yeah, with with uh with tons of investment, you do a lot of neat interesting things.
• 46:05
  SPEAKER_04: Okay, let me ask you something, Erdom.
• 46:07
  SPEAKER_04: Um within the last uh last portion of of this, you know, the time that we have, is there something that we could have asked you or should have asked you that we didn't or that that you think we should?
• 46:24
  SPEAKER_01: Well I I don't know, like the the one of the one of the things could be maybe where this is all heading towards.
• 46:32
  SPEAKER_01: I don't know.
• 46:32
  SPEAKER_01: Maybe a very very big question.
• 46:35
  SPEAKER_01: And and I think the the um I mean I I can share obviously like my perspective into that.
• 46:41
  SPEAKER_01: I think that um a lot of efforts are going into deep learning-based method methods, and it is like you know, well justified in a lot of cases.
• 46:51
  SPEAKER_01: Now, it also goes with a lot of like money invested in that because developing deep learning systems is also a very extensive uh adventure.
• 46:59
  SPEAKER_01: Um, so I think that these systems are like going to trend towards like from great demos, they will have to get to great applications because at some point they need to justify themselves from a financial perspective.
• 47:12
  SPEAKER_01: And I think that we are just living the era of you know this transition.
• 47:17
  SPEAKER_01: You know, a lot of LLM companies now are going around a phase where like they need to deploy, and uh, and then there are no very big use cases for them still, if you put aside the creativity, creativity, like help helping to write something, etc., or generate images, like there is still a lot of challenges in like applying them in the larger core of enterprise.
• 47:40
  SPEAKER_01: And I think that what is going to happen is like again, people are going to end up creating hybrid architectures to deal with them in real life situations.
• 47:50
  SPEAKER_01: And and then hopefully this will get us to the next level where you know these symbolic or like you know, some people may not want to call this symbolic, but like programming tools, etc., are going to be combined in more and more well-architected ways with these deep learning systems.
• 48:06
  SPEAKER_01: And I think this is where you know, like the AI, the grand challenge of AI will start converging because this pendulum has been shifting between statistical methods and symbolic methods, uh, like almost on a like 10-year kind of a phase over the last like you know, 40-50 years.
• 48:21
  SPEAKER_01: And and I I I would expect that there is going to be a convergence that is going to come out of that uh as we move on to the next decade.
• 48:29
  SPEAKER_02: Less extreme pendulum swings, and maybe it's going to start set settling down.
• 48:34
  SPEAKER_04: As we as we see normalizing a bit, right?
• 48:37
  SPEAKER_04: Right.
• 48:38
  SPEAKER_04: Yeah.
• 48:39
  SPEAKER_02: Well, this has been great.
• 48:41
  SPEAKER_02: I really appreciate you joining us.
• 48:43
  SPEAKER_02: I appreciate your time and sharing your knowledge.
• 48:46
  SPEAKER_02: So thank you so much for being on the show.
• 48:48
  SPEAKER_01: Yeah, thanks a lot, Chris.
• 48:50
  SPEAKER_01: And it's been a pleasure.
• 48:51
  SPEAKER_01: Thanks for inviting me.
• 48:53
  SPEAKER_02: And uh thanks uh to our audience for uh listening in and uh of course share this if you found it interesting.
• 48:59
  SPEAKER_02: And then uh uh thank you to uh Liam and Carolyn for their help behind the scenes.
• 49:05
  SPEAKER_02: So thanks again, uh Erdon.
• 49:07
  SPEAKER_02: It's been a pleasure and appreciate your spending the time with us today.
• 49:11
  SPEAKER_02: Thank you.
• 49:12
  SPEAKER_02: Okay, appreciate it.
• 49:13
  SPEAKER_02: Take care, Adam.
• 49:15
  SPEAKER_02: We'll see you next time on the uh the next Early AI podcast.
• 49:19
  SPEAKER_02: Okay.
• 49:20
  SPEAKER_02: That would be my pleasure.
• 49:23
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 49:27
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 49:33
  SPEAKER_00: The Early AI podcast is sponsored by Early Information Science and CMS Wire.
• 49:38
  SPEAKER_00: To learn more, visit early.com.
• 49:41
  SPEAKER_00: That is E-A-R-L-E-Y.com.
• 49:45
  SPEAKER_00: Thanks for tuning in.