Latest / Machine Dreams / Everyone's Worried About AI Hallucinations. This CEO Says the Silent Failures Are Worse
Transcript
- speaker-0: It's always exciting to come around and talk about the exciting, bold, fierce world of AI. This is the show where we get insights from the people actually building, securing, and deploying AI. They tell us about what's working, what's broken, and what comes next. My name is Sam Atdebayo. I'm a technology analyst covering AI, cybersecurity, and enterprise technology. And as always, I'm joined by the one and only. speaker-1: Hey everyone, I'm Leah Stern, communications strategist and former venture capitalist. Hey Sam, I missed you. How have you been? speaker-0: okay, I missed you even much more. it's good to see you again. speaker-1: Good to see you, man. Okay, so here's a problem that doesn't get talked about enough. When you ask an AI system a question about your company's data, it goes and looks something up before it even answers. Most of the time that works, but also many times the AI can get things wrong and hand you an answer that looks completely reasonable, but it's completely false. And usually nothing catches it. But our guest, Wei Mo Lu, CEO and co-founder of Puppy Graph has something to say about that. Waymo worked on query systems at Google and spent time at TigerGraph, and he now helps write the international standard for how these systems get queried. Puppy Graph's approach is to check the question before the AI runs it rather than hoping the answer comes out right. Waymo, welcome to Machine Dreams. It's great to have you. speaker-2: Yeah, thank you so much for the opportunity and very excited to share some insights we recently have in the past year. Especially since last September the agent became super popular. speaker-1: Congrats. speaker-0: All right, great. this is quite a a a topic that's very interesting to me because every now and then I I read stuff about AI models making things up, AI chatbots, confidently spew answers that are false, but they're just very confident engines. And so the question is, why does this happen? Wait a moment. speaker-2: Yeah, I think this I think we just founded a company for the agetic moment, like s IT twenty twenty three and some of our friends are working in the front end labs and share some insights and no one will write SQL in the future and agent can do all the data analytics. And at that time we decided to design product not for human beings but for agents. So that's why we came up with public graph. Because we want agent can serving real time limited data. At the same time, the most important is the ontology, which is the semantic layer that the AI understand what data you have and the what relation between the data entity. And since I think since the la beginning of last year we worked with some of a potential customer like AMD, working on some Atlantic system, but it's very hard at the beginning. But after anthropic Sonet four point six, everything works. Before that it cannot repeatable generate the correct query. But now, after you do the correct content engineering and the sta the stability and the performance is very good. speaker-1: Well, that's the technical side of things. I'm going to ask a practical question, which is puppy graph. How did you come up with the name? What does it mean? speaker-2: sis we have a puppy in our team and also puppy graph is something cute and what they are doing, so puppy graph. But most important puppygraph dot com only costs us three dollars. So this is speaker-1: That's a great reason to buy a domain. speaker-0: So, a lot of people seem to be working on on this problem of agents, you know, giving models, chatbots, giving wrong answers. And especially now that we're moving into the world of agentic where more and more AI systems there is there is like that push for more and more AI systems to start to take decisions on behalf of human beings. Which means that the bar has to be high. You can't you know, you can these agents cannot make any mistakes, but we're still battling with hallucinations and problems like that, what is gonna take us from where we are right now to a world is it is it ever gonna be possible where we have a world where none of these problems exist? speaker-2: Yeah, since for our since we only focus on one problem is that enterprise already have data and then let the agent query the correct data. This we're trying to align with the the agent harness concept, which means the learner model can do a lot of things, but you need something audit the result before it does something. And for example, if you just run some query And you can always generate a SQL. And but you cannot make sure it's correct or not. Sometimes you draw in the round tables, but it just give you the answer, which is wrong, but there's no error message, no one know what happened. So what our design is that way modeling all the data in different tables, data warehouse, data lake, as a knowledge graph. You don't need to move the data, but logically you have a knowledge graph already, and after that. the AI have a concept of what kind of a data entity you have. Like for example, elementary school, you have teacher, you have student, and also you know that teacher is a teaching student. That's all the AI needs to know. But you don't need to know which table is the teacher, which table is student, how to draw them, and which table is the teacher's salary, or how student regists the class. actually you only need an orograph then all the links between the data entity. at the AI and we are handling how to map the neurograph to your tables with a deterministic fix phase. And then the accuracy can be guaranteed because all the things are deterministic deterministic by public graph engine and the AI just understands what's happening and do the reasoning. This is how we're trying to maximize the capability of AI and at the same time we audit outside the larger model. to make sure the result is accurate. Yeah. speaker-1: I know that Coinbase, I believe, runs some of their fraud detection on this, right? What what were they what were they doing before? What were they missing? And are there any other companies working with your technology that you can share? speaker-2: I think we are the first one come up with this idea since we think we can decouple the semantic layer with the real data, which means the AI only needs to know the semantic and we are handling how to query the data by semantic. And the Coinbase is using our MTP for quite a long time. They never do write a graph query or a SQL query to our engine. They just use the nature language to ask a different question. And also it's not just a one shot translate nature language to some query language. It's more like a share some insights since the learning model know the general knowledge, but they don't know what the data you already have. For example, if they want to do some complex risk and from general knowledge there are already different type of complex risks, but they don't know how to mapping this general knowledge. To your enterprise data. But after we do the ontology and the AI know, there are several kinds of potential risk, like someone transfer money too much, or there are there is a group, they trust money to each other. And larger model can think about different ideas and then try to fetch data to verify: this is potential risk, whether it exists, and this is another potential risk. And after that the agent will automatically generate a report like, this is a several risk you may have. And which is very powerful and the people even don't know how to write SQL, how to write graph query. He just asks the give an order, please generate a report, then the Larrow model just based on knowledge, based on general knowledge to look in every aspects and then fetch the related data and give you different conclusions. And after that it will be a summary generated, which is a a long running identical system. speaker-0: There's something I you talked about, and I just want to quickly ask. I I get the old idea of working or beauty around the data that the enterprise sits on, but doesn't that also present a different kind of challenge in that, you know, you have to work with enterprises who already have data that is ready for what you're doing. Because, you know, if the data is rubbish, so to speak I I don't see any situation where what you're doing, even though it sounds really great, still works for them and then they're able to get the r the right answers to the questions that they are querying their systems. So tell me what you do in such situations. speaker-2: it's it's a complicated question. Different enterprise have a very different situation about their data. Some of them have a we very well organized and the table is pretty clean, and the table name, column name, and the documents are all clear and then it's kind of a l low hang fruit. We can let the agent auto detect what they have in the data and do the data sampling. to verify the if the potential relations between data entities exist or not. And by doing this, we can very easy to have an autograph. Some of a company even build everything within one day. And then they provide the agent to their the data platform can provide the agent to their internal customer team. But some of them is pretty chaos that so maybe they have a thousand of tables, but only three like thirty of them are hot tables which access a lot, but most of them are cold and no one even knows what happened there. And in this case we need the agent with a human in the loop to verify the what happened. And usually the interactive agent and the agent will ask the user question like we believe that these two data are actually the same thing. Do you think it's correct or this user ID and that user ID saves the same one. And do you approve it or not? And by doing this, we're trying to help the customer to organize the tables by the agent itself. And then after that, they can have a solid novel graph schema which shows the ontology of the entity and also the relations correctly. And after that they can provide the agent to their external or internal customer T. but it's not an easy job. Like also there are some unstructured data and we have some partner work together and we make an introduce to each users and then they can handle the unstructured to structure data part and do the like data extraction or entity resolution and after that we can have a clear one. And also some partner we work together on some data quality issue. some of the c companies have a lot of field the data field just now and a lot of value is just missing and we need to make sure before we do the query we need to show a report of the data quality and whether the user wants to do something before they build the agent. speaker-1: Every company right now is being told to put AI in front of their data. What should they be asking that they're not already asking, do you think? speaker-2: it's really depends on the demo phase and the real in production agent. And since if just want to build demo now the Laral model is very powerful, like if you have five tables or even thirty tables and a lot of tactical sequel agents can do it very well. But in real production, like enterprise they may have twenty teams and the different teams have fifty tables. And the more interesting stuff is that most tables are related to each other. For example, the sales team have some table and the marketing has some table. But actually the list is from marketing team and send to sales team. So if you can have a whole picture and have a big graph to unify the all the data, which may be hundreds or thousands of tables, then it will be very useful. And different team can work together. And also we have some very large company. the engineer team is working on a agent for the whole enterprise and the report to their CXO office and then they can have understanding of the entire enterprise. Rather than like maybe sales team have a report and marketing team have a report and all the numbers are good. And then people will blame each other and for the real reason. But now they can have a whole picture from a lease coming in and until close the contract, what happened during the entire process. speaker-0: Waymo, you've told us about ontology, you've talked about semantic the semantic layer, you've talked about the knowledge graph and the sort of system that you're building around enterprise data, which all sounds interesting. but you know, the world of AI is often clouded with a lot of questions and debates and arguments, and it seems like the problems never go away. My last question for you is this. In the next eighteen months, what really gets solved and what are we still gonna be arguing about? speaker-2: Yeah, so what particular risk is that the AI may do too much things but no one can audit and verify the if the result is correct or not. And we already saw this yeah a lot of our friends company, like because AI are writing code and also review the code. And but for the programming is Not that there it's not a very critical problem because you can generate a bunch of test cases trying to c have a full coverage. but assuming if it financial report and then the AI agent will write a financial report and another AI agent will review the financial report. And no one's actually looking at the process, how the things generated. And then you can never know what the correct answer it is. But the numbers are pretty critical and no mistakes are allowed. Then because you know that people are lazy and when you have so powerful tools you don't want to look look in every details. And we also see that it's a very big problem. Since the AI do things too fast and People cannot follow them to for all the details. I think this is a potential risk. speaker-0: So there you have it. Weymoulu, CEO and co founder of Poppy Graph, thank you so much for joining us on Machine Dreams. We appreciate your insights and we know that a lot of our listeners and viewers have taken away a lot of great insights from the conversation. speaker-2: Sacky Soviet. speaker-1: And thank you to everyone listening. If this one got you thinking, share it and we'll see you next time on Machine Dream. speaker-0: What? Like that.