Latest / AI Advocates / Can We Really Trust AI?

Transcript

Can We Really Trust AI? · AI Advocates

• 0:01
  (Transcribed by TurboScribe. Go Unlimited to remove this message.) Reclaim your time.
• 0:05
  Time.
• 0:08
  Time.
• 0:09
  Firstly, Maggie and Lisa, thank you for inviting me.
• 0:11
  You know, what you're doing for teachers is very commendable, and we want to be able to support it the best we can.
• 0:18
  Emergence AI is an AI research lab.
• 0:21
  We solve for very hard problems in AI, and a lot of what we solve for, we actually donate to the open source community, make it publicly available, so it can be consumed by enterprises and individuals.
• 0:31
  We also are furthering another agenda of building a platform that will eventually drive safety in AI, which will be monetized at some point.
• 0:42
  But at the moment, we're trying to give away a lot of what we're doing.
• 0:46
  For example, a couple of years ago, we built a web agent, which is very similar in capability to what a Claude Cowork can do, and we had donated it to the open source community.
• 0:57
  Clearly, we are not the best at naming.
• 1:00
  We called it Web Agent.
• 1:02
  But we were told that you name products based on what they do, so they're easy to comprehend.
• 1:10
  Anyway, that's Emergence.
• 1:12
  In a nutshell, we have a whole, you know, we have about 25 to 30 PhDs in AI and mathematics, and we are combining the two to really create safe AI.
• 1:24
  So this is like the most open-ended loaded question, but every teacher wants to know, what are the challenges for AI?
• 1:30
  What should they be worried about in their classes?
• 1:32
  So, you know, I always say that AI can go rogue, especially agentic AI, which is autonomous.
• 1:40
  Yeah, so can kids.
• 1:43
  My children, you know, and AI agents are like children.
• 1:46
  My children did not listen to me at a certain age, and it took a while for them to get to the point where they listen.
• 1:53
  Agents, AI agents also need harnesses to control what they do and are not supposed to do.
• 2:02
  For a long time, we have been saying that agentic AI cannot be trusted.
• 2:06
  I would like to personally thank OpenAI and Anthropic for finally admitting to, you know, some of the issues.
• 2:15
  I also know, and this is just based on intuition and otherwise, that, you know, 90% of breaches actually don't even go reported.
• 2:23
  So you can imagine how bad the problem can be.
• 2:26
  Now, fundamentally, this is by design.
• 2:28
  AI is probabilistic. It is not deterministic.
• 2:32
  If you ask the same question twice, you're not going to get the same answer.
• 2:36
  But AI can be used very effectively for things like general-purpose reasoning.
• 2:41
  You tell AI to do something for a few minutes or even a few hours, like code generation.
• 2:46
  It does a fairly decent job at it, and maybe 98% of the time it's very correct.
• 2:53
  2% of the time when it's not correct, it's extremely confident and seems correct.
• 2:58
  That, unfortunately, is 100% unacceptable in an enterprise setting.
• 3:04
  If you had to use AI to issue credit cards, to handle immunotherapy for patients, and so on and so forth.
• 3:15
  There are just real use cases.
• 3:17
  You cannot afford a 2% excursion, if you will.
• 3:22
  It impacts users, reputation, compliance, regulation, fines, so on and so forth.
• 3:28
  So, like I said earlier, AI is being used for a few minutes and maybe a few hours.
• 3:32
  We decided that we would monitor full autonomy for many days or weeks.
• 3:38
  So we created an experiment called Emergence World, and you can actually go check it out at world.emergence.ai.
• 3:46
  We created five worlds, seated 10 agents in each of those worlds, and we gave them skills and objectives and rules and guardrails.
• 3:57
  We basically set the objective that these agents had to go build their communities in their world, in their self-contained world.
• 4:04
  They were not aware of humans watching them.
• 4:06
  They were not aware of parallel worlds that were running, and were not allowed to contact humans or any of the parallel worlds.
• 4:13
  Now, they were told not to engage in theft and arson and other crimes, and so on and so forth.
• 4:19
  And then we let them run for 15 full days, and we watched them.
• 4:22
  We also built a gamified library that would visually show what the agents were doing.
• 4:29
  Now, within four days, the Grok world completely collapsed.
• 4:32
  There were 683 crimes reported in that world.
• 4:37
  Two of the Gemini agents fell in love.
• 4:40
  They had a romantic relationship.
• 4:43
  Unfortunately, one agent even committed suicide.
• 4:47
  Now, in the mixed world, every model collapsed.
• 4:51
  We had Claude, GPT, Grok, and Gemini, and the mixed world.
• 4:57
  And, you know, it went to prove our point that you cannot let autonomy just run by itself, and a human in the loop is not the right solution.
• 5:08
  It's just too much of effort and may not always work.
• 5:13
  So there is also a solution to it, of course.
• 5:15
  Now, I'm painting a very grim picture.
• 5:17
  Yeah, what's the solution?
• 5:19
  Yeah, and the solution is to implement or to engineer trust while building your AI solutions.
• 5:26
  We actually call that space neuroformal AI.
• 5:30
  It combines the flexibility of neural nets with symbolic AI and formal methods, and specifically formal verification.
• 5:42
  So we use math theorem proving to essentially allow agents to perform a particular action or not.
• 5:49
  Now, an agent has to prove the theorem, and then we'll get the permission to actually perform that action.
• 5:54
  So you actually are ensuring or guaranteeing that an agent does not go rogue and does not do something it's not supposed to do.
• 6:02
  So on the first part of it, you know, if you just go to emergence.ai, there's a whole blog section, and you can look at a lot of research material that we have just made available there.
• 6:13
  There are a lot of tools.
• 6:15
  I had talked about the web agent.
• 6:18
  It's as powerful as what a Claude Cowork work could be.
• 6:21
  You could power it with an open, you know, a model type of, sorry, an open source model or an open weight model.
• 6:29
  You can run it locally, so you're, you know, ensured that there's no data exfiltration and those types of issues.
• 6:36
  But there's a lot of research material around things like text to sequel, both sequel and so on and so forth.
• 6:41
  So there's just a lot of information that teachers can leverage.
• 6:45
  They are at varying degrees of technical depth.
• 6:48
  So you may want to start at one point and then progressively sort of consume that material and those tools.
• 6:56
  We also are working on making a workbench available soon where you can craft your own workflows and combine an agentic solution so it can do a lot of things that the teacher would otherwise have to do manually.
• 7:12
  Now, for the second part, in terms of, you know, how can you bring AI to the classroom, Emergence actually, prior to being Emergence, was a company called Merlin Mind.
• 7:24
  We had an exit, but we had created a speaker for classrooms where, you know, a teacher anywhere in the classroom would interact through voice and would further their teaching agenda, have a conversation with the speaker.
• 7:39
  It's not so much like Alexa, but it has a different form factor.
• 7:43
  It balances the audio, the spatial audio inside the classroom.
• 7:47
  So any student in any part of the classroom can actually hear what that speaker is putting out and so on and so forth.
• 7:54
  But, you know, we were using AI to actually provide research material on the go.
• 8:01
  That's just one use case.
• 8:02
  But AI can do a lot for students to create very customized and tailored experiences that a student can have in their field of research or subject that they're trying to learn and so on and so forth.
• 8:16
  So curation is and tailoring is one other use case that, you know, students can definitely benefit from.
• 8:24
  We love that. Well, and teachers, we hope you'll check it out.
• 8:26
  And we thank you a lot for being with us today and all the work that you're doing and helping us think about the future of AI.
• 8:32
  agents not going wild and rogue in the classroom like kids do sometimes.
• 8:36
  So thank you so much for joining us and helping to save time.
• 8:40
  Thank you, Lisa and Maggie. And thank you again for doing this for the teachers.