Latest / Journey to the Fringe / Fringey Minis - Should AI's dream of electric sheep?
Transcript
- 0:00I assume you have a mini ready to go. I don't
- 0:02have one ready to go because you didn't tell
- 0:04me how you were feeling today. To be fair, you
- 0:07didn't ask. How are you feeling? Okay, thank
- 0:10you. Yeah, you're welcome. I am feeling like
- 0:14an AI update. Again? The last one we got wasn't
- 0:18really AI, it was robots. It was too, it was
- 0:20too. Don't even call into question what it was.
- 0:24Okay, I only have one more in here. So let's
- 0:26see if it's filed properly. And I feel like it's
- 0:30bordering on news of another category. So this
- 0:33one is from zdnet .com. Never heard of it. Written
- 0:39by Tiernan Ray, who is a senior contributing
- 0:42writer, April 18, 2025. AI has grown beyond -
- 0:49The world of artificial intelligence has recently
- 0:53been preoccupied with advancing generative AI
- 0:56beyond simple tests that AI models easily pass.
- 1:00The fame Turing test has been beaten in some
- 1:03sense, and controversy rages over whether the
- 1:06newest models are being built to gain the benchmark
- 1:09tests that measure performance. The problem,
- 1:12say scholars at Google Deep Mind Unit, is not
- 1:15the tests themselves, but the limited way AI
- 1:18models are developed. The data used to train
- 1:21AI is too restricted and static, and will never
- 1:24propel AI to new and better abilities. In a paper
- 1:27posted by DeepMind last week, part of a forthcoming
- 1:30book by MIT Press, researchers proposed that
- 1:34AI must be allowed to have experiences of a sort
- 1:38interacting with the world to formulate goals
- 1:40based on signals from the environment. Quote,
- 1:43incredible new capabilities will arise once the
- 1:46full potential of experiential learning is harnessed."
- 1:50Write DeepMind scholars David Silver and Richard
- 1:53Sutton in the paper. Welcome to the era of experience.
- 1:56The two scholars are legends in the field. Silver
- 1:59most famously led the research that resulted
- 2:01in AlphaZero, DeepMind's AI model that beat humans
- 2:05in games of chess and Go. What's go? It's a Chinese
- 2:09game that has black and white pieces. It's widely
- 2:13known that it has like basically infinite different
- 2:17setups of how it can play out. So it's supposed
- 2:19to be one of the hardest games to actually map
- 2:21out for a computer. More than chess? Yeah, infinitely
- 2:24more. Oh, weird. It sounds not like something
- 2:27I would want to play. Sounds difficult. I get
- 2:31that. Even though it was fairly vague. Sutton
- 2:35is one of the two Turing Award winner developers
- 2:38of an AI approach called reinforcement learning
- 2:41that Silver and his team used to create AlphaZero.
- 2:44The approach the two scholars advocate builds
- 2:47upon reinforcement learning and the lessons of
- 2:50AlphaZero. It's called STREAMS and is meant to
- 2:53remedy the shortcomings of today's large language
- 2:56models which are developed solely to answer individual
- 2:59human questions. Silver and Sutton suggest that
- 3:02shortly after AlphaZero and its predecessor AlphaGo,
- 3:05first on the scene, generative AI tools such
- 3:08as ChatGDP took the stage and discarded reinforcement
- 3:11learning. That move had benefits and drawbacks.
- 3:15Gen .AI was an important advance because AlphaZero's
- 3:18use of reinforcement learning was restricted
- 3:20to limited applications. The technology couldn't
- 3:23go beyond, quote, full information. end quote
- 3:26games, such as chess, where all the rules are
- 3:29known. Gen. AI models, on the other hand, can
- 3:32handle spontaneous input from humans never before
- 3:35encountered, without explicit rules about how
- 3:38things are supposed to turn out. However, discarding
- 3:41reinforcement learning meant something was lost
- 3:43in this transition, an agent's ability to self
- 3:46-discover in its own knowledge, they write. Instead,
- 3:50they observe that LLMs, which is, let me just
- 3:54remind you. Yeah, good memory. Rely on human
- 3:58prejudgment or what the human wants at the prompt
- 4:01stage. That approach is too limited. They suggest
- 4:04that human judge imposes an impenetrable ceiling
- 4:08on the agent's performance. The agent cannot
- 4:10discover better strategies underappreciated by
- 4:13the human radar. Not only is human judgment an
- 4:17impediment, but the short, clipped nature of
- 4:19prompt interactions never allows the AI model
- 4:22to advance beyond question and answer. I always
- 4:25feel like AI should be in a question and answer
- 4:27phase, shouldn't it? Like it needs a prompt from
- 4:29somebody to move on, lest it be able to take
- 4:32its own steps outside of that question answer
- 4:35dialogue. Yeah, that's exactly what I was just
- 4:38thinking. You would think it would need a prompt
- 4:41in a form where it's on a computer, but what
- 4:43What if you have AI running a robot that's in
- 4:47a race? Where they weren't in what we saw on
- 4:50the mini. However, what if it was AI? What if
- 4:53it could run on its own? Right? Wouldn't it run
- 4:56without a prompt then? I would hope it would
- 4:58still need a prompt. Yeah. Go run this race.
- 5:01And I guess then it would run the race to completion.
- 5:04Yeah. Until it needed, yeah. But no, I just I
- 5:08like the idea of having a prompt for it to respond
- 5:11to as opposed to doing its own thing Yeah, it
- 5:16sounds like we're getting a little too advanced
- 5:18in my mind. Yeah, and that's what the article
- 5:20is about. I'm pretty sure. In the era of human
- 5:23data, language -based AI has largely focused
- 5:26on short interaction episodes. For example, a
- 5:29user asks a question, perhaps after a few thinking
- 5:32steps or tool use interactions, and the agent
- 5:35responds, the researcher writes. The agent aims
- 5:39exclusively for outcomes within the current episode,
- 5:42such as directly answering a user's questions.
- 5:45I am just using quotations willy -nilly in this.
- 5:49I'm very sorry. There should be a lot of quotations
- 5:52in here that I'm just not using, so I hope it's
- 5:55making its way across. I just don't want to say
- 5:58quote. It feels like too much today. I had a
- 6:01bee attack recently. We basically had the deep
- 6:06state trying to stop us. basically. All in all,
- 6:10I'll continue now. There's no memory. There's
- 6:13no continuity between snippets of interaction
- 6:16and prompting. Quote, typically, literate know
- 6:18information carries over from one episode to
- 6:21the next, precluding any adaptation over time,
- 6:24end quote, writes Silver and Sutton. However,
- 6:27in their proposed age of experience, quote, agents
- 6:29will inhabit streams of experience rather than
- 6:32short snippets of interaction, end quote. Silver
- 6:35and Sutton draw an analogy between streams and
- 6:37human Silver and Sutton argue that, quote, today's
- 6:57technology, and quote, it really adds to it when
- 6:59I put the quotes in, is enough to start building
- 7:02streams. In fact, the initial steps along the
- 7:05way can be seen in developments such as web browsing,
- 7:08AI agents, including OpenAI's deep research.
- 7:12quote recently a new wave of prototype agents
- 7:15have started to interact with computers in an
- 7:18even more general matter by using the same interface
- 7:21that humans use to operate a computer and quote
- 7:24they write the browser's agent marks a transition
- 7:27from exclusively human privileged communication
- 7:30to much more autonomous interactions where the
- 7:33agent is able to act independently in the world
- 7:36as AI agents move beyond just web browsing they
- 7:40need a way to interact and learn from the world,
- 7:42Silver and Sutton suggests. Are they suggesting
- 7:45that it evolves? Is that what I'm getting from
- 7:49this? I don't know if... Essentially... They
- 7:52mean evolve. I think they're just meaning that
- 7:54they need to be able to do this. The robots.
- 7:57No, like the companies who own the robot. They
- 8:03propose that the AI agents in streams will learn
- 8:06via the same reinforcement learning principle
- 8:08as AlphaZero. The machine is given a model in
- 8:11the world in which it interacts, akin to a chessboard
- 8:15and a set of rules. As the AI agent explores
- 8:18and takes actions, it receives feedback as rewards.
- 8:22These rewards train the AI model on what is more
- 8:26or less valuable among possible actions in a
- 8:29given circumstance. The world is full of various
- 8:32signals providing those rewards. If the agent
- 8:35is allowed to look for them, Silver and Sutton
- 8:38suggest. Quote, where do rewards come from? If
- 8:41not from human data, once agents become connected
- 8:44to the world through rich action and observation
- 8:47spaces, there will be no shortage of grounded
- 8:49signals to provide a basis for reward. In fact,
- 8:53the world abounds with quantities such as cost,
- 8:56error rates, hunger, productivity, health metrics,
- 9:00climate metrics, profit, sales, exam results,
- 9:03successes. visits, yields, stock, likes, income,
- 9:06pleasure, pain, economic indicators, accuracy,
- 9:09power, distance, speed, efficiency or energy
- 9:12consumption. In addition, there are innumerable
- 9:15additional signals arising from the occurrence
- 9:17of specific events or from features derived from
- 9:21raw sequences of observations and actions." To
- 9:24start the AI agent from a foundation, AI developers
- 9:28might use a world model simulation. The world
- 9:31model lets an AI model make predictions test
- 9:34those predictions in the real world, and then
- 9:36use the reward signals to make the model more
- 9:39realistic. Quote, as the agent continues to interact
- 9:42with the world through its stream of experience,
- 9:44its dynamics model is continually updated to
- 9:48correct any errors in its predictions. End quote,
- 9:50they write. Silver and Sutton still expect humans
- 9:52to have a role in defining goals for which the
- 9:55signals and rewards serve to steer the agent.
- 9:57For example, a user might specify a broad goal
- 10:01such as improve my fitness, and the reward function
- 10:04might return a function of the user's heart rate,
- 10:07sleep duration, and steps taken, or the user
- 10:10might specify a goal of help me learn Spanish,
- 10:12and the reward function could return the user's
- 10:15Spanish exam results. The human feedback becomes
- 10:18the top level goal that all us serves. That's
- 10:22what I was just wondering. That sounds incredibly
- 10:25intrusive. What if it's not getting those rewards?
- 10:30Is that, I don't know anything about how AI is
- 10:33set up. I don't know. I mean, my AI - It's just
- 10:36us living around in the hardware, or sorry, in
- 10:39a warehouse somewhere. As much as I shouldn't
- 10:40be giving away our secrets, my AI would be rewarded
- 10:44by our high level profits on each episode. Which
- 10:48is why we're multi -million dollar enterprise.
- 10:51Yes. Please still give us money. We need it so
- 10:53bad. That is why we do all the episodes such
- 10:58as we do about corporations because we are one.
- 11:01There was a red herring out there so that you
- 11:03guys are off the trail. We really fooled you.
- 11:05We really fooled you. We're trying to throw those
- 11:07fucking orcas off our yachts. Yeah, I don't know.
- 11:13I don't know how AI works. I'm certainly not
- 11:15giving my AI treats. A big thing that I'm worried
- 11:19about with how they were describing this is they're
- 11:21talking about like a lone AI doing this, but
- 11:23that wouldn't be the case. It would be multiple
- 11:25different AIs potentially doing this at the same
- 11:28time, which likely would then mean they're interacting
- 11:31out there without human interaction with each
- 11:33other, which is literally just the dead internet
- 11:36theory. that the internet's just going to be
- 11:38robots at the end. I've never heard this theory.
- 11:41We can do an episode on the dead internet theory
- 11:43at a different time, which is really coming to
- 11:45life. I would really like to do that. This does
- 11:49keep going on a lot. So I hope you get the point.
- 11:52I don't know that I fully wrap my head around
- 11:55this. You guys got 48 hours and then you can
- 11:57just move on with it and just learn about the
- 12:00stuff that we're teaching you on Friday and just
- 12:01forget that this other one exists. I mean, I
- 12:04probably will. Yeah.