Latest / General Magic & The Soul of Tech / Ep. 7 | The Agent Safety Architecture We Abandoned | Rich Miller

Transcript

Ep. 7 | The Agent Safety Architecture We Abandoned | Rich Miller · General Magic & The Soul of Tech

• 0:00
  SPEAKER_0: This is Mike Stern.
• 0:04
  SPEAKER_1: And I'm Tom "Hersh" Hershenson.
• 0:04
  SPEAKER_1: Welcome to General Magic & The Soul of Tech.
• 0:09
  SPEAKER_2: We believed we were building something that would change the world, and in some ways we did.
• 0:14
  SPEAKER_2: In others, we didn't understand what we were setting in motion.
• 0:18
  SPEAKER_2: The story of General Magic isn't over.
• 0:20
  SPEAKER_2: We're still living inside what it started.
• 0:58
  SPEAKER_1: If you're like me, you like email.
• 0:59
  SPEAKER_1: And if you like email, you have Rich Miller, in part, to thank.
• 1:03
  He helped establish the standards and the industry of email,
• 1:05
  coming at it not as a programmer or a computer scientist,
• 1:07
  but from the network architecture and policy side.
• 1:10
  SPEAKER_1: And without further ado, Mike, Take it away.
• 1:20
  SPEAKER_0: Hi, it's nice for all three of us to be together again.
• 1:22
  SPEAKER_0: We all were at General Magic thirty-five years ago, when things were just getting started.
• 1:26
  SPEAKER_0: Rich, what are your fondest memories of being at Magic?
• 1:29
  SPEAKER_0: Not the big picture vision, but just what it was like day to day and what you remember best.
• 1:35
  SPEAKER_3: There are a number of vignettes that come to mind.
• 1:36
  One of the first was,
• 1:37
  of course,
• 1:37
  sitting in a restaurant in Palo Alto with Marc explaining to me what he wanted to do
• 1:39
  with General Magic.
• 1:40
  SPEAKER_3: He explained to me what he wanted to build.
• 1:41
  And I got quite excited about it because this whole notion of the appliance,
• 1:43
  the handheld device,
• 1:44
  kind of looked like what we were used to seeing as a large HP calculator,
• 1:47
  was suddenly going to be a communicator.
• 1:48
  And this was a dream of mine since reading Asimov's foundation and watching,
• 1:51
  in my head the characters using this Encyclopedia Galactica
• 1:52
  with these handheld devices.
• 1:57
  SPEAKER_3: Not too much longer, a big vignette.
• 1:59
  SPEAKER_3: I met you, Mike.
• 2:01
  We had dinner or drinks and I just thought,
• 2:05
  this is an incredibly good group of people that are getting together.
• 2:09
  SPEAKER_3: Next vignette.
• 2:10
  I'm getting a sense of what it would be like to use General Magic's handheld devices
• 2:18
  and I inadvertently stepped on the initial idea Bill and Andy had about the network
• 2:25
  and how it would all work together.
• 2:27
  SPEAKER_3: And then there's this small group of people doing some absolutely amazing things.
• 2:36
  Every week,
• 2:37
  every month,
• 2:38
  something that everybody would have said last month was impossible,
• 2:43
  suddenly getting done.
• 2:44
  SPEAKER_3: At first I thought, wow, we're really lucky.
• 2:47
  SPEAKER_3: And then it started to feel like, no, this is not luck.
• 2:51
  SPEAKER_3: This is something, this is a real process.
• 2:55
  There are other aspects to Magic that were uncomfortable,
• 2:59
  and that was there was a separation between the MagicCap— the device folks— and
• 3:05
  the people that were working with me and with Jim doing the networking and
• 3:10
  the Telescript part of it.
• 3:12
  SPEAKER_3: We had all come as a group, almost entirely from outside of Apple.
• 3:17
  SPEAKER_3: We had a different cadence, we had a different way of building things.
• 3:22
  And there were times when it just felt like we were living on truly opposite sides
• 3:27
  of the building.
• 3:28
  SPEAKER_3: That came across a few times.
• 3:32
  SPEAKER_3: It was hard... hard to get through.
• 4:01
  It sounds like we were replicating the culture impedance between AT&T
• 4:03
  and General Magic within the company as well,
• 4:05
  because you guys were all network nerds and not not Macintosh pirates, right?
• 4:10
  And so had a really different perspective on what software did and how you tested it
• 4:15
  and how you build it and how reliable it needed to be.
• 4:20
  SPEAKER_3: Right.
• 4:20
  SPEAKER_0: Really different ideas, right?
• 4:22
  SPEAKER_3: Exactly.
• 4:22
  I mean,
• 4:23
  when General Magic first was conceived and I first started speaking with Bill, Andy,
• 4:26
  Marc,
• 4:27
  the notion was that there were these devices,
• 4:30
  and yeah,
• 4:30
  there was something in the middle that intermediated,
• 4:34
  but it was more like the telephone system.
• 4:37
  There wasn't a lot of intelligence that was going to be built into their idea
• 4:42
  of how things would work.
• 4:44
  It really wasn't until Jim and I introduced the notion
• 4:47
  of what smart agents might be able to do,
• 4:48
  that the whole tenor of what General Magic was starting to become made a turn
• 4:51
  and said, the network isn't just a bunch of pipes, it becomes a platform.
• 4:54
  It's a place where you build things,
• 4:55
  where stuff happens, where agents actually have agency and do things.
• 5:14
  SPEAKER_0: Let's take a step back.
• 5:14
  You know,
• 5:15
  the story about General Magic most people know is that we tried to invent
• 5:18
  the smartphone too early and failed.
• 5:20
  SPEAKER_0: But that was only half the story as you've hinted.
• 5:23
  SPEAKER_0: The other half was about the nature of the network, its architecture and its capabilities.
• 5:28
  SPEAKER_0: That story is pretty much untold.
• 5:30
  SPEAKER_0: Say a little bit about it and tell us who Jim White was.
• 5:33
  SPEAKER_3: Jim White was a stalwart, a luminary in the whole area of electronic mail and messaging.
• 5:34
  He became one of the handful of real thought leaders in
• 5:36
  the international standardization of and construction of electronic mail.
• 5:37
  It would be the equivalent of somebody a century ago designing
• 5:39
  a modern telephony network and understanding all of the underpinnings and all
• 5:41
  the plumbing.
• 5:41
  SPEAKER_3: That was Jim.
• 5:41
  SPEAKER_3: He kept it all in his head.
• 5:42
  SPEAKER_3: He was extremely emotional about it, although he never showed it.
• 5:44
  SPEAKER_3: But he just truly believed that this was what the world needed.
• 5:48
  A platform,
• 5:49
  a network that wasn't just a bunch of pipes,
• 5:51
  but actually took active role in accomplishing and enabling things.
• 6:24
  SPEAKER_1: I don't want to put words in your mouth, but my sense of Jim, is two things.
• 6:27
  SPEAKER_1: First of all, he was, in a sense, a visionary.
• 6:30
  SPEAKER_1: He understood what the possibilities of the technology were.
• 6:32
  And second of all,
• 6:33
  he was quietly passionate that not only did he have a vision of it,
• 6:35
  but he had kind of an innate insistence that it should be a certain way.
• 6:39
  SPEAKER_1: And until it was a certain way, it just didn't sit right with him.
• 6:41
  SPEAKER_1: Is that the emotionality that
• 6:51
  SPEAKER_3: Yeah, absolutely.
• 6:53
  SPEAKER_1: you're referring to?
• 6:54
  SPEAKER_3: Yeah.
• 6:54
  SPEAKER_3: he could be he could be militant about things being exactly the right way.
• 6:56
  Things were done well and there was a there was a kind of a perfection to him
• 6:58
  that he brought to it that was extremely valuable and very necessary
• 7:00
  when you were trying to do what we had been doing for the past decade prior
• 7:02
  to General Magic,
• 7:03
  which was figure out how all of these little independently grown electronic mail
• 7:05
  and messaging systems could somehow be knitted into something that was
• 7:08
  the moral equivalent of an international single telephony system, but for messaging.
• 7:13
  All of the big players,
• 7:15
  who were all telcos by the way,
• 7:17
  at that time, were trying to do what they could to slow that down.
• 7:22
  SPEAKER_3: They didn't want the interconnection and the interworking.
• 7:26
  SPEAKER_3: So the standardization and commercial interests of the telcos were always a little bit at odds.
• 7:33
  But at the time we started electronic mail messaging,
• 7:37
  the stakes were so low nobody knew anything about it,
• 7:40
  so they let us go on and do our thing.
• 7:43
  SPEAKER_3: Jim and I worked on these standards for about eight years together.
• 7:47
  And in the course of that,
• 7:48
  a lot of very interesting and new kinds of technologies were coming to the fore.
• 7:55
  The whole idea of applications running on a network,
• 7:59
  being enhanced by a network,
• 8:02
  were this was foreign,
• 8:04
  this had been something that was new,
• 8:06
  and this is why the background of what we ended up doing with Telescript was
• 8:11
  so unusual, so out of out of the ordinary.
• 8:41
  SPEAKER_0: All right.
• 8:41
  SPEAKER_0: You've said "Telescript" several times.
• 8:43
  SPEAKER_0: So where did that name come from and what did it point toward?
• 8:47
  SPEAKER_3: First of all, Telescript is an homage to the whole idea of a package being pushed to an engine.
• 8:51
  SPEAKER_3: PostScript had been invented at Xerox PARC while Jim White was working there.
• 9:02
  SPEAKER_1: And just to explain the breakthrough of PostScript.
• 9:04
  SPEAKER_1: Back in the old days, you had to match a computer basically to a specific model of printer.
• 9:08
  SPEAKER_1: And the insight of PostScript is you could have an abstraction.
• 9:11
  You could basically have a language that was a language about telling a printer
• 9:13
  where to put dots on a page.
• 9:15
  As I understand it's a similar thing
• 9:16
  for communication you mentioned sending something to an engine.
• 9:18
  SPEAKER_1: And my understanding is the engine is the same as with PostScript.
• 9:22
  SPEAKER_1: You don't need to know the technical details of the underlying printer.
• 9:24
  SPEAKER_1: The printer understands PostScript.
• 9:25
  SPEAKER_1: Likewise, the network or the computer somewhere on the network understands Telescript.
• 9:30
  SPEAKER_1: And what chips it has or what language it's programmed in or whatever, that doesn't matter.
• 9:33
  SPEAKER_1: You have this
• 9:53
  SPEAKER_3: immaterial.
• 9:53
  SPEAKER_1: so-called in engine on which these Telescript agents will run.
• 9:58
  SPEAKER_3: You're dead on.
• 9:58
  And in fact you've done a real service to how Telescript emerged
• 10:00
  from this very very similar idea.
• 10:02
  SPEAKER_3: The notion that it would be an abstraction.
• 10:04
  SPEAKER_3: There would be an agreement on the rules, the actions that could be taken.
• 10:09
  And wherever one of these programs,
• 10:11
  these packages landed, it would be rendered as well as could be at that site.
• 10:13
  SPEAKER_3: You didn't have to know what kind of computer it was.
• 10:16
  As long as it said,
• 10:17
  "I understand and I speak Telescript," you could send something to it
• 10:20
  and it would do the right thing and that right thing would be the best job of taking
• 10:25
  the intent and making it happen.
• 10:44
  SPEAKER_0: What were the things that Telescript sent across the network to be interpreted and executed?
• 10:49
  SPEAKER_3: Telescript could send pretty much anything.
• 10:50
  SPEAKER_3: It could send email that had different kinds of content in it.
• 10:53
  SPEAKER_3: It could send animations.
• 10:54
  SPEAKER_3: It could also convey to the place where it landed.
• 10:56
  SPEAKER_3: "This is who I am.
• 10:57
  SPEAKER_3: I've got credentials.
• 10:58
  SPEAKER_3: And not only do I have credentials, I've got a credit card.
• 11:02
  SPEAKER_3: I've got a resource that I can use to ask locally for things to happen for me."
• 11:17
  SPEAKER_0: Such as?
• 11:18
  Buy something,
• 11:19
  build something for me,
• 11:20
  make a picture,
• 11:21
  do all of the things that we're now thinking about agents doing on our behalf.
• 11:26
  Perhaps it would search out information,
• 11:29
  collect it,
• 11:30
  and then move on in order to figure out what's the best place to go next week?
• 11:34
  SPEAKER_3: Where are the movies that I'm interested in playing?
• 11:38
  That kind of assistant that would actually take on a goal,
• 11:40
  take on an objective,
• 11:42
  and go to those places that advertised,
• 11:44
  if you want to think of it that way,
• 11:46
  "Hey, this is what kind of information I can provide to you.
• 11:48
  SPEAKER_3: And here are the things that I can make happen for your author, the author of this agent."
• 12:05
  SPEAKER_0: The thing we shipped with in nineteen ninety-four was a travel app.
• 12:10
  SPEAKER_0: Telescript could make make travel reservations.
• 12:12
  You know the way the process worked is you'd create the agent,
• 12:16
  it went to sleep,
• 12:17
  it crossed the network,
• 12:18
  it woke up in an engine,
• 12:20
  and it could,
• 12:20
  you know,
• 12:21
  it could reserve a flight,
• 12:21
  hive itself off to create a another agent that would monitor that flight
• 12:26
  and tell you whether it was on time,
• 12:27
  whether your seat had changed,
• 12:28
  something like that, and come back, and then turn itself off when it was done.
• 12:32
  SPEAKER_3: Exactly.
• 12:32
  Just to put a little bit on that,
• 12:34
  the agent would show up at what would be the kind of virtual travel desk,
• 12:38
  inquire as to what flights were available,
• 12:40
  book a reservation,
• 12:41
  enter into a transaction to reserve the seat or purchase it,
• 12:44
  would leave a small version of itself locally to keep track of what's going
• 12:48
  on regarding the flight that it just booked,
• 12:50
  wake up once a day until it's two days or one day away from the time I'm flying.
• 12:55
  After that,
• 12:56
  now I want you to wake up every hour,
• 12:57
  make sure that the flight is still on time,
• 12:59
  that conditions are right, things haven't been changed.
• 13:01
  If for some reason you find out that things have been changed,
• 13:04
  come on back home,
• 13:05
  tell me about it, and we'll make arrangements to do something else.
• 13:24
  SPEAKER_1: Really the similarities to agentic AI are unmistakable, right?
• 13:26
  SPEAKER_1: I mean, Telescript was a way of having code embody and act upon your goals and intent.
• 13:35
  The big distinction there is the agent didn't have the degrees of freedom
• 13:38
  that we're talking about today with AI.
• 13:40
  SPEAKER_1: Okay.
• 13:41
  SPEAKER_3: At that point, it was very constrained.
• 13:43
  SPEAKER_3: Here are the things that you can ask, here are the things that we can do.
• 13:48
  SPEAKER_3: Here's the menu.
• 13:48
  SPEAKER_3: You can't order off the menu.
• 13:51
  SPEAKER_3: Here's the menu of what you can do.
• 13:54
  SPEAKER_3: And here's what I require in order for you to place that order.
• 13:57
  SPEAKER_3: And it's got to be done this way and only this way.
• 14:01
  SPEAKER_3: You have to supply credentials.
• 14:02
  SPEAKER_3: You have to supply a source of funds if in fact it was a purchase.
• 14:05
  Those are the kinds of things that you would want to do today with an agent with
• 14:10
  that kind of ability to change direction,
• 14:11
  change the rules, negotiate-- that was not a part of Telescript.
• 14:14
  SPEAKER_3: That was not in anybody's wheelhouse at that point.
• 14:26
  But but that was because you were basically trying to prevent both rogue agents
• 14:32
  and rogue actors from being able to go off the menu.
• 14:34
  That kind
• 14:35
  of you know deterministic safety architecture was something you guys thought of from
• 14:38
  the beginning.
• 14:38
  SPEAKER_0: And
• 14:39
  SPEAKER_3: Exactly.
• 14:40
  as we've talked about,
• 14:40
  current LLM-based agentic AI doesn't permit that kind of deterministic safety,
• 14:45
  because the agents can modify themselves.
• 14:49
  So given that we can't build deterministic safety into LLM-based agentic AI,
• 14:54
  what can we do to prevent rogue agents and rogue actors from misusing this stuff?
• 15:02
  SPEAKER_3: Let me let me start first with what we did and why we did it that way in Telescript.
• 15:04
  SPEAKER_3: And part of that was how electronic mail standards got set or didn't.
• 15:05
  And one of the reasons why we have the possibility of someone sending you a piece
• 15:08
  of email that for all intents and purposes looks like it came from Rich Miller,
• 15:10
  but it didn't, is because there were things left out of electronic mail.
• 15:11
  Part of it had to do with identification,
• 15:13
  authentication: Here's who I am, and here's how I prove it.
• 15:15
  And then authorization:
• 15:15
  Here are the things that I am allowed to do,
• 15:17
  and here's the list of things that I am not allowed to do.
• 15:21
  That kind of constraint,
• 15:23
  that kind of fencing,
• 15:25
  was something that we built into Telescript purposefully,
• 15:28
  with the whole idea that if one of these things did get loose,
• 15:33
  it did go rogue,
• 15:34
  if it had no constraints,
• 15:36
  there would be no telecommunications company on the face of the earth
• 15:42
  that provided services that would allow for that.
• 15:45
  SPEAKER_3: It simply would not happen.
• 15:47
  So the whole notion was let's get it operating correctly within these guidelines and
• 15:54
  then over time relax as we became more capable of defining these fences with
• 16:01
  a little bit more wiggle room, a little bit more flexibility.
• 16:05
  What you're talking about today with agentic AI is starting from the other side,
• 16:12
  which is almost no constraint,
• 16:15
  and then trying to put constraints on activity after the fact, once it arrives.
• 16:21
  Once an agent attempts to do something in a in some sort of transaction,
• 16:26
  the whole notion of putting constraints around it just hasn't been built into it,
• 16:32
  hasn't been built into the protocol, into the rules.
• 17:02
  To what extent was building the safeguards in a function of Jim's notion of,
• 17:06
  and I don't like the word perfect,
• 17:07
  I like the word correct,
• 17:09
  or maybe it's just the right way that it's it's it's how it should be done.
• 17:11
  SPEAKER_1: I mean, was that a driving force in it?
• 17:13
  SPEAKER_1: Or was it obvious to you guys that you had to have that?
• 17:16
  SPEAKER_1: What was the genesis of sensitivity to that?
• 17:20
  SPEAKER_3: Yeah.
• 17:20
  SPEAKER_3: That was the fact that you had several alumni from the standards group.
• 17:23
  You had Jim, me,
• 17:24
  Doug Steedman,
• 17:25
  numbers of people on the Telescript team who had been in the electronic mail,
• 17:29
  industry for quite a long time,
• 17:31
  and all of us were pretty aware of what the failings of electronic mail were,
• 17:34
  and were likely to become.
• 17:36
  Telescript started with the notion of let's put the fence around it and
• 17:40
  then relax it over time, not the other way around.
• 17:57
  SPEAKER_0: To put it another way, you guys didn't foresee spam when you're doing electronic mail?
• 18:00
  So when it came time to be developing agents,
• 18:03
  what was going to be the equivalent of spam and how did you structure the network
• 18:06
  so you there wouldn't be any?
• 18:08
  SPEAKER_0: Is that a fair analogy?
• 18:11
  SPEAKER_3: Well, parts of it.
• 18:12
  and what you may not know is that one of my contributions while I was an architect,
• 18:17
  kind of developing the first electronic mail systems protocols for ARPA in
• 18:20
  the late 70s was figuring out how to send the same message to a large list
• 18:24
  of recipients in an efficient manner so that it wasn't one envelope
• 18:27
  with one message going to everybody that was on the list.
• 18:30
  The message would come along with this list and it would be a circuit writer and
• 18:34
  as it arrived at a place where,
• 18:36
  "Yeah here are a bunch of people locally that are on the list;
• 18:40
  I'll make a copy of myself there and then move on."
• 18:42
  SPEAKER_3: The point being that yeah I will probably go to hell for the fact that I did that.
• 19:04
  SPEAKER_0: You're the architect of spam!
• 19:05
  SPEAKER_0: That's what you're telling.
• 19:06
  SPEAKER_3: I'm an enabler.
• 19:06
  SPEAKER_3: I It's not something I'm particularly proud of now, after fact.
• 19:09
  But the idea of putting constraints,
• 19:11
  there were lots of rules that we wanted to make sure that were in there.
• 19:14
  Big part of it had to do with identity,
• 19:16
  the agents and the ownership of those agents,
• 19:18
  the responsible party for those agents,
• 19:19
  had to be extremely clear and if there was any doubt,
• 19:22
  kicked them out and said, come back when you've got all your credentials.
• 19:35
  You know the good guys in the AI arms race,
• 19:37
  supposedly like Anthropic,
• 19:38
  rather than talking about building in structural guardrails,
• 19:41
  which there are technical reasons why that can't necessarily be done that way.
• 19:44
  SPEAKER_0: You can't retrofit the car with seat belts, unfortunately.
• 19:47
  SPEAKER_0: That the way to do it is to have a set of principles that the agents are trained to comply with.
• 19:51
  SPEAKER_0: They call it "constitutional AI," and Anthropic has in fact published an AI constitution.
• 19:55
  The problem that I see with that is that in a constitutional system like
• 20:00
  the American one,
• 20:01
  there is a decider ultimately who interprets the Constitution
• 20:04
  and tells us what it means as things change and it needs to be applied
• 20:08
  to new conditions.
• 20:09
  SPEAKER_0: There's no decider in the industry notion of constitutional AI.
• 20:12
  SPEAKER_0: So to me that means it can't work.
• 20:13
  SPEAKER_0: But you're the expert, so tell us.
• 20:28
  Well, because I'm pretty involved in today's use of AI and agents to do agentic AI
• 20:30
  and agentic commerce,
• 20:31
  let me suggest that if you want to stick with the notion of a constitution,
• 20:33
  a set of principles,
• 20:33
  there is the notion that the actors involved,
• 20:34
  the agents, are not hidebound by those principles.
• 20:36
  SPEAKER_3: In other words, they learn, they're interpreted.
• 20:38
  SPEAKER_3: Context means a lot.
• 20:38
  So they are as close to guidelines as you can possibly get that are trying to direct
• 20:41
  the learned behavior, the preferred behavior.
• 20:44
  So rather than have a constitution where you have something that adjudicates,
• 20:49
  these things now look more like a form of alignment that you would find in
• 20:55
  the establishment of a treaty.
• 20:57
  SPEAKER_3: There is no international court really that you know sits above all the treaties.
• 21:03
  Now there are treaty organizations that they try to put in place
• 21:06
  that are doing some adjudication,
• 21:09
  but they're general,
• 21:11
  they don't have the same kind of teeth that a you know a constitution with
• 21:15
  a Congress has.
• 21:16
  SPEAKER_3: The approach here is going to be one of alignment.
• 21:20
  It's as I've said to you before,
• 21:22
  these are not, they're not sentient beings, but they they do learn.
• 21:26
  SPEAKER_3: The agents do learn, they are trainable, they can adopt and operate on some abstract principles.
• 21:35
  SPEAKER_3: At least they seem to be doing that.
• 21:37
  And the great hope,
• 21:40
  and it's not it's not a foregone conclusion we can do it,
• 21:43
  but the great hope is that there's a path to alignment of such that there are things
• 21:49
  that are as close to an ethical behavior and an appropriate behavior
• 21:55
  that you wouldn't go beyond those bounds unless there was something so important or
• 22:00
  so critical that you had to.
• 22:03
  SPEAKER_3: And in a way that has to kind of mirror the way humans interact with one another in societies.
• 22:09
  SPEAKER_3: And that I think is the principle that they're trying to get to.
• 22:13
  SPEAKER_3: And "constitution" is as close as they can come right now.
• 22:44
  SPEAKER_0: You know, we have no Supreme Court in the AI world.
• 22:47
  Okay, but so,
• 22:47
  how likely is it that an OpenAI agent that's being used for surveillance,
• 22:52
  or being used in a drone,
• 22:53
  is going to act ethically in the sense of not making a kill decision without a human
• 22:57
  in the loop, etcetera.
• 23:01
  I have to stand with Anthropic on that very point,
• 23:02
  which is you can't let this be done without a human in the loop.
• 23:05
  The notion of complete autonomy of the agents is in my opinion,
• 23:08
  a recipe for a disaster and how big a disaster is kind of the only question.
• 23:24
  But Rich,
• 23:24
  let me just take a step back because it seems to me that one of the contrasts
• 23:27
  between the AI industry and the development of Telescript was
• 23:29
  the competitive environment.
• 23:31
  SPEAKER_1: My understanding was Telescript was really novel.
• 23:33
  SPEAKER_1: There was no one else who was doing it.
• 23:34
  SPEAKER_1: And certainly we had at General Magic the comfort of not having anyone breathing down our neck.
• 23:37
  SPEAKER_1: And so there was time to be deliberative about it, and intentional, and building the safeguards.
• 23:42
  SPEAKER_1: Whereas AI hasn't evolved in in that way.
• 23:43
  SPEAKER_1: Everyone is trying to climb to the top of the heap.
• 23:45
  And you know,
• 23:45
  the saying that ethics are a luxury,
• 23:47
  that's going to be one of the first things to go in terms of safeguards.
• 23:53
  SPEAKER_1: What are your thoughts on how the funding model?
• 23:56
  SPEAKER_1: shapes the kind of ethical decisions made around technology development?
• 24:10
  Let me take one of your early points and then I'll I'll hopefully get to the meat of
• 24:13
  the question you just asked.
• 24:13
  When we decided that what had to happen was that the network had to be part of the
• 24:17
  the active delivery of all of these services, that the network had to be a platform.
• 24:22
  SPEAKER_3: And we took it to companies like AT&T.
• 24:23
  Both in terms of funding and behavior,
• 24:24
  AT&T was absolutely a beast about... this can't make that kind of a mistake.
• 24:27
  SPEAKER_3: We will not allow that kind of transaction to be done using our network.
• 24:31
  It has to be as safe and as guaranteed in terms of its behavior,
• 24:37
  as deterministic in its behavior as it can possibly be.
• 24:41
  And consistently they pushed back on this is what we will allow,
• 24:47
  this is what we will not live with.
• 24:50
  And along with that came their interest in and their willingness not just to invest
• 24:56
  in General Magic,
• 24:57
  not just to commit to a revenue model in
• 25:01
  which they would do something almost unheard of,
• 25:04
  which was to take third-party software and put it into the core of
• 25:10
  a network offering, which up until that point, always had to be built by AT&T.
• 25:16
  They then invested in a management environment that would monitor this to make sure
• 25:24
  that its quality and quality covered a broad spectrum of issues.
• 25:29
  So in this particular case,
• 25:31
  the platform,
• 25:32
  the delivery mechanism,
• 25:34
  the funding,
• 25:36
  its ongoing operation,
• 25:37
  were all coming from an existing institution or a set of institutions,
• 25:42
  which already,
• 25:43
  by the way,
• 25:43
  had as a principle very high standards with respect to service levels
• 25:49
  that they would meet,
• 25:51
  and high levels of interoperation with their peers in other countries.
• 25:56
  So you're talking about such a different environment into which we were introducing
• 26:02
  the whole notion.
• 26:03
  SPEAKER_3: And what we were getting back was clear message: it's gotta be safe.
• 26:09
  SPEAKER_3: and especially if we're gonna try to make this a consumer-oriented offering.
• 26:14
  SPEAKER_3: That often was hard to take in parts of General Magic.
• 26:19
  SPEAKER_3: But arguably I think it was the right way to do it.
• 26:22
  So now the place where you went with this is,
• 26:26
  how would you characterize the situation we live in today with agents?
• 26:31
  SPEAKER_3: Who or what is the stakeholder?
• 26:34
  SPEAKER_3: Where is the source of funding?
• 26:36
  SPEAKER_3: How does that all work?
• 27:04
  SPEAKER_1: Yeah, well it's it's a I mean the phrase Wild West is overused, but it really is a free-for-all.
• 27:07
  SPEAKER_1: I mean, you don't have the notion of a single organization hosting the technology.
• 27:11
  SPEAKER_1: You don't have a controlled development process.
• 27:13
  SPEAKER_1: It's a scramble among dozens, if not hundreds, of companies.
• 27:17
  SPEAKER_1: There's a strong economic end motive to be first, not to be best, not to be proper or correct.
• 27:22
  And, you know,
• 27:23
  there are differences in the economic and business context in which any sort
• 27:28
  of innovation occurs,
• 27:30
  but it's part of the thesis that Mike and I are exploring is the loss of any sort
• 27:34
  of notion of common good and the loss of almost any sort of ethos except you know,
• 27:39
  a money grab and let me be the wealthiest person as possible,
• 27:42
  and the distortions that it has on the development of technologies.
• 27:57
  you said the magic words as far as I'm concerned,
• 27:58
  and it's the term "common good" and the notion of contribution to in the consumption
• 28:06
  of these technologies has to be a lot more in the open and unfettered.
• 28:11
  What I would point you to,
• 28:12
  and we're already seeing it,
• 28:14
  is the adoption of open weights, open source technologies.
• 28:19
  SPEAKER_3: They're not subject to the laws, because there aren't any, there's not much regulation.
• 28:24
  But the adoption of a broader set of participants to the training of AIs,
• 28:29
  to the selection of data that goes into them,
• 28:31
  to the governance of the data that goes into them,
• 28:34
  which is a big issue and will get in my mind us a great deal further along to
• 28:37
  an ethical behavior.
• 28:51
  SPEAKER_0: Yeah, but Rich, I mean, there are two things.
• 28:52
  SPEAKER_0: One is one can really disagree about what's ethical.
• 28:55
  SPEAKER_0: just just take the issue of what's patriotic.
• 28:57
  I think the Googles,
• 28:58
  the Microsofts,
• 28:58
  and the OpenAIs take the position that to be patriotic,
• 29:01
  they need to support the goals of the American government and the American military,
• 29:04
  and that providing technology that will buttress the strengths of
• 29:07
  the American military is ethical.
• 29:09
  whereas there are people on the other side who say it's not ethical
• 29:13
  to create autonomous weapons,
• 29:15
  it's it's not ethical to contribute to mass surveillance,
• 29:17
  and to surveillance capitalism in general,
• 29:18
  and to the potential violation of civil liberties that AI assisted surveillance
• 29:34
  SPEAKER_3: Mm-hmm.
• 29:35
  SPEAKER_0: makes possible.
• 29:35
  SPEAKER_0: That's what can't get adjudicated and in which markets fail, in regulating.
• 29:40
  SPEAKER_0: So, you know, that's where I'm stymied on this notion that alignment is gonna happen.
• 29:45
  Well, let's put it this way there has to be some form of governance,
• 29:47
  you know,
• 29:47
  and what we have now that has any serious impact on the way AI is developed,
• 29:50
  what it is capable of or what it is allowed to do is lagging so far behind.
• 29:52
  The speed with which all of this has happened has
• 29:54
  so outstripped any legal system's ability to keep track and keep up with it.
• 29:56
  If you're going to ask for some form of governance,
• 29:57
  and I think realistically not accept self-governance as being
• 29:58
  an answer you're willing to accept,
• 29:59
  then there's got to be some forum,
• 30:02
  some agreement, some treaty that manages and by agreement the approach to it.
• 30:09
  And if you want to take nuclear armament treaties as an example of something that,
• 30:14
  well,
• 30:14
  I guess you can say it kind of worked because we haven't blown ourselves up yet.
• 30:19
  it's not that it isn't under a lot of pressure and you know there's there's not
• 30:25
  a lot of assurance as to its longevity.
• 30:29
  SPEAKER_3: But it did make a difference.
• 30:32
  SPEAKER_3: It was
• 30:59
  SPEAKER_0: Mm-hmm.
• 30:59
  SPEAKER_3: an agreement among sovereign nations.
• 31:00
  SPEAKER_3: There was a notion of sovereignty that has been roughly accepted.
• 31:04
  SPEAKER_3: The point being any successful approach to this for AI has got to be based on something similar.
• 31:09
  And that means sovereign nations with an agreement on the issues and the kinds
• 31:13
  of protections and enough teeth in it that there are repercussions if
• 31:16
  the treaty is not upheld.
• 31:29
  SPEAKER_0: Right.
• 31:29
  So governance comes from the government and you know,
• 31:32
  so there have to be two levels of government intervention,
• 31:34
  national and supranational.
• 31:36
  SPEAKER_0: and the question is whether
• 31:37
  SPEAKER_3: This is supranational.
• 31:38
  SPEAKER_0: is whether treaty organizations are the right supranational.
• 31:39
  Okay, the second piece was to talk about an analogy:
• 31:41
  You know,
• 31:41
  when you buy a hot dog in the store,
• 31:42
  you the price does not reflect all of the externalities
• 31:44
  that having hot dogs inflicts on a society.
• 31:47
  We're seeing some of the externalities of the AI arms race being inflicted
• 31:51
  on society at large,
• 31:51
  this notion of the lack of a sense of common good by data centers,
• 31:54
  and the resistance to data centers,
• 31:57
  in terms of their environmental impact,
• 32:00
  their impact on electricity prices,
• 32:02
  their impact on the water table, and all the rest of that.
• 32:05
  SPEAKER_0: What can we do to handle the externalities of the the race for compute?
• 32:17
  Well, once again it's gonna require government,
• 32:18
  but what it comes down to is the total cost of providing AI has got to be taken
• 32:20
  into account.
• 32:20
  And you and I both know that downstream a lot of the cost is borne by the taxpayer,
• 32:22
  by consumers,
• 32:23
  by everybody downstream,
• 32:23
  with no compensation from the originators, the perpetrators, the sellers.
• 32:26
  SPEAKER_3: Part of it is the notion of having a commons and a common good.
• 32:29
  SPEAKER_3: And it's taxation.
• 32:29
  I mean, hell,
• 32:30
  large language models being trained on the world's,
• 32:34
  the internet's complete set of material,
• 32:38
  including publications that are rightfully concerned about the fact
• 32:43
  that these companies got basically a free ride to pick up all of the information
• 32:49
  that they had put out, made available on the net by one means or another.
• 32:54
  SPEAKER_3: And It gets covered by a law that says, it's fair use.
• 32:57
  SPEAKER_3: So go ahead, train your AI, you don't owe me anything.
• 33:01
  SPEAKER_3: No.
• 33:02
  This is this is once again you're you're absolutely right about the fact
• 33:06
  that it requires the notion of a common good or common will.
• 33:12
  SPEAKER_3: It does require a sense of and an understanding of who's paying for all of it.
• 33:18
  And by paying I mean paying for the outcome,
• 33:22
  paying for the parts of it that are not contributory but rather destructive or
• 33:28
  at least consuming.
• 33:29
  Those have to be taken into account and it's it comes down to a government
• 33:34
  that says,
• 33:35
  "This is how we govern it and this is how we tax these companies
• 33:39
  that are getting enormous piles of money thrown at them."
• 34:10
  SPEAKER_0: Yeah, I mean the equivalent of a carbon tax is a token tax.
• 34:13
  SPEAKER_0: And you know, people like Bernie Sanders and Elizabeth Warren are talking about those.
• 34:17
  SPEAKER_0: But in the current environment those ideas have no traction, unfortunately.
• 34:21
  SPEAKER_0: So we'll see we'll see how far we get.
• 34:25
  Mike, you and I have wrestled over how to analogize AI to other innovations
• 34:31
  that there were regulatory and statutory responses to.
• 34:35
  I certainly don't have any answers,
• 34:37
  but I would say that there's a notion of not letting the perfect be the enemy of
• 34:41
  the good.
• 34:41
  I mean, your patriotism example is well taken,
• 34:44
  but I think that you can find,
• 34:46
  or I hope,
• 34:46
  there's common ground to be found and to triage the issues so that
• 34:48
  for example there might be agreement that patriotism means something
• 34:52
  that you get 90% buy-in about and do something about that.
• 34:57
  You have you know profound disagreements on certain issues,
• 35:01
  but it seems like an incremental approach is the best that we have at this point.
• 35:12
  Yeah, you're not gonna be able to get to perfection,
• 35:15
  you know,
• 35:16
  out of the box One of the areas that I think is probably most ripe
• 35:20
  for both taxation,
• 35:20
  ownership,
• 35:20
  contribution to the common good is the whole governance of the data used by AI.
• 35:23
  It is, you know,
• 35:23
  everything from personal data,
• 35:24
  data being generated as a result of research, all of these things.
• 35:28
  SPEAKER_3: it's a weird good, has you know the economics of information, but it is the lifeblood of AI.
• 35:46
  SPEAKER_0: That's what you're working on now.
• 35:48
  SPEAKER_0: That's what Provence Data does, right?
• 35:50
  Tell us a little bit about,
• 35:52
  you know,
• 35:52
  where you've come since the General Magic experience and what you're doing
• 35:56
  to further your through line in in in terms of creating standards
• 36:00
  and accountability.
• 36:01
  It's my assessment that the way in which AI is mostly done today,
• 36:03
  which are these large language models, are going to hit a ceiling.
• 36:07
  SPEAKER_3: They're going to hit a level that beyond which they can't go.
• 36:12
  And that in point of fact,
• 36:14
  even if we restrict this to the written word or the interpretation of something
• 36:20
  that gets rendered as a written word,
• 36:23
  without having a sense of what's a fact and what is the lineage of that fact,
• 36:29
  or the history of that fact?
• 36:31
  Not just how did it change or how did the operative information change,
• 36:34
  but who made that change and what was the decision process.
• 36:38
  If you can create those traces,
• 36:40
  if you can keep that in mind,
• 36:41
  you're probably going to come closer to this ideal agent.
• 36:46
  Simply because there's memory and there's also a way of kind
• 36:50
  of protecting its sovereignty if you want to think about it that way.
• 37:00
  SPEAKER_0: This is like block blockchain for data.
• 37:02
  SPEAKER_0: Is that what you're saying?
• 37:04
  SPEAKER_3: Ooh, I hope not.
• 37:04
  but in a way it is — it's the notion that if you're going to make use of data
• 37:08
  for AI, it has to come with its lineage, it has to come with its provenance.
• 37:12
  SPEAKER_3: If it does not, you're at risk.
• 37:13
  You're at risk for fraud,
• 37:14
  you're at risk for mistakes that are made that you can't go back and find,
• 37:17
  and repair and fix.
• 37:18
  If what we are now living with is structured data in kind of
• 37:22
  a conventional database management system that for all intents and purposes forgets
• 37:26
  the past.
• 37:26
  SPEAKER_3: It's this is where this is where things are now.
• 37:28
  Doesn't retain and doesn't utilize the history of it and the provenance and
• 37:32
  the decision tracing,
• 37:33
  then we continue to be at risk and the AIs will continue to hallucinate and
• 37:37
  so forth.
• 37:57
  SPEAKER_0: So yeah, so how how how are you doing it and who are you selling it to?
• 38:00
  SPEAKER_3: Well, I'm not selling it yet, but
• 38:02
  SPEAKER_0: Hmm.
• 38:03
  SPEAKER_3: how am I doing it?
• 38:03
  SPEAKER_3: There are what most of the database people would think of as being kind of oddities.
• 38:05
  There have been strange and odd databases that have you know tried to make their way
• 38:07
  into the mainstream.
• 38:07
  SPEAKER_3: They basically haven't done a great job of getting there.
• 38:09
  But what I think now is the right way to do it is...
• 38:10
  we're at the point where the cost of cryptographically anchoring data— not so much
• 38:13
  to protect it necessarily from being changed— but to be able to determine instantly
• 38:15
  whether it's been tampered with,
• 38:16
  whether it's been associated with a source that shouldn't be.
• 38:22
  SPEAKER_3: It goes back to my comments about identity and authentication, so forth.
• 38:28
  You don't want me to dive into this rabbit hole,
• 38:30
  but I will I will tell you that in a microcosm,
• 38:34
  when you keep this kind of data and retain it...
• 38:38
  when you're talking about a rather complex process like making semiconductors
• 38:44
  or running an electric utility grid,
• 38:47
  the ability to find problems,
• 38:50
  to address them quickly,
• 38:52
  to forecast,
• 38:54
  if not predict,
• 38:55
  forecast possible outcomes that you can prepare for in advance is changed radically.
• 39:03
  SPEAKER_3: The productivity of a of a shop floor.
• 39:06
  SPEAKER_3: jumps amazingly when you have this kind of data managing and overseeing the processes.
• 39:13
  SPEAKER_3: I'm pretty certain there's there's a pony in here.
• 39:17
  SPEAKER_3: I'm not advocating getting paid by the data brokers for using my data.
• 39:23
  SPEAKER_3: We went around and looked at the money that they're making off of me.
• 39:27
  SPEAKER_3: I might get fifty bucks a year for the use of my data.
• 39:58
  SPEAKER_0: Sort of like what Spotify pays artists, you know.
• 40:01
  SPEAKER_3: But let me just try something on you.
• 40:02
  If that $50,
• 40:02
  you know,
• 40:03
  rather than paying me,
• 40:03
  were collected around all of the people for whom they were gathering data,
• 40:06
  and that $50 went into something that supported the common good,
• 40:08
  that it did in fact work towards and paid the price,
• 40:09
  or paid down some of the real costs that were incurred from a Spotify or,
• 40:11
  you know,
• 40:11
  a utility company,
• 40:12
  a marketing company,
• 40:12
  then there's a place where I think there's an enormous amount of traction.
• 40:15
  It's one of the major reasons why you have governments:
• 40:16
  to pay for the things that are for the common good.
• 40:18
  There are so many parts of this puzzle that need to be juggled and you know it's
• 40:20
  the plate spinning aspect of this that is just daunting.
• 40:24
  SPEAKER_3: But it does need to be addressed.
• 40:26
  SPEAKER_3: It needs to be organized in some ways.
• 40:28
  And if you have a couple of tools that get you,
• 40:31
  part way there,
• 40:32
  it can improve things,
• 40:33
  it can possibly satisfy certain aspects of it,
• 40:37
  it just not gonna you know be the total answer.
• 41:05
  SPEAKER_0: I mean, some people some some people are on the same page with you already.
• 41:08
  I heard a talk by somebody at the Gladstone Institute,
• 41:10
  the neurobiological research place, and they're
• 41:13
  SPEAKER_3: Right.
• 41:14
  building their own custom databases with data that is exactly the way you,
• 41:16
  you know, it's traceable, it's monitored, it's attributable and accountable.
• 41:21
  They don't just scrape stuff from the internet,
• 41:23
  they build their own custom databases for their own custom models.
• 41:29
  SPEAKER_3: Mm-hmm.
• 41:30
  Is that what ultimately the direction the industry will take,
• 41:33
  you know, th there are verticals, lots
• 41:36
  SPEAKER_3: Yeah.
• 41:37
  SPEAKER_0: lots of verticals with very carefully curated databases that
• 41:40
  SPEAKER_3: Yes.
• 41:41
  SPEAKER_0: you can use for specific things.
• 41:42
  SPEAKER_0: I mean is that where the industry should end up?
• 41:45
  SPEAKER_3: I think it's where it will end up.
• 41:46
  I don't the the notion of all of these frontier models being the all singing,
• 41:52
  all dancing, you know, source?
• 41:53
  SPEAKER_3: No.
• 41:53
  SPEAKER_3: They are foundational.
• 41:54
  SPEAKER_3: And as foundations, they cover so much of the territory that needs to be done.
• 42:11
  SPEAKER_0: Right.
• 42:11
  And those vertical models are more likely to be ethically trainable and to abide by
• 42:17
  a constitution than the closed source LLMs,
• 42:19
  you know, that are behind the major chatbots today.
• 42:22
  SPEAKER_3: Yeah.
• 42:22
  SPEAKER_3: There's at least one– two open source models.
• 42:23
  They're not as big and as powerful,
• 42:23
  but they are local open source open weights models where
• 42:26
  the training data has been scoured and's been truly curated,
• 42:28
  and their performance,
• 42:29
  when brought to these areas of hallucinations or confabulation...
• 42:30
  much different than what you will get in a in a normal garden variety LLM.
• 42:33
  The next point is right now,
• 42:34
  when you think about the amount of money that is going into frontier models,
• 42:36
  enormous amount of it is in the training.
• 42:38
  SPEAKER_3: Why?
• 42:38
  Because every time you train a model,
• 42:42
  you start from point zero and you have to run through the entire billions
• 42:49
  and trillions of parameters and pieces of data to get it to the point that it was in
• 42:56
  the last iteration, and then the additional goodies.
• 43:00
  But what happens when you actually have models that start at a certain base,
• 43:06
  what we've already got available to us,
• 43:08
  and that learning can be a continuous and a consistently monitored
• 43:15
  and managed process Where you don't have to pay for a restart every time.
• 43:48
  SPEAKER_0: Models don't have states right now, right?
• 43:50
  SPEAKER_3: They actually do, but they don't have states that are alterable, easily alterable.
• 43:55
  SPEAKER_0: Yeah.
• 43:56
  SPEAKER_3: But there is a fundamental issue here.
• 43:57
  And that is to get to the point where these models don't have to cost as much
• 44:02
  to train and to learn, and then for inference, that is generating the answers.
• 44:06
  SPEAKER_3: If that can be addressed in some very real ways.
• 44:07
  SPEAKER_3: This is one of the reasons why I publish the blog that I do and why I work on Provenance.
• 44:12
  This is part of the mission,
• 44:14
  and that is find the means by which,
• 44:16
  by hook or crook,
• 44:17
  you can get these things to be continuously learning, and I mean truly learning.
• 44:33
  SPEAKER_0: We're coming up against our time limits.
• 44:35
  SPEAKER_0: Anything you wanted talk about that we haven't asked you?
• 44:40
  I think the question that wasn't asked and therefore not answered was:
• 44:44
  Are there principles that we could return to
• 44:45
  beyond just putting some fences around things, that we did with Telescript.
• 44:48
  SPEAKER_3: And what did we learn from it?
• 44:49
  One of the things I can point to with some assurance that I'm right,
• 44:52
  is that the things that controlled at least the outward behaviors
• 44:54
  of companies like AT&T,
• 44:56
  IBM made them better citizens than the hyperscalers are today.
• 45:00
  They were concerned about their image,
• 45:02
  yeah, they were generated by the revenues and by their commercial successes.
• 45:06
  SPEAKER_3: But the principles that they upheld as a result were ones of quality and concern.
• 45:12
  And sometimes being a little bit conservative about your utilization of
• 45:16
  the new new thing is a better idea than putting people at risk.
• 45:35
  SPEAKER_1: Now we have a question that we always ask.
• 45:36
  SPEAKER_1: What is your bumper sticker?
• 45:38
  If there's a maxim you live by,
• 45:40
  if there's a through line to your life or career, what would that read?
• 45:50
  SPEAKER_3: Live it or live with it.
• 45:52
  SPEAKER_0: Yeah.
• 45:53
  SPEAKER_1: Excellent.
• 45:53
  SPEAKER_1: Rich, obviously a huge part of the General Magic story is innovation and technology.
• 45:57
  But for those of us who were there,
• 46:01
  a tremendous part of the story is also the relationships,
• 46:05
  personalities, and the people involved.
• 46:08
  SPEAKER_1: You and Jim had a remarkable not just partnership, but friendship.
• 46:11
  SPEAKER_1: And I was sad to learn of his passing.
• 46:14
  Wonder what do you most remember about him as a person and what it resonates
• 46:19
  with you still most about him?
• 46:27
  SPEAKER_3: Couple of things.
• 46:27
  SPEAKER_3: Thank you for asking.
• 46:28
  SPEAKER_3: first of all, I was very, very sad to hear of about Jim's passing.
• 46:29
  And you know,
• 46:30
  here's a guy that with whom I worked really closely for the better part of 15 years
• 46:31
  in the standards, then in our consultancy, then at General Magic.
• 46:33
  SPEAKER_3: The best thing about Jim was you went in and you'd better be prepared, because he always was.
• 46:35
  SPEAKER_3: Almost for any topic.
• 46:35
  SPEAKER_3: Didn't have to be technology.
• 46:36
  And he was,
• 46:36
  as I said before also,
• 46:37
  he could be actually militant in the way he,
• 46:38
  you know, said, This is the right thing.
• 46:39
  SPEAKER_3: But on those occasions where I would come and I'd say, here's an argument.
• 46:44
  SPEAKER_3: And as far as I can tell, it's a cogent argument.
• 46:47
  SPEAKER_3: And I would lay it out for him.
• 46:49
  SPEAKER_3: He literally would stop.
• 46:52
  SPEAKER_3: There was this dead silence, and then he'd go, "You're right.
• 46:55
  SPEAKER_3: That is the right way to do it."
• 46:57
  SPEAKER_3: And he would just literally turn on the dime.
• 47:01
  SPEAKER_3: Getting it right was more important than anything.
• 47:05
  And he was absolutely a lovable,
• 47:08
  kind person behind that,
• 47:10
  you know, very orderly, almost nerd pack engineer impression that you would get.
• 47:17
  SPEAKER_3: But the best thing, as I said, was he could be as egoless about it as anybody I've ever met.
• 47:25
  SPEAKER_3: And miss the guy terribly.
• 47:56
  SPEAKER_1: What a beautiful remembrance.
• 47:58
  SPEAKER_1: Rich, thanks so much for joining us.
• 48:00
  SPEAKER_1: Always great to see you, and this was no exception.