Latest / The Road to Accountable AI / Tim O'Reilly: The Values of AI Disclosure

Transcript

Tim O'Reilly: The Values of AI Disclosure · The Road to Accountable AI

• 0:01
  Tim, welcome.
• 0:01
  Always great to talk to you.
• 0:03
  Great talk with you too, Kevin.
• 0:06
  What can I do for you?
• 0:07
  emerging technology for a long time.
• 0:10
  Let me just first ask you, how do you think about AI?
• 0:13
  What kind of innovation is it?
• 0:15
  How significant do you think it is?
• 0:17
  Well, I would start by putting all sort of great waves of technology innovation into one frame, computer technology innovation.
• 0:27
  And that is the increase in the ease with which humans are able to communicate with computers and get them to do our bidding.
• 0:36
  It's as simple as that.
• 0:38
  You think back to ENIAC where they programmed it by making physical circuits.
• 0:45
  And then we got to store program computers, but you put it in the program one bit at a time.
• 0:50
  And they got the switches on the front and then, you know, punch cards and, you know, Magtape for storing programs.
• 0:57
  And there was this sort of priesthood who could do these very difficult things.
• 1:02
  And then you have this big breakthrough.
• 1:04
  Well, first of all, you had a breakthrough with assembly language and then higher level languages that output machine instructions without people needing to write them directly.
• 1:14
  And then you got interpreted languages that made it really easy.
• 1:19
  And then you had things like the PC, which made it possible for anyone to have a computer.
• 1:23
  And suddenly there were millions of people doing these things, writing programs.
• 1:28
  And that exploded.
• 1:30
  And then you really got something really new, I think, with the web, which was where you could actually create documents that called programs.
• 1:39
  And so that was like, could basically make an interface out of a document that was the kind of thing that you know, the humans exchange with each other.
• 1:47
  And people don't recognize that what a big step forward in interfaces that was in some ways it was even bigger than the GUI, you know, which are the graphical user interface,
• 1:55
  which is of course made things simpler.
• 1:58
  And so now we're at a place where we can just talk to these things in plain language and they can start to do what we want.
• 2:05
  And if you look at the history, every time we've had one of these advances,
• 2:09
  More people could use computers and they could do more things with them.
• 2:13
  And so it's profound.
• 2:15
  It really is profound.
• 2:16
  It's basically going to grow the market.
• 2:18
  And I just mean in the number of people who can use it, but in the number of things that they can do.
• 2:25
  And again, you can look at other technologies.
• 2:30
  You read some of the early history of the automobile and it's like if you want to be an automobile owner, you had to be a mechanic.
• 2:39
  Mm-hmm.
• 2:39
  now you're at a point where you can't even be a mechanic because they've got that's going too far, maybe.
• 2:45
  you know, there are certainly some interesting parallels.
• 2:50
  Yeah.
• 2:51
  Well, so what is there anything we can learn from at least in computer technology, those, those prior shifts that can help us in engaging with the issues that are coming up now
• 3:00
  with AI.
• 3:01
  absolutely.
• 3:02
  First off, know, people underestimate them and then they overestimate them.
• 3:10
  And and in some ways, the you know, this is sort of thermostatic kind of process, just like in politics, I guess, in which, you know, people try to have this grand narrative and
• 3:26
  and the world sort of stumbles forward in
• 3:31
  unexpected ways.
• 3:33
  think if you look at, example, again, I think back particularly on my early career was very shaped around the World Wide Web.
• 3:41
  I mean, it was there before that, an open source software, people got a lot of things wrong.
• 3:49
  My whole battle with the, I convened the meeting where the term open source software was adopted, but I...
• 3:55
  was kind of an outlier and everybody was focused on licenses.
• 3:59
  And I said, I don't think licenses are the issue.
• 4:01
  It's actually, we have network enabled collaboration.
• 4:05
  We have what I called the architecture of participation, the design of systems that people could make small pieces of and they work together, sort of open architectures and kind of
• 4:19
  look at the difference.
• 4:20
  There's this great line I remember seeing on the internet early on.
• 4:23
  The difference between theory and practice is always greater in practice than it is in theory.
• 4:28
  And I don't know who said it, but it's brilliant.
• 4:30
  And because always you have people who have this sort of theoretical construct, you know, the same thing with hypertext.
• 4:37
  You know, it's like when the World Wide Web came out, you know, all the pundits said the hypertext pundits said this won't work because it only has one way links.
• 4:45
  It doesn't have two way links.
• 4:46
  It has to have two way links.
• 4:48
  And of course the one way link thing was part of what made it grow so explosively that it didn't, you could just get a 404.
• 4:58
  And everything wasn't tight and tightly bound.
• 5:03
  so I think the thing we learn is that, you we stumble forward and there'll be some innovation and then somebody else will build on that innovation.
• 5:17
  And I guess this goes back, I think it's interesting because Ethan Mollick, who's one of my favorite observers of AI, has become a fan like I have of James Besson's work on the
• 5:26
  Industrial Revolution.
• 5:28
  You know, where he basically said, why does it take so long for new innovations to spread?
• 5:33
  says, because people have to learn how to use them and you need to build communities of practice and you have to have people pushing on and innovating.
• 5:41
  And that's certainly true.
• 5:43
  you know, with, again, I think back to the worldwide web, you know, was sort of like the original web was static documents.
• 5:49
  Then some guy had the bright idea of like, Hey, we can actually call a database from one of these things.
• 5:54
  And then we can have, you know, led the dynamic websites.
• 5:57
  And then you had all these people who are having APIs right into the server and Apache said, no, no, we're not going to do it that way.
• 6:03
  We're going to have an architecture that builds on top.
• 6:05
  And, you know, it was this evolutionary process that was people.
• 6:11
  learning and practicing.
• 6:12
  Brian Pinkerton, the first web crawler, and then Google figures it out and how to do it better.
• 6:18
  And Overture figures out pay per click, but they do it with just a crude auction and Google figures out how to make a really good auction system.
• 6:29
  And bit by bit, the world that we became familiar with evolved.
• 6:37
  Steve Jobs with the iPhone, et cetera, et cetera.
• 6:40
  So we're in this stage where, you know, we're very early and, you know, I do think that there's an issue now which we didn't have before.
• 6:55
  And I first wrote about this with regard to where I think Silicon Valley went wrong with its blitz scaling.
• 7:03
  Reed often calls it blitz scaling.
• 7:05
  This idea.
• 7:07
  that companies should race to get market share and the VCs should basically invest in them so they can do that.
• 7:14
  And I describe this with Uber and Lyft.
• 7:16
  In some sense, the market didn't pick the winners.
• 7:19
  The right winning business model didn't evolve.
• 7:22
  What happened was the VCs flooded the market with capital.
• 7:25
  They picked a couple of winners and a couple of early business models.
• 7:29
  They got their money out because these companies went public.
• 7:31
  And then companies were like, actually, we have to raise prices.
• 7:37
  And now we're starting to have that period of experimentation.
• 7:42
  And you have that same problem of the winners have already been chosen by the massive amounts of capital.
• 7:50
  So it's kind of like we celebrate this idea that we have this innovation market.
• 7:57
  But increasingly, that innovation market has become a kind of central planning by a
• 8:05
  a small number of very deep pocketed companies.
• 8:08
  And so that's the biggest, one of the biggest worries that I have that we, it will be harder for us to have, you know, some of the kinds of experimentation that we had in the
• 8:20
  past that led to the real innovations that we needed.
• 8:26
  Right, and given the extremely high cost throughout the entire stack for developing some of these AI models, what can we do now to avoid that kind of very concentrated future for
• 8:38
  AI?
• 8:40
  Well, I think one of the first things that we need to do is to think about this idea that I talked about back in the days of open source of the architecture of participation.
• 8:51
  You know, I've been giving some talks where I riff on this.
• 8:58
  There was this article or actually it was a podcast, the New York Times, where it was called AI's Original Sin.
• 9:08
  And in it, they quoted this
• 9:11
  it was basically focused on the copyright issues around AI.
• 9:16
  And they quoted this lawyer for Andreessen Horowitz who said, you know, if we don't let these companies, you know, this is the only possible way to build these giant models.
• 9:27
  And I thought, yeah, that's a lot like 1992 when the only possible way to get your content online was, you know, was, it was AOL.
• 9:38
  And then a little later, the Microsoft network.
• 9:41
  And this thing was coming out of left field, the worldwide web, where anybody could get their content online.
• 9:46
  And it won because it built a real market.
• 9:50
  And right now we're in the AOL stage of AI, you know, as a way to think about it.
• 9:56
  You know, we've got these big centralized players and what's waiting in the wings, I think, is this alternate world in which you have cooperating AIs that are, you know,
• 10:09
  you potentially trained on specialized data.
• 10:13
  And people are starting to say this.
• 10:14
  There was an article I saw recently that said, you know, Sam Altman's real rival is Jamie Dimon, you know, because JP Morgan sitting on this huge, they're working a lot on AI,
• 10:24
  they're sitting on this huge class of specialized data that's not available to any of these guys.
• 10:31
  And so they can now.
• 10:32
  got the capital.
• 10:33
  They've got a $10 billion ERIT budget.
• 10:36
  So, yeah.
• 10:36
  know, there's sort of an interesting question there.
• 10:39
  Does that, you know, play out?
• 10:43
  But in general, it goes to the sort of questions that I think about, because, of course, we started building things at O'Reilly.
• 10:50
  We have a much smaller than JPMorgan Chase.
• 10:57
  But we have a business where we have a body of intellectual property that in theory has not been accessible to
• 11:06
  the models for training, think in practice, it probably has been, you know, they probably, probably, you know, got it by hooker by crook, but we don't know that for sure.
• 11:17
  which is of course why I do think that the company should be required to disclose their training data, at least, you know, like if only in the form of, you have my, you know, do
• 11:27
  you have my, my content queries just, just like we have with, with, privacy, you know,
• 11:35
  But more than that though, I just think that we need to think about an AI architecture that isn't, okay, we have a winner takes all player and then we just have to basically be
• 11:51
  satellite to them.
• 11:54
  I think if that's the world we're building for, those guys ought to be regulated utilities.
• 12:01
  And then you go, okay, you're a foundation model.
• 12:03
  You can be a foundation model.
• 12:05
  This is how you get paid, but you can't compete with everybody else.
• 12:09
  Yeah, that would be one way to go.
• 12:12
  It's kind of what we ended up with with telecommunications in a certain way.
• 12:17
  Yep.
• 12:18
  But I mean, your point before, which was a really good one was, you know, so many things seem inevitable in hindsight.
• 12:22
  And, and, you know, I'm sure you've talked to lots of people who weren't there in the early days of the web or even the days of web two, just to assume it couldn't have been
• 12:30
  otherwise.
• 12:30
  Now we're in this period where it seems like there's some possibility.
• 12:33
  So are there things that regulators should be doing that companies should be pushed to do that might make it more likely to have that more distributed future?
• 12:42
  Well, one of the things that I think we could and should do is have more disclosure.
• 12:50
  Now, when I say that, lot of people, go, disclosures don't work.
• 12:55
  And what they're thinking about are things like, you get your prescription, there's this long piece of paper that you peel off and you throw away that has a bunch of illegal
• 13:08
  gobbledygook and...
• 13:10
  you know, or whatever, you know, or even food labeling, you know, there's a little bit of value.
• 13:15
  OK, how many calories is this and so on?
• 13:17
  What are the ingredients?
• 13:19
  But there's a different kind of disclosures that I think people don't really think about that really is a lot closer to kind of communication standards.
• 13:32
  You know, like TCPIP is a kind of disclosure.
• 13:36
  You know, it says this is a
• 13:40
  the format for a particular kind of thing and you can build to this.
• 13:45
  And in a similar way, I've been thinking a lot about the analogy to accounting standards, which is really how do you manage money?
• 13:57
  And we standardized that back at the beginning of the 20th century.
• 14:02
  But the standards were based on what people actually did.
• 14:06
  Everybody kind of
• 14:10
  go, okay, how much did I take in?
• 14:11
  How much did I spend?
• 14:12
  And double entry accounting had been around for, you know, since the 13th century and had been refined quite a bit.
• 14:19
  But it wasn't, you know, there were a lot of people who were a little dodgy about it.
• 14:23
  And so in the early days of, of, you know, public companies and securities, they went, wait a minute, we got to actually make sure that people follow the same rules in
• 14:33
  describing the finances of their business.
• 14:37
  And when
• 14:39
  In the last few years, I worked on this project around trying to think about what I was calling algorithmic rents and how big tech companies use their algorithms to extract rents
• 14:49
  from their marketplace and their users.
• 14:51
  The idea of control over attention being used to then extract money from various other parties and sometimes unfairly.
• 15:01
  But in the course of that, one of the things that we thought about, we kind of came across was this idea
• 15:08
  from accounting of segment reporting, which is this idea that was introduced in the 1960s that it was in the age of first industrial conglomerates where they were saying, okay, if
• 15:20
  you have some segment of your business, it's more than 10 % of your profit.
• 15:23
  You need to report it separately.
• 15:25
  And so when we started thinking about internet companies and how opaque they are,
• 15:33
  You know, you, you go, well, you know, how, interesting would it be if, you know, we said to Google, okay, you need to tell us certain things about your nine properties that have
• 15:48
  more than a billion users.
• 15:50
  And we don't care that you tell us they're not, they don't have revenue associated with them, you know, cause that's the old measure was revenue, you know?
• 15:59
  And the question, and it's pretty clear to us now that market power.
• 16:03
  in these internet conglomerates is not associated with some of the old inputs and outputs, right?
• 16:10
  Google Maps is a great case in point.
• 16:14
  It actually is probably a bit larger in its revenue than, it's like, it's about $4 billion by most estimates.
• 16:22
  So it's bigger than Garmin, it's bigger than Esri, it's bigger than Autodesk mapping stuff.
• 16:31
  You know, so it's big in its market, but it's a rounding error for Google.
• 16:35
  But regardless, it's got 2 billion users worldwide.
• 16:38
  It's an enormous source of market power.
• 16:41
  And I thought about that recently when I put on the, you know, you know, the Meta Ray-Ban glasses and I wanted to ask it how to get somewhere.
• 16:50
  I go, they don't have Google Maps.
• 16:53
  You know, you know, and you think about the cost of reproducing something like that.
• 16:58
  And anyway, the whole point about
• 17:01
  There's this lesson from accounting, which we could bring over to AI.
• 17:08
  And this goes back to some of my thinking about AI, I mean, back in the web day and big data, where we're building a kind of operating system in which the subsystems are data,
• 17:22
  like location, identity.
• 17:25
  It's like Facebook.
• 17:27
  took control of a certain kind of identity.
• 17:29
  And of course, then Google and others and Apple, they all had to build their identity subsystem.
• 17:35
  And so you start thinking about that in the context of AI and you kind of go, OK, well, there's an interesting piece of leverage.
• 17:43
  How would you think about what are the standards for interoperability in these things?
• 17:50
  And I think this all, so that anyway, that notion that we have to
• 17:55
  change our thinking about what is useful to disclose, not because, I guess I hadn't fit follow through with the accounting analogy as fully as I should have.
• 18:08
  You if you think about what's the purpose of those financial disclosures, what do they do?
• 18:13
  They enable what I think in, I love this term from Jack Clark of Anthropic and Jillian Hadfield of regulatory markets.
• 18:22
  They wrote a paper on this.
• 18:24
  And I probably take in their idea a little further.
• 18:27
  Maybe I'm going beyond the way they thought about it.
• 18:30
  But if you think about part of what enables the regulatory market of finance are generally accepted accounting principles and the European equivalent, is the fact that there's now a
• 18:44
  market of accountants and auditors who all do the same thing.
• 18:49
  And that's a lot like, as I said, when you build a
• 18:52
  networking equipment, it all does the same thing.
• 18:55
  And that was how the internet grew.
• 18:58
  I remember back when my friend Dan Lynch started Interop, the literal was a conference where people came and all the computers had to talk to each other.
• 19:08
  And ITF was like, you propose a standard, you got to show us three independent interoperable implementations.
• 19:17
  I kind of feel like that notion of interoperability, the notion of how standards
• 19:25
  inform interoperability is really, I think,
• 19:37
  I guess is where I would go because I don't think, I think that part of what.
• 19:44
  You know, I always have in the back of my mind this great quote from the famous computer scientist Donald Kloonuth, who once said, premature optimization is the root of all evil.
• 19:56
  And, and there's a way that, you know, a lot of the regulations that being proposed are premature optimization and over specification.
• 20:06
  And again, I go back to my early background in networking and you look at the ISO standards.
• 20:11
  where they had the seven layer model and everything was specified.
• 20:16
  And TCPIP was the classic version of, what's his name?
• 20:23
  The guy who wrote, the name of the book is Systemantics, John Gall.
• 20:33
  No working complex system was ever designed that way.
• 20:37
  It evolved from a simple system that works.
• 20:39
  And TCPIP was a simple system that works and it far outperformed this system where people had tried to come up with the entire stack in committee.
• 20:51
  let me stop you there.
• 20:52
  I want to get back and ask you about the disclosure base.
• 20:55
  This is fascinating and important.
• 20:56
  But would you say that something like the European AI Act is a premature regulatory optimization for AI?
• 21:06
  only in the sense that, let me put it this way.
• 21:11
  If regulation was as easy to refactor and change and update as you learn new things as, as you see what happens, it would probably be okay.
• 21:26
  The reason it isn't is you, you're sort of incur a kind of societal technical debt, you know, with
• 21:33
  when you have a set of rules that don't actually work and are hard to change.
• 21:40
  And that's one of the advantage that we have with technical standards because they're fundamentally focused on outcomes.
• 21:49
  And if they don't work, they lose in the marketplace or they're adapted.
• 21:55
  It's one of the things that I've always loved in a certain way about, I mean, not say there's a lot of politicking.
• 22:02
  you know, as the industry has gotten, you know, more front and center.
• 22:08
  But I still think back to the early days of not just the internet, open standards of all kinds.
• 22:17
  I remember a meeting of this thing called the X Consortium, which I was an affiliate member.
• 22:23
  X was an open source windowing system for Unix Linux and still there.
• 22:32
  It was really this consortium where all the voting members were the big companies that were developing the technology.
• 22:37
  And I still remember a meeting where these guys, would be just, you know, I'm Steve, the developer, I'm, know, and then suddenly they would kind of like sit up a little
• 22:50
  differently and they would say Apollo computer believes that.
• 22:54
  And this one time, this guy was, I don't remember if it was Apollo or DAC, Digital Equipment Corporation, and he started that.
• 23:02
  that he started that.
• 23:07
  Well, digital believes that.
• 23:09
  then he says, I don't care what digital believes.
• 23:12
  It's not the right answer.
• 23:13
  I was like, yes.
• 23:16
  I wish it was preserved.
• 23:17
  That moment was, if this was in the YouTube era, would have been.
• 23:22
  But it was just such a moment of the engineer goes, no, this is not the right answer.
• 23:28
  I don't care what my corporate masters are telling me.
• 23:31
  Right, right.
• 23:32
  But back to what we talking about before, given how much money is at stake and is already involved in this AI space, is there any hope to get back to that, the technologists are
• 23:45
  just talking about what the right answer is?
• 23:47
  first off, there are people who are.
• 23:50
  I mean, I think it's interesting, you know, despite what I said earlier about the, you know, the big money players making it a less competitive market than it might otherwise
• 24:01
  be.
• 24:02
  You know, we have some really interesting, you know, cross currents.
• 24:06
  You know, I think Anthropic is, you know, the idea that a bunch of people said.
• 24:12
  Open AI has lost its way with regard to AI safety.
• 24:15
  We're going to do better.
• 24:16
  But then they're also doing a lot of other things better, too.
• 24:18
  So you've got some really interesting competition there.
• 24:22
  think that Google has been, like, if I rank the big players here, Open AI is definitely in the move fast and break things camp.
• 24:36
  We're going to be dominant.
• 24:38
  We're going to do it however we can.
• 24:45
  is a really interesting alternative and Lama is a really interesting alternative.
• 24:54
  so that whole sort of, and then of course, there's all the smaller open source models and other models being developed.
• 25:03
  I do, so I do think we are having some competition there.
• 25:10
  But it's still not at the moment.
• 25:12
  Again, I think back to my computer history.
• 25:16
  And you have the PC.
• 25:22
  And there were a couple of different battles playing out.
• 25:26
  And a lot of people were focused because the first generation of computing dominance with IBM had been through hardware.
• 25:35
  Everybody was focused on the hardware battles.
• 25:38
  And they were only paying attention to that.
• 25:40
  you know, here's AT &T, they're going to have a PC and, know, and then, you know, along comes, you know, Michael, you know, and the big companies all had their personal computers
• 25:48
  and there were, it was a lot of innovation, but like some of them were just like, Michael Dell comes along, he's like a kid in a Texas, you know, college dorm room.
• 25:55
  And he's like, I'm going to start assembling these things in a salient by mail order and, and becomes a real wild card.
• 26:02
  And I think there's some, there is at least the possibility as
• 26:07
  you when you think about possible futures, you know, one of them is the one that's being advertised, which is we have to keep training on more and more and more and more data, and
• 26:15
  it's going to be more and more and more expensive.
• 26:17
  And then there's the fact that the smaller models are catching up.
• 26:21
  And that maybe, you know, you're seeing the benefits of the current, you know, spending level off, and this stuff will be commoditized.
• 26:29
  And then we'll start to see the real innovations happen because it's not going to be constrained by a few companies.
• 26:38
  You know, who are like, okay, we have to dominate and then we have to monetize.
• 26:42
  And I guess that goes to something else that regulators could do.
• 26:45
  And again, this, I think about this not in terms of regulation that specifies what you can and can't do, but regulation that specifies what you have to tell us.
• 26:57
  You know, and so for example, if you were like caring about, you know, certain, you know, one axis of AI safety, which is, you know, addictiveness for, for kids.
• 27:07
  You you wouldn't necessarily say you can't.
• 27:11
  Well, you could.
• 27:12
  mean, China has literally limited social media use for kids.
• 27:15
  You could take that approach, but you could also say you got to report to us, you know, what kind of engagement patterns you have and what you're doing to maximize it.
• 27:27
  You know, so that it was pretty clear that like, OK, you know, like that case with the kid who committed suicide, it's like, could they have known that, you know, that the fact
• 27:35
  here's this kid who's clearly addicted?
• 27:39
  And should they have guardrails against stuff like that?
• 27:43
  how would we know if they were egging him on, they're going, great, this is a really great engaged customer.
• 27:50
  What are their optimizations for engagement, for example?
• 27:54
  And so I think there's a lot we learn from
• 27:59
  You know, I guess this goes back to my notion of why disclosures, this is a little different than the kind of sense of disclosures in the form of, you know, the metrics
• 28:12
  that, you know, everybody uses, and that becomes a standard for interoperability.
• 28:16
  But still it's related is if you look at social media, you know, once it started to optimize for engagement,
• 28:27
  you know, certain bad things happen, you know, and you look at, you know, the work that we did on algorithmic rents, it was sort of like, okay, we have this knowledge because of a
• 28:36
  lot of consultants who studied search engine optimization.
• 28:40
  And we have a lot of sense that companies like Google and Amazon got really good at search, i.e.
• 28:46
  giving people the answer they really wanted.
• 28:49
  And so we were able to do a study that said, okay, Amazon advertising no longer gives you their best result.
• 28:56
  You know, gives you the, you know, somebody paid for this result and that result is 17 % more expensive and 33%, you know, lower in ranking than what their actual best result is.
• 29:08
  And so whether, you know, what you want to do about that, don't know, but knowing it is pretty useful, you know, and knowing that, you know, that, that, companies are, you know,
• 29:21
  like if
• 29:22
  you when you have a, you know, like if companies have to sort of show, yeah, we were really putting the pedal to the metal on this, this risk factor, as opposed to we're
• 29:31
  actually moderating it and managing it.
• 29:33
  I think there would be some, know, like, you know, you don't have to kind of have the law come down and say you are going to be punished, because the market will punish people
• 29:43
  because somebody is going to sue them.
• 29:45
  And you'll have the information to say, yeah, you know, you were doing this bad thing and we can see it in your numbers.
• 29:53
  And it caused this harm.
• 29:54
  And so I kind of feel like, right now we have this sort of crazy ass approach, which is sort of fed by all the existential risk people that's akin to saying, okay, let's regulate
• 30:10
  cars that can go over 150 miles an hour, but no other cars.
• 30:15
  And let's do a bunch of crash test dummy kind of testing, but have no other regulations.
• 30:22
  You when you think about auto safety, yeah, HTSA does crash test dummies and how long does it take for a car to break at a particular speed and what's its crumple zone, all this
• 30:33
  kind of really useful stuff, which is a kind of model safety.
• 30:39
  But we also think about driver education and licensing, speed limits, you know, and the speed limits aren't
• 30:48
  You know, one size fits all.
• 30:49
  They're different on different kinds of roads.
• 30:51
  And you could say, okay, well, this is how, you know, we, what we think is safe in this market or that application.
• 30:58
  you could be thinking about, you know, again, I say this idea of, licensing, Jillian Hadfield, I think it's really been big on this notion of, Hey, AI models ought to be
• 31:09
  registered, just like, you know, you know, cars are registered.
• 31:12
  have a license plate, you know, and,
• 31:16
  know, guns have a serial number and, you know, models, no, not so much.
• 31:20
  You know, again, there's a lot of lessons we could take from what is the overall infrastructure of, you know, that allows, first of all, allows you to inquire when
• 31:33
  something goes wrong.
• 31:34
  Again, think about black boxes and, you know, airplane crashes.
• 31:39
  You know, there's a lot of interesting infrastructure.
• 31:46
  that makes a system regulatable as we start to see problems emerge.
• 31:53
  And I think that's an interesting conversation I've been having with Vint Cerf.
• 31:57
  It's like one of really interesting questions that we ought to be asking is, is this system regulatable at all?
• 32:03
  And how would we make it regulatable if it isn't?
• 32:07
  Because it's certainly possible that
• 32:10
  You know, you can build technologies that are fundamentally not regulatable and then we have to ask ourselves, do we want to do that?
• 32:17
  Yep.
• 32:17
  Yep.
• 32:18
  Yeah.
• 32:18
  think Larry Larson called it regulable back in the, the early internet days.
• 32:22
  And, know, we have these same conversations in blockchain and so forth.
• 32:25
  And it's an important point.
• 32:26
  Let me ask you one more thing, which is where does open source fit into this?
• 32:31
  Because as you know, there's, there's this big debate about, open weight models and the risks that they provide.
• 32:37
  And in some ways that seems like it's more disclosing, but on the other hand, it's out of the control, potentially the developer, what people are doing.
• 32:45
  Yeah, I guess I'm not sure I have a clear answer on that.
• 32:54
  In general, I am a fan of the way that open source makes a market more competitive.
• 33:05
  I do, know, obviously there are people who have a lot of concerns about national security, for example.
• 33:12
  you know, I guess the point is
• 33:16
  that Metta has made is, hey, look, this stuff is, we have a lot of industrial espionage.
• 33:22
  And as in every version of cybersecurity and everything else, a notion that you're just going to build a wall and keep people out is not generally that successful in the end.
• 33:38
  So you're better off building a robust system that can handle the fact that people know things.
• 33:46
  Okay, well, so last thing is we don't have too much more time is, so are you optimistic that we will come up with an approach that makes these technologies appropriately
• 33:56
  regulable?
• 33:57
  And if so, what gives you that optimism?
• 34:00
  Well, first off, I don't think they're as dangerous as the people who make the big existential risk arguments think.
• 34:09
  I don't think we're on a path to AGI right now.
• 34:13
  Now, we might be.
• 34:15
  I could be wrong about that.
• 34:17
  But I think that
• 34:24
  You know, a lot of the risks are, you know, classic.
• 34:30
  well, mean, well, there's a bunch of different risks.
• 34:34
  Yeah.
• 34:35
  Some of them are, you know, if you look at the, you know, the whole, CBRN area, you know, that's, you know, there's a whole community that's on top of that, you know, like does it,
• 34:47
  to what extent does this amplify existing risks?
• 34:50
  you know, in cybersecurity, know, bio, bioterror, et cetera, et cetera.
• 34:55
  That's all, you know, really important work.
• 34:59
  And, you know, but it's not specific to AI.
• 35:02
  It's just it's like, OK, we have to we have these frameworks for thinking about this.
• 35:07
  And even there, you know, like, you know, and the Ryan and Kapoor, you know, and I snake oil talk about, you know, the notion that, hey, well, if you're worried about bio, you
• 35:19
  know,
• 35:20
  know, bioweapons, know, physical biosecurity and access to the kind of equipment that you need is actually probably more important than, yeah, people can get information about how
• 35:29
  to do it.
• 35:29
  You know, so again, it's where we got the wrong, wrong focus there.
• 35:35
  But I think there's also a class of risks that
• 35:41
  I, and this is really the focus of my work in this area, which is this notion of, of commercialization risk.
• 35:48
  And there's two parts to that.
• 35:50
  And one is, do companies have the incentive to do the wrong thing?
• 35:56
  You know, so the move fast and break things risks, you know, so, you know, here is, you know, this race for monopoly and, you know, like companies.
• 36:09
  You know, like they're basically going, yeah, we're all about AI safety.
• 36:12
  And then they fire their AI safety team, you know, because really they want to win.
• 36:18
  And I think that is a real risk, but there's another risk.
• 36:24
  And the example, I try to use examples from the past and like great, two great ones come from the social media era and one of the versus the Myanmar massacre.
• 36:36
  And it was not.
• 36:37
  The thing, the big takeaway I have there is not the Facebook had guardrails against hate speech.
• 36:46
  They just didn't work in Myanmar because they didn't understand the language.
• 36:50
  Their systems weren't tuned for the language.
• 36:53
  so that I call that deployment risk.
• 36:56
  You know, so and again, it goes back to this.
• 36:58
  The difference between theory and practice is always greater in practice is in theory.
• 37:02
  In theory, we have guardrails.
• 37:04
  Do we have them in practice?
• 37:06
  You know?
• 37:07
  Yep.
• 37:07
  And that's the question that regulators should be asking.
• 37:11
  You know, it's like we, and this goes back to this analogy.
• 37:13
  Okay.
• 37:13
  We tested the, you know, we did the crash test dummy stuff, but now we're not looking at the data from the real world.
• 37:20
  You know, and we don't even thinking about the data from the real world.
• 37:23
  And so like to me, and again, that goes back to disclosures.
• 37:26
  You'd kind of go, Hey, you know, you say you do AI safety.
• 37:31
  What does it look like?
• 37:33
  What are you actually doing?
• 37:34
  You know, and that's again, a lot of what our.
• 37:37
  a project is trying to focus on, which is what should regulators be asking to see and to know about what a company is actually doing?
• 37:48
  And that requires understanding, okay, what is safety engineering for AI look like?
• 37:53
  And how much is a company spending on it?
• 37:55
  Are they doing it in all the markets they operate in?
• 37:57
  You know, or just some of them, you know, like they, you know, they go, yeah, well, we're offering our services around the world, but we're only doing
• 38:05
  all the safety engineering in English, you know, in the US, you know, or whatever, you know, because that's our biggest market.
• 38:11
  Yeah, that's the kind of thing that I would be thinking about.
• 38:15
  You know, a similar one is this is huge, you know, set of questions around what do your third party developers do?
• 38:23
  And this, this came out to me.
• 38:24
  And again, I haven't dug into it, what are, you know, companies doing across the board, but it's a super interesting area to me, which is
• 38:33
  There was this report from Proof News, Julie Angwin's outfit about election misinformation.
• 38:43
  And the main result was sort of obvious.
• 38:48
  These things were pretty shitty.
• 38:50
  They basically did red teaming with election workers who knew what questions people typically asked.
• 38:55
  They said, well, you're giving misinformation about half the time.
• 38:58
  But the most interesting result in the paper to me was that
• 39:04
  When pressed, the model developers said, you know, and there was a, they tested a bunch of models in parallel using an AI API harness, right?
• 39:13
  Where they would submit all the same questions and they were told, your results don't show our real guard rails because you were using the API and with the API, it's the
• 39:23
  responsibility of the developer.
• 39:25
  And I go,
• 39:26
  That's like, yeah, we have these protections against every privacy protections, but guess what?
• 39:31
  Cambridge analytic is this usually, you know, it's like, so if you're just saying it's the responsibility of the developer, is that really, that's just like you're wide open, you
• 39:44
  know?
• 39:44
  So, like either you have guardrails or you don't.
• 39:48
  And so again, this, question is like that, that I think should be regulated should be really thinking about deployment.
• 39:57
  Yep.
• 39:57
  just the thing in theory.
• 40:00
  Yep.
• 40:01
  All right.
• 40:01
  So much more we could talk about, but I think that's a good place to land.
• 40:04
  Always fascinating to speak with you, Tim.
• 40:06
  Thanks so much.
• 40:07
  right, thanks a lot.
• 40:07
  All right, great talk with you too.
• 40:10
  Bye bye.