Latest / The Road to Accountable AI / Helen Toner: AI Safety in a World of Uncertainty

Transcript

Helen Toner: AI Safety in a World of Uncertainty · The Road to Accountable AI

• 0:02
  Helen, welcome to the Road to Accountable AI.
• 0:05
  Thanks so much.
• 0:07
  Why should we be so concerned about AI safety?
• 0:10
  Lots of new technologies have risks and dangers.
• 0:15
  I mean, should we be so concerned?
• 0:17
  I think the answer is probably yes.
• 0:19
  I don't think it's definitely yes.
• 0:23
  So AI is a very broad technology.
• 0:25
  It's kind of an everything technology.
• 0:27
  Economists would say a general purpose technology.
• 0:30
  So in some ways it's hard to even talk about AI safety as one specific thing.
• 0:36
  But I think there's different pieces of this we could tackle.
• 0:39
  One piece would be, what are we already seeing from AI that's being deployed in the world?
• 0:43
  And I think...
• 0:45
  There's clearly potential harms, potential concerns that are already being realized.
• 0:50
  For example, if governments use an AI system to determine who gets benefits or to determine how they're making criminal justice decisions.
• 0:57
  These kinds of things are already happening.
• 1:00
  And if those systems are, certainly if they're biased, but also if they just don't work very well, if they're not necessarily particularly reliable, which is often true, that's a
• 1:10
  problem that really harms people's rights.
• 1:15
  is very concerning for multiple reasons.
• 1:19
  So I think we already have evidence.
• 1:21
  Or you could also look at something like self -driving cars, where is it an AI safety issue when a self -driving car crashes?
• 1:27
  You could call it that.
• 1:28
  You could not call it that.
• 1:29
  think either way, we can all agree we want to make sure that we're minimizing self -driving car crashes, because those are bad.
• 1:37
  I think maybe what you're gesturing at when you ask the question, though, is something a little broader, like should we be worried about
• 1:45
  kind of more catastrophic or civilizational scale risks that some are warning about.
• 1:52
  And I think we should be worried about them.
• 1:54
  I think we should be taking them into consideration and trying to understand how likely they are, what kinds of things could reduce them, if for no other reason than that the
• 2:06
  people developing this technology are telling us that they're a real possibility, which is,
• 2:11
  pretty unusual state to be in for those developing new technology.
• 2:16
  those researchers or engineers are often the most optimistic about what their technology is gonna do.
• 2:23
  So I think if nothing else, the fact that you have some of the kind pioneers of AI, including of deep learning that the paradigm we're currently in, suggesting that sort of
• 2:34
  this technology they've developed could really cause
• 2:39
  massive scale harms of different kinds, different experts have different views on this, of course.
• 2:43
  I think that alone should make us sit up and pay attention.
• 2:47
  And so then you have that combined with what are we already seeing in terms of harms from systems that are already deployed?
• 2:54
  What kind of pace of progress are we seeing in terms of the research, meaning that we might not necessarily have tons of time to kind of wait and see and figure this out as we
• 3:03
  go?
• 3:04
  To me, that all kind of combines together as an initial high level case for
• 3:08
  this at least being worth thinking about, worth taking seriously.
• 3:13
  Which is not to say worth totally freaking out about and panicking about and throwing everything else out the window, but I think certainly taking seriously.
• 3:22
  Isn't there potentially a tension between those two categories you described that to the extent that we are spending time and energy addressing what are today hypothetical future
• 3:31
  harms that it takes away from the ability to address the issues that are happening right
• 3:36
  Yeah, I kind of have two answers to this.
• 3:38
  think in principle, these things don't have to be intention at all.
• 3:42
  I think they have very much this sort of problems we're seeing from AI already, kind of existing harms.
• 3:49
  There's sort of a community of researchers focused on that, different community of researchers focused on more sort of anticipatory or speculative harms that we might see
• 3:57
  for much more advanced systems.
• 3:59
  And I think those two communities can make things seem like they have to be intention much more than I think they need to be.
• 4:05
  think there are actually
• 4:06
  plenty of priorities that different groups with different priorities, not just those two, but many others, should be able to agree on.
• 4:14
  even things as basic as the importance of technical capacity in government, us needing to have more expertise, more folks working in federal, state, other governments who
• 4:25
  understand the technology, or the need for transparency from the companies developing these systems, whether that's a company selling a sentencing algorithm
• 4:36
  a local government or whether it's a company developing, trying to develop artificial general intelligence, I think there should be kind of transparency requirements for both
• 4:45
  of those companies and likewise, accountability requirements in terms of who's responsible, who's liable if things go wrong.
• 4:52
  So in principle, I think there really doesn't need to be nearly as much tension as there has been.
• 4:57
  And I think there's been a little bit of movement towards sort of more coalition building there.
• 5:04
  And I think that's positive.
• 5:06
  At the same time, I definitely have to acknowledge that in the sort of last, I guess, year and a half, the kind of post -ChatGPT moment where there's been a bunch more interest, a
• 5:16
  lot of kind of policymaker concern and motivation, I have been dismayed at the extent to which it seems difficult to have conversations about both of these types of issues.
• 5:24
  It really seems like it's maybe easier than I would have expected to get.
• 5:31
  policymakers or the public really freaked out about the more cataclysmic possibilities, and then that's the only thing they want to talk about, and that sort of can dominate.
• 5:39
  And then in response to that, you can have people who are concerned about the issues we are already seeing here and now pushing back and trying to get that completely out of the
• 5:47
  discussion.
• 5:48
  Let's say have those existential or catastrophic issues totally off the table because they're worried about them potentially dominating if they're sort let in the door even at
• 5:56
  all.
• 5:56
  So.
• 5:58
  I think in principle, they really don't have to be intentioned in practice.
• 6:01
  I understand why people see them as sometimes being intentioned.
• 6:04
  And personally, I would love to see more conversations, more white papers, more policies.
• 6:10
  And I think maybe Biden's executive order from last October is a good example of this, where kind of all of these issues are present.
• 6:18
  And the fact that you are considering these really potentially scary civilizational scale issues doesn't mean that everything else is crowded out.
• 6:25
  And likewise, the fact that you want to talk about
• 6:27
  present -day issues that we're already seeing doesn't mean you feel the need to kind of banish that more speculative piece.
• 6:32
  Because I really do think that we need to both be observing what we're seeing now, thinking about trends, thinking about possible futures, and trying to do that kind of all
• 6:42
  as holistically as we can.
• 6:44
  What do you think are some of the ways that those communities can come together and find some common
• 6:50
  Yeah, I mentioned a couple.
• 6:52
  I do think that these ideas around transparency and accountability really apply across the board and figuring out exactly what that means, exactly how you implement that, what kind
• 7:00
  of requirements are you imposing on what kind of companies.
• 7:04
  Obviously, there's a lot of detail to figure out there.
• 7:07
  But I think that high level framing makes a lot of sense.
• 7:11
  yeah, things like I think another helpful concept has been this idea of rights or safety affecting systems, which I think
• 7:20
  forget where it originated, but it's certainly now been codified into sort of US government, federal government guidance for how agencies use AI.
• 7:28
  Can you look at systems that are either affecting people's rights or affecting their safety and kind of impose a higher bar on those?
• 7:36
  I think also sort of things like incident tracking.
• 7:42
  when you have in aviation, for example, if there's a plane crash, there's this whole infrastructure set up around going to figure out what went wrong.
• 7:50
  doing investigations, sharing information between different parties.
• 7:53
  We have similar things for traffic accidents, for cybersecurity incidents in a number of other, I think in healthcare there's similar things.
• 8:00
  And that's another kind of thing where I think both communities interested in what we're seeing today and also communities interested in sort of trends and future possibilities
• 8:10
  should be pretty motivated to try and understand what are the failures we're seeing and how can we think about preventing them going into the future.
• 8:19
  Another area would be what sometimes gets called AI evaluations, but I think more broadly measurement science for AI, which to me is trying to get the whole field on more of a firm
• 8:30
  scientific footing.
• 8:32
  A really big problem both for systems that are causing problems today and also for concerns about more advanced systems in the future is that right now we just really don't
• 8:42
  understand the systems that we're building.
• 8:45
  The best experts in the world can't really tell you
• 8:48
  why these AI models do what they do and why they succeed at some things and fail at other things and kind of get a little wacky in other situations.
• 8:59
  And I think one underlying reason for this is that, mean, machine learning researchers themselves will describe the field as more like alchemy than like chemistry.
• 9:10
  So meaning in chemistry, my husband is a chemist, we have really, really strong foundational
• 9:17
  understanding of the physical laws that underlie what's going on.
• 9:21
  And so for anything that we're observing in the lab, we can really try and pull apart in detail what exactly is going on there versus alchemy was sort of more just like throw
• 9:32
  things in a pot, try what happens if you do it at this temperature, what happens if you do it at that temperature, what happens if you do it under the light of the full moon, of
• 9:39
  experiment a lot of different, in a lot of different ways and much more of an empirical try things out, see what happens kind of And I
• 9:46
  machine learning researchers think of their field as well as much more of an empirical try things out, see what happens.
• 9:51
  Sometimes it's really bizarre and no one can really tell you why sort of a field.
• 9:56
  to come back to the question, think improving that situation, giving us more of a solid understanding of what is going on inside these algorithms, what kinds of bounds can we put
• 10:06
  on where they will perform well and poorly is the kind of thing that helps both with present issues and with the more future facing issues.
• 10:15
  Yeah, it's a really interesting point because on the one hand, it's scary in a way that the people that are developing these technologies don't quite understand themselves, how
• 10:23
  they work.
• 10:24
  And as you pointed out before, are some of the ones who are raising the alarm at some of the risks.
• 10:29
  On the other hand, there's a value isn't there and things that work.
• 10:33
  We don't necessarily know how, but it was not really anticipated that chat GPT would be this great leap forward.
• 10:40
  And people are now actually using it in business without having to work out all of the mechanistic details behind the scenes.
• 10:47
  So is it really important to get to some point where AI is more like chemistry?
• 10:54
  Yeah, I I think it's definitely a balance.
• 10:56
  And I don't think the answer is to say, we have to freeze everything, not use this technology at all until we understand it perfectly.
• 11:02
  That clearly wouldn't be the right way to trade off the different benefits and risks that we're seeing.
• 11:08
  That's part of why I like the rights and safety affecting framing, because I think it helps distinguish between, for example, is my email client suggesting a way that I can
• 11:22
  send an email response?
• 11:24
  Like, know, there's sort of limited ways that that can go wrong as long as I'm always reading and checking it.
• 11:28
  Or is my Spotify suggesting music for me to listen to?
• 11:31
  Like, I really think that experimentation in those kind of fields, like, why not?
• 11:35
  But I also think that it makes sense for us to look at fields or applications where that isn't actually appropriate.
• 11:43
  Or at a minimum, you if we are gonna use systems that we don't understand very well to have alternatives, to have the ability to, you know, if you're applying for...
• 11:51
  alone and you get denied by the algorithmic system to either ask for an explanation or ask for human review, have some kind of due process, some kind of recourse available to you.
• 12:01
  So I think it is definitely a balancing act of figuring out where is it OK to just go right ahead as long as it seems to work basically OK, and where is that more of a concern.
• 12:14
  And in plenty of cases, I think we can
• 12:17
  also rely on existing structures that are in place to make sure that things work well enough.
• 12:22
  if you think about some kind of chemical plant or an energy plant putting in new software, I believe that there are existing standards for what kinds of tests you're gonna do, what
• 12:34
  kinds of confidence you're gonna have about what you're putting in place there.
• 12:38
  so, or aviation would be another example where by and large, airlines are not,
• 12:47
  airplane designers are not just going in and throwing things in because they seem to work.
• 12:51
  They have really high safety standards for what actually does work.
• 12:54
  So think in many sectors, we already have mechanisms to manage this trade -off.
• 12:59
  And it's just, there are, but there are some sectors where that's not the case.
• 13:03
  For example, government use of algorithms, I think has not been super well regulated in the past.
• 13:08
  Or also some of these more general purpose systems we're seeing now, the chat GPTs of the world.
• 13:13
  where it's not really relegated to a particular sector or use case.
• 13:16
  And we're also kind of lacking these existing institutional mechanisms to determine sort of how good, how reliable, what kind of guardrails do you need.
• 13:23
  So I think, yeah, in some cases we're pretty well set up, but in other cases we maybe need new mechanisms or new guardrails.
• 13:30
  You've been working in this area for a number of years now.
• 13:34
  How would you say your own view on these questions has evolved since, say, you know, 2021?
• 13:42
  Yeah, I think.
• 13:46
  changed in a few different ways.
• 13:48
  One thing that's been absolutely fascinating to see has been the scale of response to ChatGPT as a product.
• 13:54
  think something that was really, I that was surprising to a lot of people in the field was there had been a series of advances, a series of relatively gradual advances starting
• 14:05
  maybe in 2012, sort of depends on how you count.
• 14:09
  And ChatGPT was a moment where there wasn't some big technical breakthrough, but it turned out
• 14:14
  having a system that worked well enough, that was not super straightforward to get it to be really toxic and racist and sexist the way that some other chatbots had been, and that
• 14:24
  was available for free in a very kind of user -friendly interface, that really obviously kicked off a wildfire of attention among the public, among policymakers.
• 14:34
  And I think in many ways that has been quite heartening to me and to others in the field to see that, because I started getting interested
• 14:43
  interested in this field in 2013, 2014, started being professionally focused on it around 2016.
• 14:49
  And a big part of why I was interested was because it seemed to me like it was going to matter so much, like it was potentially going to be such a huge deal for society.
• 14:57
  And so there was kind of a sort of a little bit of a feeling of being sort of lonelier in the wilderness for people in the space of like, we think this really matters, we think
• 15:04
  this is going to be a huge deal, but a lot of people aren't really paying attention to what's going on there.
• 15:08
  And so I think one thing that has shifted over the last couple of years has just been
• 15:14
  in some ways feeling validated, but also feeling relieved or really pleased that there is this much greater interest, lots of really, really smart people with expertise and really
• 15:24
  important other fields starting to pay attention to what is happening in AI and how does it relate to their field and what might be concerning, what might be great opportunities.
• 15:34
  So that's been a big change in the positive direction, I think.
• 15:40
  Maybe in the more concerning direction,
• 15:42
  how I feel about this, because it varies from day to day or week to week, is just looking at the continued pace of progress in the technology.
• 15:52
  Yeah, we really are seeing, so far it continues to look like if you take this, you know, basically the design that underlies Chatchie PT, which was developed in 2017 by Google
• 16:03
  researchers, a transformer.
• 16:05
  It does seem like if you keep just sort of scaling up the amount of computational power you're using, the amount of training data that you're using to develop.
• 16:12
  develop a system based on a transformer, that you do just keep getting more more sophisticated outputs.
• 16:18
  And it's very contested what exactly that looks like, whether the trend where there are an exponential going up to infinity or whether things are plateauing and it's really not
• 16:28
  going to be very interesting over the next few years.
• 16:30
  Experts really, really differ on that.
• 16:33
  But to the extent that we continue to see a pretty rapid clip of progress there, that also
• 16:41
  you know, is cause I think for some concern or some caution, given that so many of these questions about, you know, how do we control these systems?
• 16:48
  How do we understand them?
• 16:49
  What kind of societal guardrails that we need are still very open.
• 16:51
  So to the extent that we only have, you know, single digit years to, to kind of figure that out versus, you know, 10, 20, 30, 50 years.
• 16:59
  I think that's a little, a little more alarming.
• 17:02
  But yes, how I feel about kind of the balance of those different factors changes a lot depending on, depending on, what side of the bed I got out of what's been in the news.
• 17:11
  how things have been looking recently.
• 17:14
  Yeah, absolutely.
• 17:15
  So the other end of the time you were at OpenAI was around the unsuccessful effort to remove Sam Altman as the CEO.
• 17:24
  And so let me just ask you generally now that the dust has settled a little bit, what lessons do you take from that whole
• 17:31
  I mean, many lessons, personal lessons, lessons about how different people operate under pressure, lessons about how media works, how public narratives get built out of facts on
• 17:43
  the ground.
• 17:44
  Many, many lessons.
• 17:45
  think maybe one big important one that's especially relevant for this conversation is just that I think it, almost regardless of what you think of the board's actions, whether,
• 17:55
  regardless of who you think was in the right,
• 17:58
  I think it's very hard to come away from that situation feeling great about these companies ability to govern themselves and especially to govern themselves in really high
• 18:07
  pressure moments when the rubber is hitting the road and big incentives are at play.
• 18:13
  And I think that's in part because of the sort of financial incentives in question, the potential profit incentives.
• 18:19
  I think also when you're talking about people at these companies, not just OpenAI, but also companies like Google, Anthropic, Meta that are explicitly trying to build
• 18:28
  what they would call artificial general intelligence.
• 18:31
  We could have a whole other conversation about that term, but they're trying to build very advanced, very sophisticated AI systems.
• 18:37
  People who are kind of involved in those efforts think of those systems as incredibly powerful or will be incredibly powerful if they're built.
• 18:45
  So there's these sort of financial profit incentives, but there's also all these incentives around power dynamics and who is in control that I think it's very difficult
• 18:54
  for sort of voluntary self -governance structures.
• 18:57
  to handle.
• 18:59
  So think a big lesson for me sort of reflecting on that period and also that I hope would be a lesson that others can draw as well is that it really does look like we're going to
• 19:07
  need some kind of outside oversight over specifically that subsection of the industry that is working on building those incredibly capable, incredibly powerful AI systems.
• 19:17
  And I think there's been some good progress over the last year or two in trying to develop some of those outside oversight structures, but they're still pretty tentative.
• 19:24
  Many of them are still relatively voluntary.
• 19:26
  So think we need more.
• 19:28
  Yeah, what do you think those oversight structures should look
• 19:32
  I mean, I think there's been a great start from, I think there's a lot of reason to praise the companies involved and some of the voluntary measures that they're starting to
• 19:40
  undertake.
• 19:41
  There's been a move towards putting in place structures sometimes get called our responsible scaling policies.
• 19:48
  Open EYES is called its preparedness framework.
• 19:50
  Google has a frontier safety framework, which are basically structured around
• 19:57
  basically trying to forecast or trying to describe what types of capabilities they might observe in the systems they're developing that they would not be ready for, or that they
• 20:07
  would need to have certain godras in place before they could keep developing beyond that point.
• 20:11
  And so different companies look at different criteria, but it includes things like cybersecurity capabilities.
• 20:18
  Do you have a really effective, an AI system that can really effectively hack different computer systems or?
• 20:25
  autonomous behavior.
• 20:26
  Do you have a system that can really effectively go out and sort of pursue goals autonomously, maybe preserve itself, prevent itself from being shut down, commandeer
• 20:34
  server space or whatever it might be, bio weapons, chemical weapons, nuclear weapons, or other capabilities that are often brought up?
• 20:42
  And I think it's a really good step that companies are sort of setting this up in a voluntary way to say, here's the kinds of things that we're concerned we might observe.
• 20:50
  Here are the kinds of tests we're running to see if we observe them.
• 20:55
  And here's what we'll do if we do observe them and we don't have the right guardrails in place.
• 21:00
  I think over time it would be great if those kind of structures could become a little more codified, could be a little bit less just up to the companies to kind of interpret and
• 21:09
  decide what to do and a little bit more standardized.
• 21:12
  Maybe you need sort of external third party certification or review of those kind of testing processes.
• 21:18
  Maybe there's a role for government.
• 21:21
  I think that's one really.
• 21:24
  valuable potential step.
• 21:25
  Another category, I'm not a lawyer, so I never want to claim to have too detailed a view on this, but I think the category of liability or accountability, responsibility, would be
• 21:39
  a really great area to have more clarity on.
• 21:43
  Right now, the most obvious comparison for AI systems is software.
• 21:48
  And generally, when it comes to software, understanding, again, not a lawyer.
• 21:52
  But my understanding is that the companies developing software products that cause harm are generally not held liable.
• 21:58
  It's generally pretty hard to kind of make a case that they were at fault, unlike in many other industries where the original developers of things can be held liable.
• 22:08
  And I think there's, I think trying to set up a liability regime that sort of sets the incentives more effectively for these developers could be really valuable.
• 22:21
  And just to make explicit something I like a lot about both sort of that type of approach and also the sort of responsible scaling type of approach is that it is not about setting
• 22:31
  up a really heavy handed government regime that comes in and has to approve a lot of things the companies are doing and really a sort of assuming that something very dangerous
• 22:41
  is happening and that you need a really, really close government eye.
• 22:45
  Instead, I think things like responsible scaling policies or setting up good liability regimes
• 22:50
  leave a lot of room for judgment by the developers about what they're doing and what is appropriate, and a lot of room for flexibility depending on how the technology develops,
• 22:58
  which I think is really important given how much uncertainty and disagreement there is about what those developments will look
• 23:05
  Yeah, let me ask you a little more about the responsible scaling policies.
• 23:09
  How is it possible to come up with a set of tests around technology that doesn't exist yet where we don't know the capabilities and also where, as you talked about, the financial
• 23:21
  and other incentives on these organizations are against putting into place the kinds of guardrails that we might want to see.
• 23:29
  Yeah, so it's definitely to me an example of this dynamic of the of the rollout and the implementation of these AI systems that seem to work well, being a little bit in tension
• 23:39
  with, or definitely getting out ahead of our scientific understanding of those same systems.
• 23:44
  And so sometimes you'll see criticism from the industry, from researchers of, how could you kind of put these policies into place or how could you kind of make these plans
• 23:52
  without having much better scientific understanding of what's actually going on?
• 23:56
  And I think that's a fair criticism, but it also,
• 24:00
  It's tricky if you're trying to set the bar of we can only put in place any kind of restrictions, any kind of guardrails once we really understand perfectly what's happening,
• 24:09
  but we can go ahead with selling these products, with getting hundreds of millions of users, with implementing them into government services, regardless of whether we have that
• 24:17
  scientific understanding, right?
• 24:18
  So I think it's about, certainly it would be better if we could do all of this based on many years of experiments, tons of empirical data.
• 24:28
  great scientific consensus among experts about what problems to test for and how to test for them.
• 24:34
  But in practice, we have this field that is moving quickly, that is rolling things out to large numbers of people in pretty consequential ways.
• 24:44
  And so I think we don't really have a choice other than to try and find ways of putting in place sort of common sense tests or common sense limits around what we might see.
• 24:58
  And I do think that there's definitely, you know, it's more of an art to it than a science.
• 25:02
  If you look at something like Anthropic has a responsible scaling policy that goes into a reasonable amount of detail about sort of what kinds of tests, for example, they would run
• 25:11
  on the cyber capabilities of their models, the hacking capabilities, and they really kind of detail, you know, what they would be testing for, what would count as success, how they
• 25:23
  would kind of grade those different evaluation methods.
• 25:27
  And I think that is a really valuable way to do it, that they've kind of put that out into the public to say, to allow other experts to go in and say, you know, this makes sense,
• 25:35
  this doesn't make sense.
• 25:37
  Because in a lot of other cases, the companies are developing those kinds of evaluation methods in -house and determining in -house how to interpret them.
• 25:46
  So yeah, ideally, we would have super well -grounded, well -validated empirical science and theoretical science to guide this, given
• 25:55
  the technology is charging ahead without that.
• 25:57
  I think we just sort of do the best that we can and the more that that can be happening in the opener in consultation with independent experts so that assumptions can be checked and
• 26:07
  they can be improved as we go, the better.
• 26:10
  And with regard to the role of government here, you suggested in your TED talk that policymakers should focus on adaptability rather than certainty in AI regulation.
• 26:20
  What exactly does that mean and how does that connect up with what
• 26:23
  Yeah, I think it connects to a lot of what we've discussed already.
• 26:26
  I think to me, kind of two elements, if you think of sort of progress in AI as driving a car, I think a lot of discussions can get fixated on, you know, should we be hitting the
• 26:38
  accelerator to kind of drive progress as fast as we can, or should we be freaking out and slamming on the brakes?
• 26:43
  I think that's a really impoverished way of thinking about our options.
• 26:46
  And so the kinds of things I think we should be focusing on, I think are maybe more analogous to.
• 26:51
  making sure we have a really good view out of the windshield.
• 26:54
  It's not foggy that our headlights are on, et cetera.
• 26:57
  And making sure that we are really good at steering, basically.
• 27:00
  And so what that means for AI is a lot of things that we've already talked about.
• 27:03
  The clear view out the windshield to me is things like tracking incidents in the real world so that we know how AI is causing harm.
• 27:10
  It's things like improving our measurement science for AI so that we know we have a much better sense of, OK, we can run these tests or we
• 27:17
  pull apart this model, reverse engineer this model in this way and understand what's going on inside it.
• 27:23
  And the steering is, think, making plans for if we do observe, if this is something we do observe, then what do we do about it?
• 27:32
  So that's sort of in the responsible scaling policy type approach of, OK, well, if we have a model that is this good at hacking, then we need to have these cybersecurity protections
• 27:42
  in place around it.
• 27:43
  Or if we have a model that would give
• 27:47
  If it was stolen by an external actor, would give them the ability to commit massive bio attacks or something like that.
• 27:52
  Then we need to have these sorts of protections in place around it.
• 27:57
  So yeah, think there's a lot that we can do when I think about adaptability.
• 28:03
  To me, that means both being able to determine, being able to tell as early as possible what trajectory are we actually on, and also then being able to respond to that evidence
• 28:13
  as it comes in.
• 28:16
  And what about the kinds of geopolitical concerns where China in particular is often pointed to as aggressively pushing forward on these technologies and has a lot of
• 28:29
  government involvement and there's already restrictions being put into place in the United States.
• 28:34
  Given though that so much of the technology is open source or it's based on architectures, they're in public research.
• 28:41
  Is the genie fully out of the bottle in terms of how other actors who may be hostile to what companies and the government of the United States think are able to exploit the
• 28:52
  technology?
• 28:54
  I don't think the genie is fully out of the bottle.
• 28:56
  think maybe the genie's arm is out of the bottle or maybe like their head and part of their torso or something.
• 29:03
  But it really depends on how the technology progresses.
• 29:07
  Again, like many of these questions, it seems like right now the best estimates I've seen make it look like kind of the open source models if we're talking about kind of general
• 29:19
  purpose, large language models, multimodal models.
• 29:23
  And we're looking at the very most advanced versions of those.
• 29:26
  It looks like the open source is something like one and a half to two and a half years behind the best companies, on how you count and a bunch of other assumptions.
• 29:36
  And is one and a half to two and a half years, is that far or is that really close?
• 29:42
  Depends a lot, I think, on what you expect.
• 29:43
  So if you talk to people inside companies like OpenAI or Anthropic,
• 29:48
  they'll tell you like, we might have human level AI in two years, three years, four years.
• 29:53
  In that case, know, a one and a half to two and a half year buffer is a lot.
• 29:56
  If you talk to someone who expects more that, actually this paradigm that we're using, the kind of scaling transformers, large language models, just pouring more data and compute,
• 30:04
  that's kind of, you know, we've kind of like squeezed most of the juice out of that particular lemon.
• 30:09
  Then perhaps one and a half or two and a half years behind is nothing.
• 30:12
  So it does depend a little bit on the trajectory.
• 30:15
  I tend to think that there's, that that's a pretty meaningful gap.
• 30:19
  But it also depends on kind what you're trying to do with it as well.
• 30:22
  Because in some cases, for plenty of use cases, you don't necessarily need the most advanced models.
• 30:27
  so a lot of my work, I work at a center at Georgetown called the Center for Security and Emerging Technology.
• 30:35
  We focus on AI and national security.
• 30:37
  And so if you're thinking about AI in a military setting, things like onboard cameras for drones and kind of automated processing that they're doing, that's not a cutting edge.
• 30:48
  large language model, that's a totally different type of system.
• 30:52
  Or if you're thinking about China's use of AI for surveillance, again, totally different type of system.
• 30:56
  I think it really depends on which specific concerns you're wanting to focus on.
• 31:03
  And let me just throw out one other concern that people talk about, which is the concentration of power.
• 31:07
  We have a small number of these frontier labs that have access to the compute and the data and the researchers and so forth.
• 31:16
  Is there any hope of a more decentralized environment for cutting edge AI?
• 31:22
  Yeah, again, I think it will depend on how the technology progresses.
• 31:26
  I think to the extent that we're in this, we continue in this current paradigm of bigger is better and you need more compute and you need to be spending, you know, the public
• 31:34
  estimates we have are that kind of the current generation of most advanced systems.
• 31:39
  know, GPT -4, Gemini 1 .5, Claude, you know, 3, 3 .5.
• 31:45
  These systems cost sort of in the ballpark of a hundred million dollars to train.
• 31:49
  And that's just a hundred million dollars.
• 31:51
  potentially a huge amount of that is just for paying for the computational power, the enormous, enormous data centers with the most cutting, thousands of the most cutting edge
• 32:00
  chips.
• 32:02
  That's not very accessible.
• 32:03
  That's sort of a naturally centralizing force if you have something so capital intensive.
• 32:09
  And so to the extent that, and those companies, as far as we can tell, in many cases are continuing to invest in scaling that up.
• 32:14
  So going from $100 million to train a single system to potentially a billion dollars to those conversations about
• 32:20
  10 billion or more than that.
• 32:23
  So to the extent that trend continues, that is a very naturally centralizing force.
• 32:29
  At the same time, you are also seeing over time, there's a of a strange dynamic at the frontier of these general purpose AI systems where it is both the case, the last sort of
• 32:40
  five or 10 years, it is both the case that the most advanced systems at any given point of time are the largest and the most expensive to develop.
• 32:49
  And also over time it gets cheaper and more accessible to build those advanced systems.
• 32:54
  So there's sort of a natural centralizing at the very frontier of the field and also a natural decentralizing over time if you're looking at a certain level of capability.
• 33:04
  So if you're looking for a system, for example, that is like, you know, what's a benchmark?
• 33:08
  That is like a, you know, as good as a software engineer intern, then the first company to develop that is gonna be very expensive.
• 33:17
  probably going to be in a small number of hands.
• 33:18
  But over time, the number of people that have access to AI that is as good as a software development intern is going to go up a lot.
• 33:27
  again, I think remains to be seen depending on the trajectory of the technology, whether those centralizing dynamics sort of, think at the moment they tend to prevail because the
• 33:35
  most advanced systems do tend to be more useful and significantly more sophisticated than the most advanced systems.
• 33:41
  But if that slows down a little bit or if something else changes.
• 33:45
  then it could be that those decentralizing dynamics of over time, the costs fall and the accessibility rises, that might end up mattering more.
• 33:53
  Should governments try to put their thumb on the scale to promote more decentralization?
• 33:59
  I think to some extent, I think there is interest.
• 34:04
  I think it's not necessarily the right task for governments to be trying to be competitive at that very most expensive level.
• 34:14
  But I think to the extent that governments can do things like in the US, you have the National AI Research Resource, which currently is just in pilot mode, but there's a lot of
• 34:22
  interest in kind of scaling it or establishing it more firmly.
• 34:25
  And I think similar initiatives underway elsewhere, where the idea is
• 34:31
  provide some computing power to researchers or to civil society or to universities or to others, potentially to provide data sets in a way that makes them more accessible.
• 34:39
  I think there's people thinking hard about different models for what gets called public AI.
• 34:45
  What else might you need?
• 34:46
  Might you need expertise?
• 34:47
  Might you need training?
• 34:50
  What are the different ingredients that you can make available?
• 34:51
  I think there is definitely a role for governments to play in making that of broader suite of access to AI opportunities more widely available.
• 35:00
  But yeah, I would distinguish that from trying to compete with these mega tech companies, which I think is probably not the right use of resources at this point in time, especially
• 35:10
  also given that it's not just a matter of resources, but also talent.
• 35:14
  And it's going to be very difficult to compete with industry to get the concentration of talent you need and so on.
• 35:19
  So I think not yet on that particular question.
• 35:23
  Okay, and final thing is, let's say I'm at a Fortune 500 company that's investing in deploying AI, but doesn't have the resources or the capabilities to develop our own
• 35:34
  foundation model.
• 35:34
  So we're using an open AI or Gemini or something like that.
• 35:39
  How should I be thinking about these kinds of questions about regulation and AI safety and the kinds of concerns you've been discussing?
• 35:47
  Yeah, I think it depends on the industry and the use case.
• 35:50
  And generally, if for any given Fortune 500 company, they will tend to know their own industry and their own use case much better than I can.
• 35:58
  But I think thinking through what kinds of documentation do you need to be keeping, what kinds of potential regulatory scrutiny could you come under?
• 36:06
  Maybe right now, it's not clear that your AI systems are subject to that.
• 36:10
  But if it is a rights or safety -affecting use case, maybe it's worth.
• 36:14
  anticipating that it could be in the future and starting to document sort of what you're doing and why.
• 36:20
  More broadly, I think if you're using a model developed by someone else, it's worth digging as much as you can to get as much information about that model, how it works, how
• 36:29
  it was tested, what data it was trained on.
• 36:31
  Different companies will give you different levels of access into that information.
• 36:35
  And I think it is very relevant and therefore worth looking
• 36:39
  And then certainly, mean, know, Fortune 500 company will have a sophisticated legal department, so I'm sure they'll be onto this.
• 36:44
  looking very, very closely at the terms that are being offered, is there sort of some kind of indemnification or release of claims around, you know, if something goes wrong, who is
• 36:54
  at fault?
• 36:55
  I've definitely heard stories of some of the providers of these models trying to contractually, you know, push all of the liability onto the B2B customer as opposed to
• 37:07
  having
• 37:07
  sit with the model developer.
• 37:09
  So just keeping an eye on those kinds of things and being willing to push back or to potentially consider a different provider if you're not satisfied with what you're
• 37:19
  getting.
• 37:21
  And maybe one last piece would be just to take seriously the possibility that we might see this continued progress over the next five years or so of continuing to have more
• 37:32
  sophisticated models.
• 37:33
  And so thinking about in your product development or in your usage of the models,
• 37:37
  What might it look like?
• 37:38
  What might it mean for your use case if the underlying model is getting more sophisticated?
• 37:43
  Is that something you can just easily roll up and make use of and squeeze more juice out of?
• 37:47
  Or is there a world where that would actually undercut your product, for example?
• 37:51
  I think there's been a really common dynamic in AI over the last five or 10 years has been that at any given point in time, you can maybe create sort of a specialized model to do a
• 38:03
  specialized thing, but then maybe a year or two later,
• 38:06
  it could be pretty straightforward to develop a more general model that just is better than your specialized model without trying very hard.
• 38:14
  So just being aware of those kinds of dynamics in the field, hopefully clear at this point in the conversation, I think there's a ton of uncertainty about whether that will
• 38:22
  continue, what it will look like.
• 38:24
  But being aware that that's a possibility and that a lot of experts do expect that sort of continued wave of progress and thinking about how could you benefit from that as opposed
• 38:31
  to being vulnerable to
• 38:33
  Good, well that's a perfect way to end being on the right side of uncertainty.
• 38:37
  It's been a fascinating conversation.
• 38:39
  We've covered a lot of ground.
• 38:40
  Helen, thank you so much for joining
• 38:41
  Thanks so much for having