Latest / The Road to Accountable AI / Elham Tabassi: Framing and Taming AI Risks

Transcript

Elham Tabassi: Framing and Taming AI Risks · The Road to Accountable AI

• 0:01
  Elham Tabasi, welcome.
• 0:03
  Glad to have you on the podcast.
• 0:05
  Thanks for having me.
• 0:06
  Glad to be here.
• 0:08
  But first, why don't you tell us what NIST is, because it's an important agency, but
• 0:15
  one that maybe people are not necessarily familiar with, even if they know something
• 0:20
  about the US government and AI policy.
• 0:24
  Oh, yeah, happy to.
• 0:25
  And that's a great starting point.
• 0:28
  So NIST, or National Institute of Standards and Technology, is a non
• 0:33
  -regulatory agency.
• 0:34
  We are under the Department of Commerce.
• 0:37
  And NIST was established back in 1902 with a unique mission to advance US innovation
• 0:43
  and industrial competitiveness.
• 0:46
  Here at NIST, we have a very broad portfolio of research.
• 0:50
  from building the most accurate atomic clock to cybersecurity to artificial
• 0:55
  intelligence, advanced communications, but most importantly, a long tradition of
• 1:00
  cultivating trust in technology.
• 1:02
  And we do that by advancing measurement science and standards, measurement science
• 1:06
  and standards that makes technology more reliable, secure, private, fair, or in
• 1:12
  other words, more trustworthy.
• 1:14
  And that's exactly what we have been doing in the space of AI.
• 1:18
  Absolutely.
• 1:19
  And so you were one of the architects of the NIST AI Risk Management framework,
• 1:23
  which came out in January of 2023.
• 1:27
  Other than the general goal of trust and technology that you just described, where
• 1:31
  did that project come from?
• 1:32
  How did that come about?
• 1:34
  Yeah.
• 1:36
  So AIRMF that was released a little over a year ago, it was directed by congressional
• 1:45
  mandate.
• 1:46
  It is a voluntary framework for managing the risk of AI in a flexible, structured,
• 1:52
  and measurable way.
• 1:56
  The whole point is that
• 1:58
  We harness the power of AI for good and AI can serve all people in a beneficial, fair
• 2:04
  and equitable manner.
• 2:07
  These three attributes that I talked about, flexible, structured and
• 2:11
  measurable, we wanted to be flexible to be able to keep up the pace with this rapid
• 2:18
  change of the technology, but also so that the different organizations...
• 2:23
  with different resources be able to implement that.
• 2:27
  We didn't want to be a heavy thing that only the big organizations that have AI
• 2:33
  governance or a lot of people doing that be able to operationalize it.
• 2:38
  Structured in the terms that when we start working on AI RMF, the concept of the
• 2:44
  trustworthiness, the question of what constitutes trust were not quite answered
• 2:49
  when there has been a lot of really high value
• 2:52
  high -level value -based documents that talk about these things.
• 2:59
  None of them try to get those principles into sort of the building blocks that the
• 3:06
  developers and the employer of the technology can use that.
• 3:09
  So working with the community, we try to come up with unpacking the concept of the
• 3:13
  trustworthiness, and come up with the components of the AI system that's
• 3:18
  trustworthy, which those characteristics are.
• 3:22
  valid and reliable, safe, secure and resilient, privacy -enhanced, explainable
• 3:28
  and interpretable, fair with harmful bias, managed and transparent and accountable.
• 3:34
  Bring the community on a shared understanding around each of them to get
• 3:40
  to the next attribute of the measurable to have a good understanding of what we want
• 3:46
  to measure to be able to measure them.
• 3:48
  So the importance of being able to...
• 3:52
  define trustworthiness into technical and socio -technical terms that developers of
• 3:58
  the technology, deployers of the technologies, users of the technology can
• 4:02
  understand, and then build a community together to come up with the scientific
• 4:06
  underpinning for how to measure those because if you're really serious in the
• 4:11
  business of improving things, first we have to be able to measure them.
• 4:14
  We can't improve things that we cannot measure.
• 4:18
  So I want to get back to that question about measurement because it is really
• 4:21
  critical.
• 4:21
  But first, let me just ask you, that's a big challenge talking about
• 4:25
  trustworthiness around AI and all those principles you described.
• 4:28
  They make sense.
• 4:29
  But how did you and the team get your arms around that?
• 4:32
  How to decide what kinds of aspects to focus on and how to operationalize these
• 4:37
  broad principles?
• 4:39
  Great question.
• 4:40
  So how we do our work in this case and also in many other cases is to work
• 4:46
  collaboratively with public and private stakeholders.
• 4:49
  So basically everything that we do, we start with a formal process of request for
• 4:54
  information.
• 4:55
  That's where the government put an announcement into a federal registered
• 4:58
  notice, announce to the whole public that we're doing this and ask for comments.
• 5:03
  Then we work with the community to do rounds of workshops, listening sessions.
• 5:08
  to get answers to all of these questions.
• 5:13
  And in doing so, we very early on realized that we should definitely listen to a
• 5:22
  community with the expertise that are building the technology or designing the
• 5:25
  technology, those that have the expertise in computer science, mathematics,
• 5:29
  statistician, but also the community that study the impact of the technology,
• 5:34
  psychologists, sociologists, philosophers.
• 5:38
  but also the community that are going to be in charge of either top down or bottom
• 5:43
  up, try to operationalize and implement it and figure out how it basically fits into
• 5:48
  the bigger risk management of the organization, so business groups.
• 5:54
  And we also did listening sessions with the AI users, one of our best listening
• 6:00
  sessions when we went to community colleges, when we went to minority serving
• 6:05
  institutions and talked with the students.
• 6:07
  So by getting a lot of input from the community, input from a very diverse sets
• 6:14
  of backgrounds and expertise, we came to where we are in terms of the
• 6:23
  trustworthiness characteristics, but also the whole document.
• 6:27
  Doesn't that broad amount of input though make it a challenge to come up with a
• 6:31
  consensus about what direction to go, what the principle should be?
• 6:35
  It does.
• 6:36
  And I think that's part of the reasons that we want to do that, because it is
• 6:40
  really important to keep all of the voices included.
• 6:43
  And when you do that, and if you really go for the diversity, of course we're going
• 6:48
  to hear inputs both sides of the spectrum and trying to figure out the right space.
• 6:56
  And I don't want to say middle ground, because there's a lot of the things to
• 6:58
  consider.
• 6:59
  You want to be scientific, high scientific integrity and scientific validity, right?
• 7:04
  So we want to make sure that whatever we put there is technically robust and
• 7:08
  empirically validated.
• 7:09
  It is operationalizable and implementable today, but also it doesn't get expired in
• 7:16
  a very short amount of time.
• 7:20
  So we heard when we start this process from everything and everywhere, from the
• 7:27
  civil society groups that wanted everything to be absolutely
• 7:31
  public and open and audited and a lot of vigorous tests and transparency
• 7:38
  requirements, a lot of really good reasons for that.
• 7:41
  And we also hear from the technology developers, importance of the IP and
• 7:45
  innovations and how those things have to be protected.
• 7:49
  But that's the beauty of this, when you give everybody the voice and we are
• 7:55
  objective and want to convene.
• 8:00
  try to make value for every participant in the discussions to see how those consensus
• 8:08
  are being built.
• 8:09
  And the other thing is that when we get input from a very vast group of the
• 8:15
  community, NIST has the pen in writing this.
• 8:18
  So we would kind of make a decision where we have to go.
• 8:21
  Of course, we go out and explain and make sure that to the extent possible, we have
• 8:27
  the buy -in.
• 8:27
  Those buy -in are important because that's
• 8:29
  That's where the adoptions happens, but that is part of the process, building the
• 8:34
  consensus.
• 8:36
  How can we measure the trustworthiness or responsibility of an AI system?
• 8:44
  That's a very difficult, you know, that's a technical challenge.
• 8:47
  That's a question that the community is looking at this.
• 8:50
  It's difficult at many different dimensions.
• 8:54
  So our institutions, our work, the group that I have been a part of this, and by
• 8:58
  the way, I joined this in June 7, 1999.
• 9:02
  So in a few months, it's going to be my 25 years and all my career here, I've been
• 9:06
  working on computer vision, machine learning type of efforts.
• 9:13
  and a lot of tests and evaluations.
• 9:16
  So if some of your listeners are familiar with the face recognition tests or
• 9:20
  fingerprint work that we have done, I've been part of that team doing that type of
• 9:25
  evaluation.
• 9:26
  So with a rich history and experience and understanding of how to do evaluations,
• 9:33
  evaluations of the AI system poses a new set of questions and challenges.
• 9:38
  One is, of course, what we mean by
• 9:41
  safe AI, what we mean by secure AI.
• 9:44
  But even if we can come up with some understanding of those, AI systems are not
• 9:52
  just about data computing algorithm.
• 9:57
  There are complex interactions of the data computing algorithm with the environment
• 10:01
  they're operating in, the human that operates that.
• 10:04
  So the solutions and the answers are not only technical and technology.
• 10:08
  but they all had to be answered through the socio -technical lens.
• 10:12
  What that means is that the evaluations that we have done before to get systems
• 10:17
  out of their context of use, bring them to the laboratory, run the test data through
• 10:22
  them and test them, and those are important.
• 10:24
  We have to do them.
• 10:25
  They give us some baseline of the capabilities and limitations and failure
• 10:28
  modes, but they are not going to be sufficient to surface all the sorts of the
• 10:34
  problems, negative consequences that can come.
• 10:36
  out of the AI systems.
• 10:38
  We are hearing a lot about the red teaming of AI systems with spot checking and
• 10:43
  basically trying to see how AI systems can get into undesirable behavior.
• 10:51
  But beyond those things, we believe that another set of testing where AI systems
• 10:57
  are being tested in their native context of use with the humans, individuals that
• 11:04
  are
• 11:04
  going to use it, it's really important.
• 11:06
  We call them field testing.
• 11:07
  We have seen examples of that in sort of, you know, pharmaceutical.
• 11:11
  FDA is doing this.
• 11:12
  So it's, there are all these difficulties of exactly what we are testing for.
• 11:20
  And then the spectrum of the test that we are doing is not going to be enough.
• 11:25
  The laboratory testing is not going to be enough.
• 11:27
  We have to augment it with other testing.
• 11:29
  And then the...
• 11:32
  The other thing is that we are not going to be able to test multi -purpose AI
• 11:36
  systems for all of the different use cases that are being used.
• 11:40
  So how to scale all of these things?
• 11:42
  So a difficult problem.
• 11:44
  It is something that we have been working on since the release of the AI RMF.
• 11:48
  One of the four functions on AI RMF is the measure function.
• 11:52
  AI RMF provides, in addition to giving the structured approach to what we mean by
• 11:58
  trustworthiness, the meat of that is...
• 12:00
  72 recommendations categorized in four functions of map, measure, manage, and
• 12:06
  govern to how understand and identify AI risks, how to measure them based on those
• 12:12
  measurement results, what type of risk responses and mitigations are to be
• 12:17
  implemented, and the govern functions give recommendations basically for the
• 12:23
  processes and procedures, all of the embeddings that's needed for the right.
• 12:28
  risk management processes.
• 12:32
  So after the release of the AIRM, we put a lot of our effort into the measure
• 12:36
  functions, which quite aligns and become the foundation work that our latest
• 12:44
  assignment as part of the EO is built on top of that.
• 12:47
  So a lot of work is happening as part of the task that we have with the EO as part
• 12:52
  of the AI Safety Institute.
• 12:55
  We are standing up.
• 12:57
  and its consortium to work with the community on figuring out the evaluations.
• 13:01
  The last thing I will say in this space is that it's a tough problem and we are not
• 13:06
  going to get the answers in the first go.
• 13:08
  So we have to be on the understand that it's going to be iterative and as we do
• 13:13
  more of this research, run evaluations, you know, write guidelines based on those
• 13:17
  guidelines, conduct evaluations.
• 13:19
  The evaluations will tell us that what was the gaps in the guidelines, the evaluation
• 13:24
  can also say the gaps in the
• 13:26
  So hopefully we are going to generate a positive feedback loop that the
• 13:31
  measurement science is getting strengthened by empirical evidence and
• 13:38
  science is going to be built around that.
• 13:40
  But also the practice and adoption of the AIs are going to get stronger because of
• 13:45
  the test results.
• 13:47
  Can you talk about some of the directions that you and the community are going to
• 13:51
  try to come up with those benchmarks for some of those issues?
• 13:56
  All right, so October 30th, Executive Order 14110, Safe, Secure, and Trustworthy
• 14:03
  AI was released.
• 14:06
  It's a very comprehensive set of instructions to many of the federal
• 14:11
  government agencies on how to promote safe, secure, trustworthy AI.
• 14:16
  Section 4 on safety and security tasks several agencies, but predominantly NIST.
• 14:23
  on advancing the measurement for and evaluations for safe, secure, trustworthy
• 14:30
  AI systems.
• 14:31
  In particular, it directs NIS to develop guidelines for evaluations, reteaming,
• 14:42
  safety, cybersecurity.
• 14:44
  It asks NIS to facilitate development of consensus -based standards.
• 14:49
  It also asks NIS to provide testing environment for
• 14:52
  evaluations of the AI systems.
• 14:54
  These guidelines and infrastructures will be voluntary resources for the community
• 14:58
  to use.
• 14:59
  So right now we had our heads down drafting the documents, which we are
• 15:04
  hoping to go out for public comment.
• 15:07
  Based on the public comment, we're hoping to improve it to meet the deadlines of the
• 15:10
  EO.
• 15:12
  And the EO gives us a 270 days deadline that gets us to the end of the July.
• 15:19
  But all of these could work.
• 15:20
  All of these actions that the EEO sets in the motion, we think that should not and
• 15:25
  cannot stop in end of July.
• 15:27
  So we're looking into the AI Safety Institute and its consortium to give a
• 15:34
  long lasting approach to all of these efforts in the research and evaluation
• 15:38
  builder.
• 15:40
  So what we are working with the community is first develop the guidelines that we
• 15:44
  have asked through the EEO, working through the consortium with the consortium
• 15:48
  members.
• 15:49
  when it's out to refine them, improve them, conduct evaluations, and based on
• 15:56
  those kind of lived experiences, improve the guidelines.
• 16:00
  Eventually, we definitely look like to have sort of standardized test
• 16:06
  environments, standardized methods and metrics and methodology for testing so
• 16:11
  that the community can use them and help with the scaling of the evaluations that
• 16:16
  needed to be done in this space.
• 16:19
  Have you seen good examples of either private sector companies or public sector
• 16:25
  organizations taking the RMF and operationalizing it?
• 16:29
  We have heard a lot and we are really grateful for that.
• 16:33
  So when the AIRMF was released January 26, 2023, a lot of different companies
• 16:40
  voluntarily and entities, I should say, gave us some sort of testimonies on how
• 16:45
  they are planning to use AIRMF.
• 16:48
  And that again goes back to building that engagement because when it comes out,
• 16:52
  everybody sees themselves in it.
• 16:54
  And then since then, we have heard from...
• 16:57
  large medium, small size businesses that are using it.
• 17:02
  Some of those have given us, we call them use cases, how they're using it
• 17:07
  internally.
• 17:07
  Some of those are posted on our website.
• 17:10
  Some of the other one has been talking to us and letting us know that, particularly
• 17:14
  for the larger organizations, how they are aligning their internal processes with AI
• 17:19
  or MF.
• 17:19
  And we're always happy to work with the smaller size businesses to...
• 17:24
  help them with the operationalization.
• 17:26
  So one of the things, for example, we did in March of 2023, we put out the
• 17:32
  trustworthy AI resource center.
• 17:34
  So in addition to the release of the ARRMF, with the goal of keeping ARRMF, we
• 17:40
  call it as evergreen as possible, with a revision cycle of three to five years, we
• 17:46
  put out a playbook out that basically gives a lot of...
• 17:51
  informative references and information and descriptions on exactly what to do for
• 17:56
  each of those 72 recommendations, what they can do today, what are the tools and
• 18:02
  standards that can be used today for implementations.
• 18:07
  So the playbook, we put a revision cycle of every six months so we can keep it up
• 18:12
  with the latest changes in the technology as tools and information becomes
• 18:19
  available.
• 18:20
  We...
• 18:21
  We made sure that this is sort of interactive, filterable based on the role
• 18:26
  of the AI actor, designer, developer, deployer, the area that they want to get
• 18:31
  information out from bias or privacy.
• 18:35
  And we're also working with the community to add more tools and support, supportive
• 18:44
  documents, informative references, implementations that can help with the
• 18:48
  implementation of that.
• 18:49
  But across the board, we have seen...
• 18:51
  It's being used.
• 18:52
  Another interesting or welcoming happening is that at the time of the release of the
• 19:00
  AI RMF, we also released Crosswalks.
• 19:04
  So AI RMF was not the first document being out, and we want to definitely make sure
• 19:08
  that we build based on the knowledge of the community and leverage.
• 19:13
  For example, the definition of AI system in the AI RMF is based on the OECD
• 19:18
  definition.
• 19:20
  So if we...
• 19:21
  based on the input of the community.
• 19:22
  We improved OECD definition and then OECD took some of our improvement, put it back
• 19:27
  into their definitions, definition of AI systems.
• 19:32
  So we want to make sure that we are not adding confusion to the market by yet put
• 19:38
  another document out.
• 19:39
  So we put out crosswalks that says that, for example, how AI RMF relates to OECD AI
• 19:45
  recommendations, the draft of the EU AI Act.
• 19:49
  few weeks ago, but at that time it was a draft with some other documents that's
• 19:54
  coming out of the White House, but also with ISO, the standards that were coming
• 20:00
  out of the ISO international standard organizations.
• 20:04
  ISO Insights is the technical tag to ISO, US technical tag to ISO.
• 20:10
  Basically pick that up and as ISO is coming up with other standards, they
• 20:14
  develop crosswalks with AI RMF.
• 20:17
  And hopefully all of these things that...
• 20:19
  I think is an indication that AI -RMF are being used, so that's why they need those
• 20:25
  crosswalks, but also hopefully bringing more semantic interoperability in the
• 20:31
  short run.
• 20:34
  You mentioned the executive order and the AI Safety Institute that was one of the
• 20:39
  things that was created out of that process and you're now taking over as CTO
• 20:43
  of that USAI Safety Institute.
• 20:45
  How is that different from the work that NIST has done before and what's on the
• 20:49
  agenda for the Institute?
• 20:52
  Yeah, we look at both the EO and the Safety Institute as basically, that's the
• 21:04
  phrase I use, supercharging our efforts in advancing trustworthy responsible AI.
• 21:11
  So I think in terms of the goal and vision to advance the science, practice, and
• 21:19
  policy of AI systems,
• 21:20
  increasing adoptions of the AI is the work that we are doing.
• 21:25
  And all of these things are giving us more capacity and strengthens our engagements
• 21:34
  to do better and more.
• 21:36
  Eventually, we also want to keep an eye on the, or help with the adoption.
• 21:41
  So as a measurement science agency, we want to make sure that the right
• 21:45
  measurement science tools, standards, guidelines are there.
• 21:49
  for anybody that wants to design, develop, or use AI systems.
• 21:53
  And hopefully, by doing this, we actually help the adoption rates of AI in science,
• 21:59
  in health, climate, agriculture, all of those areas that we think that that's
• 22:06
  where AI technology can bring many, many benefits.
• 22:13
  What do you think the biggest gap is right now or the biggest area where we haven't
• 22:17
  seen enough progress on some of the elements that are going to be necessary to
• 22:21
  promote that?
• 22:23
  I might be biased coming from a measurement science agency, but I would
• 22:28
  say measurement and evaluations.
• 22:30
  I think everybody wants to save, secure, trustworthy AI, but we don't know how to
• 22:34
  measure them.
• 22:34
  We don't have the metrics there.
• 22:36
  We don't have the methodologies there.
• 22:39
  There are documents, policies they like to talk about, assuring that AI systems are
• 22:48
  safe.
• 22:48
  You know, we saw it in the EU AI Act.
• 22:50
  They want to see...
• 22:51
  some sort of assurance or conformity assessment, but those specifications
• 22:56
  hasn't been written yet.
• 22:58
  So what do we mean by safety?
• 23:03
  And policies usually stay at the abstract high level that AI should be safe or non
• 23:09
  -discriminatory, but bringing it down, getting everybody on the same page, that's
• 23:14
  what it means, and how to assure for them.
• 23:17
  I think that's the biggest, that's one of the biggest gaps.
• 23:22
  In general, in this AI space, I will say that we know a lot less than what we
• 23:30
  should.
• 23:31
  So anything that we can do to be able to know more what to expect, how to
• 23:39
  characterize the behavior functionality of the systems, these are the good partners.
• 23:47
  How much are those really scientific questions though?
• 23:51
  And part of what I'm thinking about is you get to some issues like bias, where
• 23:55
  there's various different tests about algorithmic fairness, but some of them are
• 23:59
  fundamentally incompatible.
• 24:00
  And you mentioned the socio -technical aspects of the systems where there may be
• 24:04
  things that are not just about yes or no.
• 24:07
  So how do you and others in the community work to identify what it is where we can
• 24:14
  actually get an answer through the kind of science you're talking about?
• 24:17
  Yeah.
• 24:18
  Thank you for that question.
• 24:20
  So back in March 2022, we put out a document out, special publication 1270.
• 24:29
  It has a name towards standards for understanding harmful bias in AI systems.
• 24:36
  But these are some of the foundational work that we're doing with the community
• 24:40
  that.
• 24:41
  Again, as you said, there are tools for bias, but what are really measuring?
• 24:45
  Are they measuring the right thing?
• 24:46
  What should we really measure?
• 24:48
  What is bias across the life cycle of AI anyway?
• 24:51
  So that document talks about a lot of these things.
• 24:54
  And if I want to summarize, the most important to me, the most important point
• 24:58
  of that document is that bias is not just about disparity and demographic disparity.
• 25:07
  A lot of us, when you're talking about
• 25:09
  bias, the first thing that comes to our mind is demographic disparity.
• 25:12
  The document talks about computational and statistical bias as one type of bias, but
• 25:19
  there are also other types of bias, systemic biases.
• 25:22
  If you go and do a statistically valid data collection of the community and
• 25:28
  society today, that's going to be biased and the biases baked into the society will
• 25:34
  get into the systems because our training data will carry those biases.
• 25:39
  It also talks about another type of bias, which is cognitive biases.
• 25:43
  How the different individuals process information given by AI systems or
• 25:49
  interact with AI systems.
• 25:51
  And based on that, how we react to those is different.
• 25:54
  So there's biases there too.
• 25:56
  Understanding all of these types of biases, having a wholesome understanding
• 26:02
  of now in this topic what the bias is and...
• 26:05
  how biases showed itself at the different stages of the AI lifecycle become really
• 26:11
  important on other steps of the evaluation.
• 26:16
  So the science part of that, again, becomes to having a sort of a wholesome
• 26:21
  approach to what could be the sources of the biases, where they're coming from, and
• 26:27
  that information can help us to do the measurement.
• 26:32
  You asked the question about the socio -technical.
• 26:35
  So all of these problems are socio -technical.
• 26:38
  So there is not just a technical or technology solution for them.
• 26:41
  We just talk about cognitive science biases, for example.
• 26:46
  And the fact that one of the components of the responsible AI is human centricity.
• 26:54
  So you have the humans there.
• 26:57
  And that gets to another challenge that we have with building the science.
• 27:01
  for the measurement that a lot of things that we are used to on, for example,
• 27:08
  measuring length, mass, electricity, those areas that NIST work on coming up with
• 27:15
  standards for measuring them at the turn of the last century, which were critical
• 27:21
  for the underlying technologies of that time.
• 27:25
  Right now we have...
• 27:27
  quantitative metrics to measure them, intropable ways of measuring them.
• 27:33
  We are at the distant of technology, we are trying to do the very same thing for
• 27:38
  the technology of our time.
• 27:42
  As you mentioned, some other jurisdictions, the European Union in
• 27:45
  particular, have now a comprehensive legislation.
• 27:52
  states.
• 27:53
  Talk a little bit about what the function is of the work that NIST does in that
• 27:59
  context where there aren't necessarily specialized regulations.
• 28:03
  So NIST is a non -regulatory agency.
• 28:10
  We don't work with regulators.
• 28:13
  None of the things that we put out is for mandatory use.
• 28:17
  Everything is for voluntary use.
• 28:18
  And even the evaluations that we do while we evaluate for, for example, accuracy or
• 28:25
  security or privacy or any of these things.
• 28:28
  We don't make any judgment on how safe is safe, how private is private, how accurate
• 28:33
  is accurate, because there is no one size fits all and it depends on the context of
• 28:37
  use.
• 28:38
  What we are trying to do and our work with the policy makers is to provide empirical
• 28:44
  evidence to help with development of the policies that are clear and also make sure
• 28:51
  that the underlying science and technical tool and standard exists.
• 28:57
  to help with the enforcement of the policy.
• 29:00
  We just talked about it, that the laws can come and say that AI must be safe, but
• 29:05
  safety means.
• 29:06
  And we see it what happened with the EU Act as well that after its passing, now,
• 29:13
  since then, like European centralization body are tasked with coming up with the
• 29:17
  standards for basically checking the conformity assessment to the requirements
• 29:23
  in the law.
• 29:24
  So...
• 29:25
  So we are not regulatory.
• 29:26
  Everything that we put out is for voluntary use.
• 29:29
  But we give sort of empirical evidence for clear policy making, evidence -based
• 29:37
  policy making, and the technical tools and the measurements and evaluations needed
• 29:44
  and standards needed for enforcement of that.
• 29:48
  I use the word standards several times, and it may be.
• 29:51
  helpful if I clear that while standard is our middle name, we are not standard
• 29:57
  development organizations such as ISO or IEEE.
• 30:00
  There are really good policies in the US in effect and in existence that the job of
• 30:06
  the federal government is to help industry develop the standards.
• 30:11
  So they are in the driver's seat.
• 30:13
  We are supporting their work because, again, we bring those objective technical
• 30:19
  contributions.
• 30:20
  in the development of the standards.
• 30:23
  Back in 2019, we had a task by an executive order to write a plan for
• 30:29
  federal government engagement in development of AI standards.
• 30:33
  With this current EL, we have a task to develop a plan for global engagement to
• 30:40
  promote consensus industry standards.
• 30:44
  And...
• 30:45
  So we work a lot with the community, both industry and standard development
• 30:49
  organizations on promotion and facilitate development of the good standards, but we
• 30:53
  are not developing standards.
• 30:57
  What is your sense about the current state of global cooperation and coordination on
• 31:02
  AI response?
• 31:05
  So a lot of really, a lot of conversations has been happening for the past many years
• 31:11
  that I have been involved and a lot of really good work is happening.
• 31:14
  OECD has, OECD AI Governance Party is doing a lot of good work on bringing
• 31:21
  communities together but also provide a lot of technical clarity for policy
• 31:25
  meeting through a lot of the work that they're doing, a lot of documents that
• 31:29
  they're putting out.
• 31:31
  We saw.
• 31:33
  a really good progress through the G7 kind of code of conduct that try to bring more
• 31:40
  sort of a global interoperability on sort of voluntary commitment that happened in
• 31:49
  the US.
• 31:51
  Bottom line is that global conversations and global interoperability for the AI
• 31:56
  space is really important.
• 31:58
  It is for any type of standards, but particularly in the space.
• 32:02
  The technology is obviously very global and the users are across the globe.
• 32:10
  So we are thinking about really maximizing the benefits of AI while minimizing its
• 32:17
  negative consequences, harnessing the power of AI for good.
• 32:21
  We need more global engagement, both at the research side to build.
• 32:30
  the scientific underpinning on the standard side.
• 32:34
  So we all are kind of using the same standards and also on the implementation
• 32:41
  and operationalization side.
• 32:42
  So we can all benefit from this technology.
• 32:46
  So a lot of really good work is happening and a lot more can be done, I guess.
• 32:51
  Absolutely.
• 32:52
  Well, that's a really great place to wind up.
• 32:54
  Thank you so much for a really fascinating conversation.
• 32:57
  Thanks for having me.
• 32:58
  Enjoy the conversation.