Latest / The Road to Accountable AI / Shameek Kundu: AI Testing and the Quest for Boring Predictability

Transcript

Shameek Kundu: AI Testing and the Quest for Boring Predictability · The Road to Accountable AI

• 0:01
  should make welcome to the Road to Accountable AI.
• 0:04
  Thank you very much for having me, Kevin.
• 0:07
  First, just tell us what AI Verify is and what's the problem that you're trying to address.
• 0:15
  So AIVerify is a subsidiary of uh IMDA, which is a government-owned authority in Singapore.
• 0:24
  It's a not-for-profit subsidiary, and it's focused on a singular problem.
• 0:30
  The ultimate objective is to enable the adoption of AI at scale, but the hypothesis is that there are many barriers to adoption of AI at scale.
• 0:38
  One of them is the lack of adequate, reliable testing capabilities.
• 0:44
  when it comes to AI systems.
• 0:46
  And we try and focus on that specific problem, which is how to make reliable testing accessible at scale.
• 0:55
  What got you personally interested in this area of AI testing?
• 0:58
  You came from the financial sector, from standard chartered.
• 1:01
  I know you worked for a startup before AI Verify, but what made you think this was an important area to focus on?
• 1:07
  Well, actually, funnily enough, the background in financial services where I was group chief data officer at one of the global systemically important banks.
• 1:16
  And the course of that well before machine learning uh became as prevalent as it is in many parts of finance, uh we've always had concerns about the quality of data and the
• 1:29
  reliability of systems that are used for things such as capital calculations or anti-money laundering or anti-fraud initiatives, et cetera.
• 1:38
  So the thinking around we need to make data and algorithms reliable is now more than 11 years old, at least in my personal career history.
• 1:48
  And so it's always been a thing to say, how do we ensure that if more and more of the world is going to be dependent on these things, uh we are able to ensure their reliability
• 1:59
  among other things.
• 2:01
  And so that was the original uh inspiration.
• 2:06
  I then moved, as you mentioned, to a startup that was focused on testing traditional ML systems and later, the Ruby AI systems.
• 2:13
  And when that got acquired, I joined this uh government not-for-profit subsidiary focused on the same space.
• 2:23
  As you mentioned, um this area of testing and data quality and model quality and so forth is not something new, at least in industries like financial services.
• 2:33
  What's different or more challenging today than in the world of model risk management, say for banks?
• 2:39
  Yeah, I think it's a great question.
• 2:42
  think there's a few different dimensions, but the two that come to mind the most are the democratization of models and the lack of boundedness of many of the models, particularly
• 2:58
  with generative AI systems.
• 3:00
  So by democratization, what I mean is that if you go into banking, for example, or insurance for that matter, the number of
• 3:07
  individuals and departments that were actually involved in building models and that were allowed to put those models to work was quite limited.
• 3:16
  And there would typically be departments that had people with a background in quantitative sciences and statistics, etc.
• 3:23
  So those who were worried about model risk could basically look forward to engaging specialists that already knew a lot of these dimensions.
• 3:31
  So it was a relatively narrow
• 3:34
  Now suddenly, well before generative AI, particularly with AutoML in, I would say, late 2010, the second half of 2010, suddenly everybody and their dog was a data scientist,
• 3:47
  right?
• 3:48
  And that meant that you could have a very large number of models, which was frankly great from a very, very, you know, from a democratization of data science perspective, from the
• 3:58
  kind of impact these models could have.
• 4:01
  But it did mean that managing these
• 4:03
  suddenly became a much bigger challenge.
• 4:05
  But that was nothing compared, if you think, to the challenge of managing generative AI systems, because most of them, as you know, they are foundation models, general purpose AI
• 4:13
  systems.
• 4:14
  And so when you implement one of these systems, it is very difficult to put a boundary around what each of these systems might be doing.
• 4:24
  So for example, even if you're picking something um as narrowly bounded as anti-money laundering,
• 4:31
  Is the application you're using solely going to generate a proposed narrative for closing the investigation?
• 4:38
  Or is it also a source of information to the investigator?
• 4:42
  Well, depending on the answer to the question, you have to manage different risks.
• 4:45
  So I think these two factors, among many other things, of course, with generative AI, an additional factor is the fact that the data involved is no longer just structured data.
• 4:54
  It's a lot of unstructured data as well.
• 4:56
  And all of these three factors technically have made a significant difference to the
• 5:00
  surface area of risk.
• 5:04
  For someone who's not familiar, can you explain just broadly speaking, what does mean to test the generative AI model?
• 5:12
  So I can but actually to be fair we focus a little bit more on the generative AI application itself and there is an important distinction Yeah, there's an important
• 5:22
  distinction, which is the model can work or not work within a set of conditions But in most real-life implementations whether it's in a bank or a hospital or an airport, etc
• 5:34
  That model is embedded into a broader end-to-end system.
• 5:37
  And so if you look at the potential points of failure
• 5:40
  It's not just the model itself, it's the data that feeds into the model, it's the linkages between the various non-AI components in the system, etc.
• 5:48
  Having said that, think if you think about the focus area of uh what...
• 5:53
  uh Sorry, if you think about testing itself, in effect, you are bombarding the system, the application, with a series of inputs, which in many cases, if it's a chatbot or any kind
• 6:07
  of...
• 6:08
  interface like that, it's a bunch of prompts, and you're evaluating the answers that come out of the application for a variety of risks that are relevant to your use case.
• 6:18
  So for example, if it's a hospital that's using it to, I don't know, summarize medical reports, then you're probably looking at accuracy and completeness.
• 6:27
  If it's a public facing chatbot, you might be interested in accuracy, but you're also interested in the extent to which the system
• 6:36
  The application is say, spouting inappropriate language as an example, it can be made to say inappropriate stuff.
• 6:43
  uh If you've built something which gives reason to malicious attackers to try and break into your system, then of course you're worried about those aspects.
• 6:54
  So at its core though, you have an application, you're bombarding it with a bunch of inputs, typically prompts of some kind, user prompts of some kind, and you're assessing
• 7:04
  how the system reacts to that.
• 7:06
  And that's all at its simplest.
• 7:08
  That's all that testing a generative AI system involves.
• 7:12
  And what about assurance?
• 7:13
  Does that mean something deeper than testing?
• 7:18
  It's a great question and m it's not something that I'm an expert in, but as we were designing our assurance pilot, we looked into it and one of the big four uh accounting
• 7:29
  firms, audit firms, me off to a definition.
• 7:33
  And my take from a rather long, several page long definition was that assurance, it can be generic, for a context like this, assurance is, hey, the...
• 7:46
  The system is performing as you have said it will perform along a set of predefined conditions.
• 7:54
  So you're not giving a generic level of assurance.
• 7:57
  You're giving a bounded assurance to say, it does this, this, this, and this.
• 8:00
  Now, you can argue the testing also has to be that.
• 8:03
  But I guess testing is a more informal thing.
• 8:05
  You could test it 20 different ways.
• 8:08
  But when you say, I've actually provided assurance on this system, you are actually doing it against a very specific set of risks.
• 8:15
  measured using a specific set of metrics and possibly with a specific set of thresholds.
• 8:22
  Okay, you mentioned the global AI assurance pilot that you've just uh completed.
• 8:27
  uh First, just tell us a little about what that project is.
• 8:32
  Sure.
• 8:33
  So the Global AI Assurance pilot wasn't actually for pilot projects.
• 8:39
  It was for live use cases of generative AI.
• 8:42
  The word pilot is used because the testing aspect of it was more pilot-like.
• 8:46
  And in each of these cases, these were in banks and asset managers and hospitals and insurers and an airport and a few other very real-life organizations, manufacturers.
• 8:57
  Each of these instances, the idea was to get that application technically tested.
• 9:03
  for the risks that matter most to those use cases.
• 9:07
  Except we didn't do the testing.
• 9:08
  We paired these 17 organizations with specialist testing firms from around the world.
• 9:14
  So you had basically a deployer of an AI system paired with a tester of an AI system.
• 9:20
  And the objective was to help define, help make that connection between, this is the use case.
• 9:27
  What are the set of risks we should be worrying about in this use case?
• 9:31
  How do those risks translate into measurable concrete tests?
• 9:35
  And then how do you go about conducting those tests?
• 9:39
  And what do you learn from the exercise?
• 9:40
  mean, ultimately, the objective is to create a set of norms and best practices which then feed into standards for technical testing.
• 9:48
  But this space is so immature that we thought it was appropriate to start with something a bit more experimental in that sense.
• 9:56
  m But yeah, real life testing.
• 9:59
  of generative AI applications, not the models.
• 10:01
  We were not testing an open AI model or an anthropic model.
• 10:05
  We were testing the final application, which may be using indeed those models.
• 10:10
  Right.
• 10:10
  So can you say a little bit more about what's missing from the testing of the systems, given that the model developers are engaged in a great deal of work on testing and
• 10:22
  alignment and so forth?
• 10:24
  Absolutely.
• 10:24
  So let's take a couple of examples from the pilot.
• 10:27
  Maybe before I get into examples, I think the way to think about it is the focus for model developers, and indeed most research labs, most policy work in this space, is on the
• 10:39
  model's capability and safety.
• 10:42
  And that doesn't necessarily work for two reasons in real life applications.
• 10:47
  One is context and the other is complexity.
• 10:50
  So let me take each of these one by one.
• 10:52
  Context, let's take
• 10:55
  that example of a hospital that is using a large language model to summarize uh two medical reports and extracting key information and then making a recommendation on how
• 11:08
  soon should this patient come back for a repeat test.
• 11:10
  uh Now, the model underneath that could have been tested even on medical data to a great deal of accuracy.
• 11:19
  But it wasn't tested.
• 11:21
  in the context of this particular hospital with these specific reports.
• 11:27
  And so even though it did very well on a generic benchmark, it may or may not do that well in this specific context.
• 11:34
  Second is what does very well mean?
• 11:36
  The context is also, is it one in 1,000 errors?
• 11:40
  Is it no errors at all?
• 11:42
  Is it five out of 100 errors?
• 11:44
  You don't know that.
• 11:45
  And particularly with the lack of repeatability of many of these systems,
• 11:48
  It is very important in most real life use cases to say, no, I've done my own testing.
• 11:54
  So that's context.
• 11:55
  The second part is complexity.
• 11:57
  As I mentioned in this example, the reason why the system might not do well could indeed be the model not doing well.
• 12:03
  But it could also be that the source reports that came in had missing information or were formatted in a way where the model couldn't read it.
• 12:11
  It could be because the output was generated in a way in which the human in the loop, a medical doctor,
• 12:16
  could not understand what was put in front of them, and therefore did not make the right decision.
• 12:21
  So that's the complexity part.
• 12:23
  So when you test an application, you must include those aspects as well.
• 12:27
  Just testing whether the underlying model was good in theory is no longer useful.
• 12:34
  How did you source and match the companies, both the ones being tested and the ones doing the testing?
• 12:41
  To be honest, we were very surprised.
• 12:43
  We initially thought we'll get maybe five pairs.
• 12:45
  uh But it turns out, while there's a huge amount of activity in the testing of models, both for safety and capability, there haven't been too many experiments at this scale, 17
• 13:00
  real-life applications, um when it comes to, indeed, the use of generative AI inside mainstream non-digital industries.
• 13:11
  And so when we reached out to the market with this, we got an overwhelming number of applications.
• 13:19
  think we got 25.
• 13:20
  Primarily the deployers were from this part of the world, Singapore or the neighboring area.
• 13:25
  But the testers came from all over the world, from the US, from Canada, from France, Switzerland, Germany, um other parts of Asia, et cetera.
• 13:35
  the main reason was that they all saw the value of this particular pilot.
• 13:40
  as a template for how we might work out the what and the how of technical testing of generative AI applications.
• 13:48
  What to test and how to test.
• 13:52
  What did you find?
• 13:54
  Well, we I think the first thing we found to be honest is this confirmation of the fact that There is a big difference between testing models and testing applications So it was a
• 14:05
  hypothesis and confirming that hypothesis was probably the biggest single lesson learned that yes You do need to test for applications differently In some ways your model might be
• 14:16
  very capable your model might be performing at the frontier etc.
• 14:20
  What you need is
• 14:22
  boring predictability in the context of specific applications.
• 14:25
  It may or may not be doing 20 other things very well, but does it do this one thing that I needed to do in my specific context well?
• 14:33
  So that's the main lesson learned.
• 14:35
  beyond that, I guess there were four specific kind of practical tips, if you will.
• 14:40
  The first was the need to spend time on figuring out what to test.
• 14:45
  I know it sounds very obvious, but the reality is even as humans, we are not very good always at saying.
• 14:51
  Take that doctor's example.
• 14:52
  Is this a good summary?
• 14:53
  Is this recommendation correct?
• 14:55
  Maybe the recommendation correct or not easier to guess.
• 14:57
  But is this a good summary?
• 14:58
  Well, you ask two different doctors.
• 15:01
  They'll probably say, this one says this is good.
• 15:02
  The other one says maybe it could be slightly better.
• 15:05
  So finding ways to specify that in an automated fashion, quite difficult.
• 15:10
  So specifying what to test takes effort.
• 15:13
  That was our first lesson.
• 15:14
  The second major lesson, no surprise for at least somebody who comes from a data background, is that you can't assume
• 15:20
  Or rather, you should assume you will never have good test data to hand.
• 15:24
  Never have enough test data to hand.
• 15:26
  You can never have good test data to hand at the level that you need.
• 15:30
  So you must prepare for it.
• 15:32
  How do you do that?
• 15:32
  You use aspects like simulation testing, adversarial red teaming, automated generation, et cetera.
• 15:38
  Because that's the only way, using a mix of human creativity and AI, that you can generate enough um test cases to make the application testing really worthwhile.
• 15:50
  The third thing we learned was the need to look under the hood.
• 15:53
  What we mean by this, again, is if you go back to that example of the hospital or indeed of any of the various agentic AI systems that are being built out, there are multiple
• 16:02
  steps inside the application.
• 16:04
  And testing just the output might not be enough.
• 16:07
  Testing, for example, did the data extraction happen well?
• 16:11
  Did the online tool call as part of the agentic flow, which might be a web lookup, did that work well?
• 16:18
  Did a SQL query that was embedded work well?
• 16:20
  Testing each of these interim steps provides you two advantages.
• 16:23
  First, it provides you traceability in case things go wrong, you know where it went wrong.
• 16:28
  Second, it provides you bit of redundancy in the testing because even if your final output testing was not accurate, at least you figured out along the way that something went
• 16:36
  wrong.
• 16:36
  And the last and very important uh lesson was that actually, while it is inevitable that one has to use LLMs as judges to test LLM enabled systems,
• 16:47
  it has to be done with a lot of caution and a lot of skill because they can be unreliable and you're kind of building unreliability on top of unreliability.
• 16:56
  uh And there is a lot of care to be taken in terms of fine tuning or prompting these LLM as judges.
• 17:04
  And also a lot of human validation of the test results, which leads me to my last point.
• 17:11
  I know I'm cheating, said four, but there is a fifth one, which is across every stage of the
• 17:17
  testing journey.
• 17:18
  Today, the technology is certainly not at a stage where you can do away with the human.
• 17:23
  In fact, the role of the human expert is paramount across the testing lifecycle.
• 17:28
  I'm not talking about the main application.
• 17:30
  I'm just talking about deciding what to test, deciding where to test, figuring out how to generate the right test data to test, and actually validating or calibrating the automated
• 17:41
  testing results.
• 17:41
  In each of these aspects, the human role is absolutely paramount today.
• 17:46
  It might be that there'll come a day when the human becomes redundant for testing of gene AI systems.
• 17:52
  Today is not that day.
• 17:55
  So it's not going to be a matter of just push a button, get a test result and green light.
• 17:59
  can, but you wouldn't really be able to rely upon that.
• 18:03
  Yeah.
• 18:04
  Not at this stage.
• 18:06
  just so that people understand, you say it's inevitable to use LLMs in the testing process, is that just because of the scale of how many tests are being done?
• 18:17
  I think it's flexibility as well as scale.
• 18:19
  So actually, it's worth thinking about what we mean by using LLM's as part.
• 18:23
  So one part as we discussed earlier and I think you alluded to is that test data itself often has to be generated.
• 18:31
  You'll almost never have enough historical data.
• 18:34
  And so how do you do that?
• 18:35
  Well, if you want to generate realistic test data, one way you can do that is by creating scenarios saying, hey, I have these nine personas of customers for this application.
• 18:45
  For each of these, I give you some seed data.
• 18:48
  Can you go in and kind of create some more?
• 18:50
  So that test data generation, whether it's for adversarial purposes, which is red teaming, or more kind of normal purposes, whether it's stress testing or...
• 19:00
  So synthetic test data generation is indeed one area.
• 19:03
  But the other aspect why you need LLMS is suppose you are assessing something as nuanced as adherence to company values or...
• 19:14
  alignment with um brand image, or the language is not tight enough, or these are all concise enough.
• 19:26
  These are all quite difficult to mathematically define.
• 19:30
  You can't create rule-based engines always.
• 19:32
  So the first instinct is indeed to use an LLM to say, hey, you are an LLM as a judge, and you prompt it with appropriate codes and give some examples.
• 19:41
  What we have seen is that
• 19:42
  As people become more confident about what they are testing, they're able to replace many of these LLMs with smaller models, which have the advantage of being more efficient and
• 19:53
  also of being more reliable.
• 19:55
  So what I suspect you'll see in many cases is LLMs as a judge are indeed used as the first choice to understand how to write the test properly.
• 20:03
  And then once you've got a good handle of it, you find something far more efficient and far more uh reliable in some ways to do just that bit of
• 20:12
  testing.
• 20:12
  eh That's, think, the path that you'll see in many cases.
• 20:17
  I want to go back to your first recommendation on testing what matters.
• 20:21
  And it actually connects in with what you just said, that uh there's so many different unique aspects in every one of these applications.
• 20:29
  And it's not just a matter of the hospital has one set of test requirements, the hospital might have 100 different use cases, which are distinctive.
• 20:38
  So given all of that diversity of requirements, how can we ever get to a point of having
• 20:45
  testing standards and benchmarks that are consistent and universal.
• 20:51
  That's a very good question.
• 20:53
  And one, to be honest, that I don't think we personally, uh I personally, and our foundation has fully learned it.
• 20:59
  But I think we've got some useful insights from this pilot.
• 21:03
  First of all, there's some meta standards or meta best practices, if you will, such as how do you go about the process of zeroing in on your risk that matter and therefore
• 21:15
  converting that into tests?
• 21:17
  So there's that aspect itself, which is the learning
• 21:20
  how you do x.
• 21:22
  That's what actually we did see quite a bit of value from across the, when you have 17 different use cases, one of the advantages you get to see.
• 21:29
  And you can see there are some patterns on how you go about zeroing in on the use cases.
• 21:34
  So that's one part.
• 21:35
  The other thing I would say is while the specific prioritization of risks might be different by use case, if you have a summarization use case, well, you can only test so
• 21:48
  many things about a summary.
• 21:49
  In most cases, you can test its completeness, you can test its conciseness, you can test its uh accuracy or the lack of hallucinations, you can test its robustness over time or
• 22:01
  robustness to small changes.
• 22:03
  And these things can be standardized.
• 22:05
  Standardized with the smallest for now, meaning there is a reasonable level of agreement on what are the possible ways of testing these things.
• 22:13
  And so that can form the basis of norms.
• 22:16
  These norms...
• 22:17
  can over time become standards on how to test, for example, a summarization logic in an L or a summarization application.
• 22:27
  Where I think we will always struggle and maybe not even attempt is what is the right level of threshold.
• 22:32
  So let's make it practical.
• 22:33
  Maybe for summarization you say, the score I'm assessing is based on a weighted average of the key claims made in the summary and assessing whether those claims are grounded in the
• 22:43
  source context data.
• 22:45
  OK, this can be reasonably standardized.
• 22:47
  But do I need a scoring threshold of 9 out of 10?
• 22:49
  Do I need a scoring threshold of 8 out of 10 key facts?
• 22:53
  That will depend on the use case.
• 22:55
  Or as we found in the hospital case, it might be that 9 out of the 10 is pointless if the 10th happened to be the number of polyps in the colonoscopy report.
• 23:03
  Well, the whole thing became 0 because the most important fact got lost.
• 23:07
  So I think if you just step back, the way in which you assess risks, the way in which you convert that into
• 23:15
  metrics for a given LLM archetype, like summarization uh or translation, et cetera.
• 23:23
  Those things can be standardized.
• 23:26
  The approaches to doing the assessment can be standardized.
• 23:30
  But the threshold of what is good enough, or even how many data points I need to, how many test data cases I need to make myself comfortable, that will depend use case by use case.
• 23:43
  Presumably that's also something that governments might also get involved in that regulators may want to say for this medical use case, this is an appropriate standard.
• 23:53
  So I understand this is a, you know, it's a private effort doing these pilots, but how do you think, you know, globally we get to having good conversations with regulators and
• 24:04
  policymakers about their relationship with testing?
• 24:08
  Yeah, I think it's a really good question and we're still figuring out the answer to that.
• 24:12
  I would say if I step one step before governments, I think there is a role for standards bodies to play.
• 24:18
  And indeed, here in Singapore as well as internationally, we are speaking to standards bodies to see how far can technical standards for AI, generative AI testing be?
• 24:27
  How far can they go?
• 24:28
  And then we can get to, well, do these pilots provide you some inputs for that?
• 24:33
  As to government or rather more formally regulators,
• 24:37
  kind of setting standards for specific class of applications.
• 24:41
  To some extent, they do that.
• 24:42
  If you look at software as a medical device, uh there are uh very tight standards, et cetera.
• 24:48
  I would see those to be much more use case specific.
• 24:51
  And will they do that?
• 24:52
  I assume so.
• 24:54
  When will they do that?
• 24:55
  I don't know.
• 24:55
  I think the technology and the usage has to be mature enough.
• 24:58
  We're certainly working at a Singapore level to assess when is that right time together with our
• 25:06
  various sector regulators.
• 25:07
  But I think it's too premature to say when.
• 25:09
  But back to the standards of what to test and how to test, not what is good enough, but how you can go about doing the test, that one is more with the standards bodies.
• 25:20
  And I think we can make more progress there relatively in shorter time frames.
• 25:25
  Yeah, what do you find in engaging with the standards bodies in terms of how sophisticated they are and how willing they are to go down that
• 25:34
  I they're very sophisticated, the ones I have spoken to both in Singapore and ISO, IEC, in Europe, CEN, CEN, ELEC.
• 25:41
  The issue is not of sophistication because remember, many of these have been doing standards for hundreds of, or for tens of years, if not more, right?
• 25:50
  I know at least one or two of these bodies have more than a century worth of history.
• 25:54
  The challenge is not are they sophisticated enough or are they willing to do it?
• 25:58
  The challenge is they know exactly how difficult it is to...
• 26:02
  get something to a level of maturity that can be a standard.
• 26:05
  So I think the onus is on people like us who are trying to get this more crowd, not crowd-sourced, private sector efforts together to suggest what they might be.
• 26:15
  I think the onus is, to some extent, to us to say, can we match that level of expectation that is there from standards bodies to put something as a technical standard?
• 26:25
  And I think I found them very collaborative, very knowledgeable, very keen to see if it can be made, it can happen.
• 26:32
  ah But equally, quite clear about the level of rigor that is needed in order for something to qualify as a technical strength.
• 26:42
  What's next for the AI assurance pilot?
• 26:44
  Is this something that you might do regularly or are there follow-ons to the pilot that you did?
• 26:51
  Yeah, so there's a few different things that we are considering.
• 26:53
  Some of them more concrete than others.
• 26:56
  So certainly, we are going to continue it in some form, probably not call it a pilot again, given that it's already been a pilot.
• 27:03
  But more on a rolling basis, we're toying with the word clinic, which is actually just this morning.
• 27:08
  Since this morning, I've had two entities in Singapore saying, hey, I've got a generative AI application.
• 27:13
  Can you help guide us how we can think about the risk prioritization?
• 27:19
  and then maybe suggest some metrics and then connect us with one of the technical testers and then we take it from there.
• 27:24
  So that's a lighter version of the pilot in some ways.
• 27:27
  The other thing you already referred to is providing inputs into standard setting efforts.
• 27:32
  So not start trying new efforts on our own, but there are existing efforts at setting AI testing standards.
• 27:38
  How do we feed into that?
• 27:40
  So that's a second area.
• 27:42
  um The idea you talked about sector regulators concretely say,
• 27:47
  for a certain application in a certain industry, here is the acceptable threshold.
• 27:52
  That could happen, but that's dependent, as you would have highlighted, on the sector-specific regulator.
• 27:58
  So we'll certainly explore that idea.
• 28:00
  One thing that we've heard a lot of demand for, but we are not yet sure how we progress, is accreditation of testing providers.
• 28:09
  Because the idea is saying, look, we understand maybe you're not in a stage where you can declare my application to be good fit for purpose, but could you at least
• 28:17
  point us to a set of companies that in your jurisdiction or as per your standards, is able to do a decent job of assessing or of testing these applications.
• 28:29
  I think that's something we have carefully listened to and we are considering how best to take that forward.
• 28:35
  Again, very early, we just finished it last week, so we haven't fully fleshed it out.
• 28:40
  Absolutely.
• 28:41
  And just let me ask you one or two final questions.
• 28:44
  The first one is just generally, can you talk a little bit more about Singapore's uh role and position in global AI governance?
• 28:53
  um Because it's, you know, it's most of the focus when I talk to people in the US is about legislation in the US, the European AI Act and things like that.
• 29:02
  this seems like a very different approach.
• 29:06
  Yeah, I think there are regulations which are not specific to AI, but there was uh something around the use of deepfakes, the most recent general election, et cetera.
• 29:16
  But that's a good example.
• 29:18
  I regulation in Singapore, at least in this space, will tend to be very, very narrowly focused on a very specific risk.
• 29:24
  Generally speaking, the idea is to provide good forward-looking guidelines, which you'll see across.
• 29:31
  I mean, I was the co-author of one of the first.
• 29:34
  In finance, for example, the feet principles of the Monetary Authority of Singapore back in 2018.
• 29:40
  So forward-looking guidelines, extensive engagement with industry, as you can see in this example.
• 29:46
  We're not coming up with here's the norms for testing.
• 29:48
  saying let's reach out to banks, hospitals, insurers, and work with technical specialists to figure out what it can be.
• 29:55
  um Lots of investment in kind of facilitating beyond these pilots, these kind of...
• 30:03
  industry level exchanges, et cetera, to take that forward.
• 30:06
  And then open source tooling, to the extent it makes sense to try and embed some of these not hard guidelines, but soft standards into something very concrete.
• 30:20
  I think that is what we've been following.
• 30:22
  are uh obviously a small country.
• 30:25
  Nevertheless, uh we do have a reasonably relevant footprint in the AI space.
• 30:31
  And we think for our most...
• 30:33
  This is not for the world.
• 30:33
  This is for our own economy.
• 30:36
  It is extremely important that we're able to use AI at scale with the right level of trust.
• 30:41
  And we think the way we can do that is through this mix of guidelines, not legislation, not regulation, plus um extensive industry engagement on very practical aspects like this
• 30:53
  pilot.
• 30:54
  And then finally, open source tooling where it makes sense to translate these guidelines, these soft standards, into something very usable and concrete.
• 31:04
  And just broadly beyond what you're doing in AI Verify, what do you see as the next stage in terms of AI governance, responsibility for AI getting two more robust standards and
• 31:20
  testing?
• 31:22
  When you have a hammer, everything looks like a nail.
• 31:25
  So I will just give you my perspective.
• 31:27
  uh My perspective is, think, the conversation on AI governance.
• 31:31
  And you know this much better than me, given your position.
• 31:34
  The conversation on AI governance from a what should the guidelines say, what are the right ethical principles, what is the right framework, I think that's done.
• 31:45
  And I think there will be a set of conversations about how much of that is written down as legislation or regulation.
• 31:53
  fine, that's in certain geographies.
• 31:55
  But other than that, the focus really needs to be, how do we convert this into something that is possible to operationalize at scale without depending on massive amounts of human
• 32:07
  makers and checkers?
• 32:09
  And to do that, that is where we think um technical testing standards, uh ideally things that can be automated, uh is super important.
• 32:21
  angle that I already alluded to, that's why my comment about hammer and nail is I do think there will be greater realization of the difference between model testing, which is often
• 32:32
  about safety, alignment, human machine alignment, or just generally more broadly a little bit about the frontier risks, which are very important, versus, OK, I just need to use
• 32:43
  this in this one application.
• 32:45
  How do I make sure it does what it says on the tin?
• 32:47
  And remember, it's an application, not just the model.
• 32:50
  Both are needed.
• 32:51
  I think a lot of the conversation has been on the former so far.
• 32:54
  We will see more of the latter if you look at what Arvind and Sayesh Kapoor, Arvind and Sayesh Kapoor of the snake oil, AI snake oil fame.
• 33:05
  They talk about AI as normal technology.
• 33:07
  I really like that.
• 33:08
  right.
• 33:09
  Then I think the vice chair or one of the board members of Alphabet wrote a guest column in The Economist, which was about how to really make AI reliable.
• 33:20
  you should focus on the points at which AI is touching the real world.
• 33:24
  And then focus on controlling and regulating if you need that, which means don't regulate general purpose models.
• 33:31
  At least his argument was, I'm honestly saying I'm agreeing.
• 33:34
  But when you're using a general purpose model to do something, software is a medical device, yes, then focus on that.
• 33:41
  So if you take that and then all of this, there is, think, uh a healthy
• 33:46
  movement now in addition to the folks who are looking at model safety of also saying let's think about where these models are hitting the real world and let's think about how to
• 33:57
  operationalize the reliability testing.
• 34:01
  So away from safety to reliability and from models to applications.
• 34:05
  uh That's my version of I have a hammer and so I'm looking for an edge.
• 34:12
  Wonderful.
• 34:12
  Shumig, thank you so much for your time.
• 34:15
  Thank you.