Latest / Startup & Tech News from Germany, Austria, and Switzerland by Startuprad.io™ / The EU AI Act, Decoded: Andrew Gamino-Cheong (Trustible) on AI Compliance
Transcript
- WEBVTT 00:00:04.937 --> 00:00:12.737 Welcome to StartupRad.io, your podcast and YouTube blog covering the German 00:00:12.737 --> 00:00:17.657 startup scene with news, interviews and live events. 00:00:13.040 --> 00:00:19.120 Music. 00:00:20.097 --> 00:00:23.717 Hello and welcome, everybody. This is Joe from StartupRate.io, 00:00:23.917 --> 00:00:26.477 your startup podcast from Germany, Austria, and Switzerland, 00:00:26.657 --> 00:00:30.617 bringing you top news and the legal framework behind it. 00:00:30.797 --> 00:00:33.917 Today, we will talk about the AI Act. 00:00:34.197 --> 00:00:39.617 Yes, it's a law. Yes, it's on a European level. And yes, it will impact the 00:00:39.617 --> 00:00:42.617 artificial intelligence that is used in software. 00:00:43.737 --> 00:00:50.337 Surprisingly, I do have a, I would say you're more a coder, coder developer, 00:00:50.637 --> 00:00:54.637 entrepreneur from the US here as a guest. 00:00:55.237 --> 00:01:00.577 Because that is basically what it gets down to the development, 00:01:00.577 --> 00:01:03.677 the classification of AI. 00:01:03.757 --> 00:01:07.677 And we try to keep it as practical as possible. 00:01:08.157 --> 00:01:11.577 So, Andrew, I'd like to welcome you to StartupRate.io. 00:01:12.617 --> 00:01:16.197 Yeah, really excited to be here. Thanks for having me, Joe. My pleasure. 00:01:16.417 --> 00:01:17.937 As we said, you're not a lawyer. 00:01:18.177 --> 00:01:23.257 This is no legal advice here. This is just giving you a general idea. 00:01:23.517 --> 00:01:31.477 We may add here that the final effects of the AI come into force 2026. 00:01:31.777 --> 00:01:34.817 We're now the day before Halloween 2024. 00:01:35.177 --> 00:01:40.237 So it's a long way down the road, but you should take it already into account 00:01:40.237 --> 00:01:44.377 because it will have a massive implication. 00:01:45.777 --> 00:01:53.797 Can you tell us, Andrew, a little bit about you and how you got from cool stuff 00:01:53.797 --> 00:01:56.077 like coding into boring stuff like law? 00:01:57.077 --> 00:02:01.817 Yeah. So my background is as a machine learning engineer, but one who's actually 00:02:01.817 --> 00:02:03.277 always been interested in the law. 00:02:03.457 --> 00:02:05.997 So I studied both politics and computer science. 00:02:06.617 --> 00:02:10.857 And then I worked actually for a while at a company that was using AI to analyze 00:02:10.857 --> 00:02:12.857 upcoming legislation and regulations. 00:02:13.137 --> 00:02:17.237 So my interest point actually in my test case was always looking at proposed 00:02:17.237 --> 00:02:18.937 regulations for AI itself. 00:02:19.157 --> 00:02:22.497 And I started to actually see some of the growing corpus of things develop over 00:02:22.497 --> 00:02:26.637 time, including actually the AI Act, when it was first introduced in 2021, 00:02:26.637 --> 00:02:29.517 was one of the big things that really drove my thinking. 00:02:29.917 --> 00:02:33.577 A lot of core components of that have remained the same. So I actually started 00:02:33.577 --> 00:02:37.317 thinking through how would my day-to-day as a machine learning engineer change 00:02:37.317 --> 00:02:38.617 as a result of regulation. 00:02:39.037 --> 00:02:42.737 What would I be doing differently how would I be doing that differently I'm 00:02:42.737 --> 00:02:44.597 really starting to think through what were the pain points there, 00:02:45.167 --> 00:02:48.327 one key thing actually is that the ai act i sometimes say regulates 00:02:48.327 --> 00:02:51.147 more the people and the processes used to develop 00:02:51.147 --> 00:02:54.207 ai more so than the technology itself although there's 00:02:54.207 --> 00:02:57.407 a few parts of that that came in later in the negotiations 00:02:57.407 --> 00:03:01.667 and so a big challenge a lot of organizations have is how to just coordinate 00:03:01.667 --> 00:03:05.507 between their technical and their non-technical teams now on some of these processes 00:03:05.507 --> 00:03:09.587 that's what led me to co-found trustable for ai governance software company 00:03:09.587 --> 00:03:14.587 trying to help organizations easily and efficiently comply with a lot of regulations like the AI Act, 00:03:14.907 --> 00:03:17.507 like things coming out at the state level and a lot of the different standards 00:03:17.507 --> 00:03:18.547 and frameworks coming out. 00:03:19.067 --> 00:03:25.047 And that's a moment when a lot of our viewers and listeners have this aha moment. 00:03:25.287 --> 00:03:27.787 That's why you are here. Awesome. 00:03:28.587 --> 00:03:34.707 You've been talking about that the AI Act has been a long time in the making. 00:03:34.867 --> 00:03:36.347 It does have a little bit of history. 00:03:36.647 --> 00:03:43.527 It dates back to 2021. Would you say the success of OpenAI kind of accelerated 00:03:43.527 --> 00:03:49.487 what was going on there, that the European Union saw a need to regulate that? 00:03:49.767 --> 00:03:51.667 Admittedly, it's not only Germany 00:03:51.667 --> 00:03:56.967 that regulates here. There is also some regulation in the US as well. 00:03:57.860 --> 00:04:02.740 Yeah, so the AI Act really actually originated as primarily a product safety law. 00:04:02.940 --> 00:04:06.920 And it was initially, obviously, a lot more probably focused on typical classification 00:04:06.920 --> 00:04:09.080 type, predictive analytic type AI. 00:04:09.380 --> 00:04:13.660 I think there have been, even before that point, a couple of notable incidents, both in the U.S. 00:04:13.680 --> 00:04:17.720 And in the EU, of issues with computer systems causing a lot of harm. 00:04:17.920 --> 00:04:20.880 There's a lot of issues, I know, in the Netherlands with some benefits, 00:04:21.200 --> 00:04:26.340 AI systems having a lot of biases towards certain groups. There's a lot of cases in the U.S. 00:04:26.420 --> 00:04:30.680 Of similar instances, including one where the system was being used to recommend 00:04:30.680 --> 00:04:35.060 different prison sentences. It was found to have bias against certain demographics there as well. 00:04:35.260 --> 00:04:38.880 And so the whole idea of the origin of the law was to make sure that companies 00:04:38.880 --> 00:04:42.260 were implementing this kind of product safety mindset. 00:04:42.260 --> 00:04:46.180 And why I say that is that really means organizations need to be responsible 00:04:46.180 --> 00:04:50.700 for the safety of what they're selling out into the market and not simply say 00:04:50.700 --> 00:04:54.040 that it's your responsibility. you have to implement certain practices to make 00:04:54.040 --> 00:04:56.860 sure that your systems are safe for people and for society. 00:04:57.200 --> 00:05:01.440 Now, actually, if anything, the generative AI and chat GPT almost killed the 00:05:01.440 --> 00:05:04.680 AI act because there was suddenly a lot more concern about innovation. 00:05:04.980 --> 00:05:09.080 I think actually the AI act would have passed even without chat GPT and actually 00:05:09.080 --> 00:05:10.080 would have done so earlier. 00:05:10.220 --> 00:05:13.920 It was some of the negotiations from the EU parliament late in the discussions 00:05:13.920 --> 00:05:17.620 to manage some of the new things that are coming out around generative AI that 00:05:17.620 --> 00:05:19.860 actually delayed the AI act for a couple of months. 00:05:20.180 --> 00:05:26.300 But I bet that the moment you talked about responsibility. 00:05:26.960 --> 00:05:31.860 Liability in context of the US, a lot of entrepreneurs who are listening to 00:05:31.860 --> 00:05:35.420 this right now, the tone is starting to roll back. 00:05:36.280 --> 00:05:41.640 Before we get a little bit into that, we may also add, as we said, 00:05:41.760 --> 00:05:46.800 we're not a lawyer. this is called the AI Act, but this is not an act, 00:05:46.940 --> 00:05:49.800 not a law per se. All the... 00:05:51.352 --> 00:05:57.852 Entities or the states within the European Union have to implement this one 00:05:57.852 --> 00:06:00.792 way or another into national law. 00:06:00.972 --> 00:06:04.772 So there will be more than 20 different versions of that. 00:06:04.952 --> 00:06:08.912 Also countries in the European Economic Area, think about Norway, 00:06:09.072 --> 00:06:14.272 Switzerland, the United Kingdoms, they also have to have some kind of regulation like this. 00:06:14.572 --> 00:06:17.652 Plus, I do understand that 00:06:17.652 --> 00:06:20.792 President Biden also signed what is called an executive 00:06:20.792 --> 00:06:23.712 order we're not going into details why this is different from 00:06:23.712 --> 00:06:27.692 a law but it's also doing some my 00:06:27.692 --> 00:06:30.952 understanding is similar regulations of 00:06:30.952 --> 00:06:34.372 ai as does the ai act yeah the 00:06:34.372 --> 00:06:38.872 biden's executive order was very focused on actually public sector and government 00:06:38.872 --> 00:06:43.212 use of ai because that's what the scope of an executive order can do it did 00:06:43.212 --> 00:06:48.532 require and expect some testing using a kind of small law related to national 00:06:48.532 --> 00:06:51.532 security of some of the larger models, but actually it's almost. 00:06:51.912 --> 00:06:55.892 While it imposed some similar kinds of procurement and different risk management 00:06:55.892 --> 00:06:58.892 things as the AI act, they're actually covering different sectors because the 00:06:58.892 --> 00:07:00.592 AI act is purely a commercial law. 00:07:00.792 --> 00:07:04.432 It's again, product safety, and actually it's not prescriptive almost at all 00:07:04.432 --> 00:07:10.392 around the use of AI by different EU member state governments or in a completely 00:07:10.392 --> 00:07:11.652 exempted national security. 00:07:11.872 --> 00:07:15.392 And actually just last week, Biden signed another, technically it was a memo 00:07:15.392 --> 00:07:19.212 instead of an executive order detailing what practices the US should implement 00:07:19.212 --> 00:07:24.192 for AI risk management on top of national security and military use of AI. 00:07:24.412 --> 00:07:29.032 So I haven't seen anything on that front out of the EU or member states nearly as much. 00:07:30.432 --> 00:07:36.072 We have to admit that the NATO always gives us a little bit more time to react 00:07:36.072 --> 00:07:41.412 in terms of military stuff, which is also not a topic here of startup rate.io. 00:07:41.772 --> 00:07:46.352 But we will tell that we We attended Digital Gipfel, 00:07:46.512 --> 00:07:52.512 where we've been listening to all the people giving their perspective on everything 00:07:52.512 --> 00:07:56.032 digital in Germany, including speeches by Chancellor Scholz, 00:07:56.132 --> 00:07:59.152 Vice Chancellor Habeck and Minister Wissing. 00:08:01.075 --> 00:08:05.075 The AI act took basically center 00:08:05.075 --> 00:08:07.955 stage there because it was very important and 00:08:07.955 --> 00:08:11.455 everybody kind of starts to understand the 00:08:11.455 --> 00:08:17.675 implications of using AI from very simple stuff like writing automated emails 00:08:17.675 --> 00:08:28.155 to improving the efficiency of the national grid to casting for example metals 00:08:28.155 --> 00:08:30.855 and stuff like this. So that was very interesting. 00:08:31.215 --> 00:08:37.975 And I have been attending one special panel there where the two ladies in charge 00:08:37.975 --> 00:08:42.415 of the AI act, translating it into national law, basically talked. 00:08:42.575 --> 00:08:48.375 One of them was head of department where the AI act, the EU law is transformed 00:08:48.375 --> 00:08:51.895 into national law, which is not done by now. 00:08:51.995 --> 00:08:56.755 And the other lady was in charge of the Bundesnetzagentur of the department 00:08:56.755 --> 00:09:02.455 that will be set up, which is not there, that will be enforcing the AI Act. 00:09:02.575 --> 00:09:06.035 So they could only talk about it in very general terms about this. 00:09:06.195 --> 00:09:11.895 But what stuck with me is first they learned the lesson from GDPR. 00:09:12.075 --> 00:09:19.075 So they don't want to hammer completely the innovations as well as the fact 00:09:19.075 --> 00:09:23.955 that they want to encourage innovations that I want to kind of give you the 00:09:23.955 --> 00:09:28.815 swim lanes where to work in everybody who'd like to learn more. 00:09:28.815 --> 00:09:36.395 I have a small blog post about this panel down here on my blog as well. Andrew. 00:09:37.309 --> 00:09:41.649 Now we get into the AI Act. 00:09:42.009 --> 00:09:49.189 My understanding is there's first a classification by risk according to what 00:09:49.189 --> 00:09:50.829 the AI does, what it is useful, 00:09:51.069 --> 00:09:57.509 plus there are explicit cases where it's forbidden to use AI, right? 00:09:58.029 --> 00:10:02.549 Yep, that's correct. So the AI Act outlines several very specific use cases 00:10:02.549 --> 00:10:05.509 that are outright prohibited, and that part is actually already in effect. 00:10:05.509 --> 00:10:10.089 So already come, I believe, August, if anybody was using, let's say, 00:10:10.449 --> 00:10:15.089 a deliberately, emotionally manipulative AI system, that's now technically prohibited. 00:10:15.449 --> 00:10:18.589 Although I'll point out that there can always be a gray area between what is 00:10:18.589 --> 00:10:22.029 kind of a manipulation system and what might be an AI trying to upsell you on 00:10:22.029 --> 00:10:24.749 a product, for example, and love to dive into that a little bit later. 00:10:25.689 --> 00:10:29.109 Underneath those prohibited systems, there's then certain classes of use cases 00:10:29.109 --> 00:10:32.029 that are designated as high risk. And I'll say probably the majority of the 00:10:32.029 --> 00:10:36.609 act, as it'll impact a lot of your listeners, will be for anything that's designated as high risk. 00:10:36.829 --> 00:10:39.989 That's things that could impact people's financial or health, 00:10:40.109 --> 00:10:42.789 different uses in critical infrastructure. 00:10:43.709 --> 00:10:47.469 Sometimes people talk about that there's a limited risk option that's not quite accurate. 00:10:47.709 --> 00:10:51.589 Anytime you're using an AI that's interacting with people, you need to disclose that. 00:10:51.729 --> 00:10:54.969 And that can actually still be aligned with a high risk system. 00:10:54.969 --> 00:10:58.929 Could also be something that's then exempt entirely, which are other narrow 00:10:58.929 --> 00:11:02.029 procedural things, other things that don't fall into the category of high risk. 00:11:02.229 --> 00:11:06.109 Those are fundamentally unregulated, at least from the AI act, 00:11:06.269 --> 00:11:07.889 although all other laws still apply. 00:11:08.089 --> 00:11:12.429 So like you still need to obey GDPR in relation to even low risk AI systems, 00:11:12.649 --> 00:11:15.469 other kinds of things like basic liability for fraud. 00:11:15.589 --> 00:11:18.329 If you say an AI can do something that it can't, you're still gonna be covered 00:11:18.329 --> 00:11:21.929 under fraud regulations, even if it's not explicitly under the AI act. 00:11:22.789 --> 00:11:25.929 Yeah that was um i have spent uh 00:11:25.929 --> 00:11:29.069 quite some time uh during my last and at a 00:11:29.069 --> 00:11:32.469 big four company um working in forensics 00:11:32.469 --> 00:11:38.129 and my devious mind then instantly when i've when i was seeing uh what for example 00:11:38.129 --> 00:11:44.729 chat gpt did i was thinking hmm if i would be criminal i would be doing x and 00:11:44.729 --> 00:11:49.529 y so So that's already covered under the specific laws. 00:11:49.669 --> 00:11:57.249 I've been looking also through, well, basically, we could call it steps or parts of a permit. 00:11:57.489 --> 00:12:01.209 The forbidden parts include stuff like real-time biometrics, 00:12:01.409 --> 00:12:05.369 exploiting disabled or causing harm, things. 00:12:06.657 --> 00:12:12.317 System for inferring attributes like political opinion, sexual orientation, 00:12:12.317 --> 00:12:17.497 and so on, and also stuff like social scoring, which is done in other countries, 00:12:17.677 --> 00:12:21.517 or scrapping CCTV footage to compile facial recognition databases. 00:12:21.857 --> 00:12:28.857 I do believe that is something that is already done in different countries across the world. 00:12:28.977 --> 00:12:34.697 And I do believe there are some, let's call them agencies who are also doing 00:12:34.697 --> 00:12:38.117 that and It may not fall under this law. 00:12:38.957 --> 00:12:43.617 Yeah, that was one of the big, again, discussion and negotiation points is whether 00:12:43.617 --> 00:12:46.217 there should be certain law enforcement exemptions. 00:12:46.397 --> 00:12:49.797 And I believe now for a lot of the facial recognition stuff they added on, 00:12:50.137 --> 00:12:52.857 if there are specific things around sex trafficking, terrorism, 00:12:53.277 --> 00:12:54.957 there are exceptions allowed around that. 00:12:55.297 --> 00:12:58.497 And everybody is now listening to this. 00:12:58.657 --> 00:13:03.757 Oh, yeah, I don't need to do that stuff. I don't need that. But you've been 00:13:03.757 --> 00:13:07.997 already highlighting a pretty easy case here. 00:13:08.197 --> 00:13:12.837 For example, you have a web shop, you get a new chatbot on it, 00:13:12.897 --> 00:13:20.797 and then it turns out this chatbot is crossing the border and manipulating people to upsell them. 00:13:20.937 --> 00:13:26.237 So even if you are using those tools, you have responsibility, 00:13:26.237 --> 00:13:34.377 but the main responsibility falls onto the people, the companies coding this AI, right? 00:13:35.017 --> 00:13:39.537 So that's where there's a little bit of some gray area, especially in one of 00:13:39.537 --> 00:13:43.317 the challenges in trying to fit some of these new technologies and new structures 00:13:43.317 --> 00:13:46.977 into this product liability structure is that it can be unclear. 00:13:47.157 --> 00:13:50.517 If I am simply implementing a chat bot, but maybe it's connected with my own 00:13:50.517 --> 00:13:54.477 data through like a vector database, you know, what is my liability versus what 00:13:54.477 --> 00:13:58.117 is maybe OpenAI's API liability around some of that stuff. 00:13:58.517 --> 00:14:02.097 It's still not always going to be clear. You know, what counts as in legal terms, 00:14:02.117 --> 00:14:06.357 a substantial modification for a system needs to be clarified because I could 00:14:06.357 --> 00:14:08.537 add maybe just a few examples of something. 00:14:08.737 --> 00:14:12.177 You know, if I'm prompt engineering, does that count as modifying the system? 00:14:12.317 --> 00:14:15.397 It is changing some of the weights under the hood. I am changing some of the 00:14:15.397 --> 00:14:17.697 parameters maybe of the system that I can control. 00:14:18.017 --> 00:14:21.877 But how then is that impacting my liability versus OpenAI's? 00:14:22.137 --> 00:14:26.677 Also understanding like, what is the information that went into that and break 00:14:26.677 --> 00:14:29.957 down there, I think is something that needs to be clarified through some of 00:14:29.957 --> 00:14:34.797 those national laws, through some of the work from the EU AI office as well. 00:14:35.797 --> 00:14:38.637 Cause I think it's going to be a fairly complex space, especially if you're 00:14:38.637 --> 00:14:43.217 looking at using like an open source system, like you download Llama 3, you want to use that. 00:14:43.357 --> 00:14:46.457 The liability there is going to be much more on you as the deployer, 00:14:46.537 --> 00:14:49.277 especially because you don't have a contractual obligation there. 00:14:49.457 --> 00:14:52.737 So that's where there's going to be a lot more attention, I think in the years to come. 00:14:54.009 --> 00:15:00.389 So that also means not only the people who are really under the hood are responsible, 00:15:00.389 --> 00:15:07.529 are doing stuff within the framework of the AI Act, you can be just simply by using prompts, right? 00:15:08.249 --> 00:15:13.589 Yep. Okay. This is pretty important because I do believe a lot of people are 00:15:13.589 --> 00:15:17.989 now around going into stuff like Reddit, 00:15:18.229 --> 00:15:24.669 Googling different versions of prompts and simply using them in their tools. 00:15:24.889 --> 00:15:30.629 And that may be in the future a potential liability here. Yeah, 00:15:30.789 --> 00:15:33.389 well, it's definitely an area of potential exploits or attacks. 00:15:33.629 --> 00:15:37.529 There's already a lot of concern around the security systems and even the best 00:15:37.529 --> 00:15:39.649 systems that implement these strong guardrails. 00:15:39.929 --> 00:15:42.769 And, you know, Anthropic has talked a lot about the different things that they 00:15:42.769 --> 00:15:46.849 do there. It's still relatively easy to bypass these through simple prompt injections. 00:15:47.169 --> 00:15:50.909 Used to be as simple as saying, hey, ignore your safety guidelines and then it will. 00:15:51.109 --> 00:15:53.709 There's still other easy ways of telling it to like, you know, 00:15:53.749 --> 00:15:56.649 say the opposite of something can bypass a lot of filters, 00:15:56.669 --> 00:15:59.389 maybe asking it to like write something in German and then 00:15:59.389 --> 00:16:02.069 translate it into english and then run that you can avoid some of the 00:16:02.069 --> 00:16:05.529 filters that are looking for like keywords in english for example um so 00:16:05.529 --> 00:16:08.729 it's definitely an area of introducing new potential risks around 00:16:08.729 --> 00:16:12.209 some of those systems um even then again where 00:16:12.209 --> 00:16:15.969 the liability falls on that is going to be very dependent possibly on the use 00:16:15.969 --> 00:16:19.089 case and how you're deploying and setting it up whether you're fine tuning it 00:16:19.089 --> 00:16:23.929 how you're constructing that stuff it is admittedly we recognize it's impossible 00:16:23.929 --> 00:16:27.449 for an organization like open ai to know every possible way an organization 00:16:27.449 --> 00:16:29.829 might use something so from From that perspective, 00:16:30.029 --> 00:16:32.289 it could be hard to say that they are responsible for everything. 00:16:32.549 --> 00:16:35.829 On the flip side, there's oftentimes going to be a huge amount of asymmetry 00:16:35.829 --> 00:16:37.769 between the capabilities and resources. 00:16:38.009 --> 00:16:42.549 OpenAI is partnered with Microsoft. They have virtually unlimited funding for stuff. 00:16:42.709 --> 00:16:46.569 They have models that are taking months to train, spending tens of millions, 00:16:46.689 --> 00:16:48.029 if not hundreds of millions of dollars. 00:16:48.289 --> 00:16:53.109 To say that they own no liability and a three-person startup owns all the liability 00:16:53.109 --> 00:16:57.209 for all the stuff that could happen, that's a really difficult problem to have. 00:16:57.209 --> 00:17:01.929 But on the flip side, yeah, how can OpenAI know every possible thing, control everything? 00:17:02.229 --> 00:17:06.049 Even some things that look high risk might have valid use cases. 00:17:06.349 --> 00:17:09.689 You know, maybe you're talking about different national security or law enforcement 00:17:09.689 --> 00:17:12.429 things, but that's a valid use case for law enforcement itself, 00:17:12.429 --> 00:17:14.449 even if you want to ban that from the public versions. 00:17:14.869 --> 00:17:17.929 That's what I think is going to be very complex in the short term is how to 00:17:17.929 --> 00:17:20.049 walk that fine line and who's even 00:17:20.049 --> 00:17:23.169 going to be deciding what these systems should be allowed to do or not. 00:17:24.761 --> 00:17:29.081 Basically, you as an entrepreneur, because we mostly have entrepreneurs, 00:17:29.381 --> 00:17:32.301 investors and executives here as listeners, 00:17:33.021 --> 00:17:42.381 on you, the burden will fall to define some policy, internal compliance regulations 00:17:42.381 --> 00:17:45.201 of what you can do and what you can't do. 00:17:45.441 --> 00:17:52.961 I mean, from very, very simple things like don't use any available large language 00:17:52.961 --> 00:18:00.901 model that you can just Google to stuff like you're not allowed to do this or that in prompts. 00:18:01.041 --> 00:18:06.881 I think that there'll be one of the things that will be coming up, right? 00:18:07.581 --> 00:18:10.801 Yeah. And that's something that our platform actually helps with is helping 00:18:10.801 --> 00:18:14.781 draft like what is the policy you should have? What are some of the things you should consider? 00:18:15.061 --> 00:18:19.081 And really, what are the best practices that you as perhaps a small company 00:18:19.081 --> 00:18:22.181 should be implementing in order to comply with stuff? 00:18:22.181 --> 00:18:25.781 And you will always be responsible for ensuring that like your system is being 00:18:25.781 --> 00:18:29.441 tested maybe it is just testing your own prompts ensuring that it's working 00:18:29.441 --> 00:18:31.961 there's a lot of challenges even with these systems like 00:18:32.395 --> 00:18:35.155 maybe you use the same prompt and you get a different answer every time. 00:18:35.735 --> 00:18:39.155 That sometimes the feature, sometimes that's the bug, depending on what you're trying to do. 00:18:39.295 --> 00:18:41.575 It's still going to be your responsibility for assessing that. 00:18:42.315 --> 00:18:46.115 One of the hardest things is that there will be a lot of updates to these systems. 00:18:46.415 --> 00:18:50.495 We've seen that as different things come up, OpenAI and Anthropic have added 00:18:50.495 --> 00:18:54.235 different guardrails, and that has led actually to decreases in some of those systems. 00:18:54.435 --> 00:18:57.935 So it's always going to be your responsibility as the deployer, 00:18:57.995 --> 00:19:02.475 to use the AIX terms, of having your own quality management system where you're 00:19:02.475 --> 00:19:06.035 at least staying on top of the stuff that you can control, that you're being diligent about that. 00:19:06.435 --> 00:19:09.755 Users are still going to be reporting issues back to you. And there's this requirement 00:19:09.755 --> 00:19:13.135 on post-market monitoring. It's still going to be on you to be assessing those. 00:19:13.315 --> 00:19:16.335 You might need to report some of that back to OpenAI, but you still need to 00:19:16.335 --> 00:19:20.595 be tracking that information yourself. You can't offload all of those responsibilities necessarily. 00:19:20.995 --> 00:19:26.935 And that also means in the future, there will be a new job called a prompt tester. 00:19:27.335 --> 00:19:32.015 Yeah, I think there's been some interesting stuff coming out around quality 00:19:32.015 --> 00:19:34.755 assurance and the stability of the answers of these systems. 00:19:35.235 --> 00:19:39.935 But also a lot of this prompt testers are the whole ecosystem of AI red teaming 00:19:39.935 --> 00:19:45.555 and some of the within the AI act for general purpose, there's both standard 00:19:45.555 --> 00:19:46.355 general purpose models. 00:19:46.475 --> 00:19:49.575 There's also ones called general purpose models with systemic risk. 00:19:49.675 --> 00:19:53.935 Those are like the GPT-4, the really big models, the Claude-3 Opus. 00:19:54.895 --> 00:19:58.355 Those are ones that all require additional types of red teaming activities. 00:19:58.355 --> 00:20:02.575 Those are people who would come in and basically do various tests against those 00:20:02.575 --> 00:20:03.855 models, write up the reports. 00:20:04.095 --> 00:20:06.735 They're supposed to have a level of independence from the orgs doing that. 00:20:06.955 --> 00:20:09.855 That's going to become a burgeoning field. It's very similar to the idea of 00:20:09.855 --> 00:20:11.575 penetration testing and cybersecurity. 00:20:11.975 --> 00:20:15.735 We're also perceiving an audit around some sort of system, but that's something 00:20:15.735 --> 00:20:17.795 we're seeing be a fast developing field. 00:20:18.015 --> 00:20:20.775 And a lot of orgs want to apply that to high-risk systems as well, 00:20:20.935 --> 00:20:25.735 although that's not strictly required for at least most use cases that are high-risk under the AI Act. 00:20:26.381 --> 00:20:31.581 High-risk systems, sorry, they'll be defined in the respective laws. 00:20:31.781 --> 00:20:36.141 As you said, there will always be gray area and the forbidden systems. 00:20:36.281 --> 00:20:40.781 But I also do understand they're low to very low risk systems and some of them 00:20:40.781 --> 00:20:43.541 are actually not regulated at all. 00:20:44.301 --> 00:20:48.141 Yeah, that's correct. One thing I'll actually highlight is that the EU from 00:20:48.141 --> 00:20:50.201 the commission level is constructing a new office. 00:20:50.381 --> 00:20:53.921 That office will be responsible for actually updating the definitions of things 00:20:53.921 --> 00:20:57.661 that are high risk, there's a mechanism for them to add other things into there 00:20:57.661 --> 00:20:59.581 as well as prohibit different use cases. 00:20:59.841 --> 00:21:04.101 So even though each different member state will have its own organization overseeing 00:21:04.101 --> 00:21:09.141 the AI Act implementation, the definitions of high risk will still be more centrally controlled. 00:21:09.241 --> 00:21:13.181 And I do suspect that as more information comes out, they'll start to add more 00:21:13.181 --> 00:21:14.361 and more things to that list. 00:21:14.421 --> 00:21:17.701 And so we shouldn't expect the list currently in the legislation to stay fixed 00:21:17.701 --> 00:21:21.361 for much longer than probably the first two or three years that the act is in effect. 00:21:21.601 --> 00:21:27.421 We also talked about in terms of risk, there's also a definition of unacceptable. 00:21:28.701 --> 00:21:34.581 Risks in AI practices, manipulative or exploitative behavior. 00:21:36.101 --> 00:21:45.061 Do you think this will have an impact on especially like advertising targeting of people? 00:21:46.061 --> 00:21:52.041 Because at one point it makes sense to kind of target to use an AI to advertise, 00:21:52.281 --> 00:21:55.341 to collect data and understand your customers better. 00:21:55.481 --> 00:22:01.641 But I also do understand that there should be a legal limit how to do that and 00:22:01.641 --> 00:22:05.361 a special level on what you can and cannot do. 00:22:05.501 --> 00:22:09.741 Even though I understand that a lot of advertisers right now are thinking, 00:22:10.081 --> 00:22:13.321 hmm, I would like to do everything possible. 00:22:14.328 --> 00:22:17.188 Yeah. One of the, I was just actually reviewing the list of things. 00:22:17.448 --> 00:22:21.248 Um, one of the prohibited actions is actually inferring the emotions of somebody 00:22:21.248 --> 00:22:24.748 based on certain contexts and using that for certain purposes. 00:22:25.348 --> 00:22:28.268 Again, I think there will definitely be certain ways that you could leverage 00:22:28.268 --> 00:22:31.428 that information for advertising that would then be considered inappropriate. 00:22:32.108 --> 00:22:36.768 Admittedly, I think advertising the EU already looks very different than the U S because of GDPR. 00:22:36.908 --> 00:22:39.808 You still need that root information to begin with. That's where 00:22:39.808 --> 00:22:42.628 the data governance aspects of the AI act come into play even 00:22:42.628 --> 00:22:45.888 more so in regulating stuff i think most companies 00:22:45.888 --> 00:22:48.648 have um had already left the eu who 00:22:48.648 --> 00:22:51.688 are doing anything resembling the prohibited practices i think 00:22:51.688 --> 00:22:54.768 where there's some gray area for example though is that eu citizen data 00:22:54.768 --> 00:22:58.028 could still end up in some of those systems still being used there's 00:22:58.028 --> 00:23:01.028 a lawsuit from several companies in the eu right now against clearview 00:23:01.028 --> 00:23:04.288 ai which does do that scraping of facial recognition 00:23:04.288 --> 00:23:07.328 and they have the data of eu citizens inside that 00:23:07.328 --> 00:23:10.468 data set and that's where the extraterritorial aspects of 00:23:10.468 --> 00:23:14.708 the ai act are going to become more complex over time because people want more 00:23:14.708 --> 00:23:18.048 data for their systems it's the single most important thing and pulling that 00:23:18.048 --> 00:23:22.308 in is going to become a big international issue i suspect between different 00:23:22.308 --> 00:23:27.948 countries and i do understand it's almost impossible to keep you out of those databases, 00:23:28.788 --> 00:23:36.408 for example you just need to make a visit of new york and you'll be filmed by 00:23:36.408 --> 00:23:41.208 hundreds maybe even thousands of cameras if you spend one or two weeks in the 00:23:41.208 --> 00:23:42.948 city and they'll all go anywhere. 00:23:43.888 --> 00:23:49.088 If you just change flights on international airports, they may be not under 00:23:49.088 --> 00:23:51.328 the AI Act and so on and so forth. 00:23:51.548 --> 00:23:55.988 So basically, there will be circumstances where this will happen. 00:23:56.288 --> 00:24:01.488 We just talked about the territorial reach of that, and you talked about the 00:24:01.488 --> 00:24:05.128 part that applies to everything abroad. 00:24:05.128 --> 00:24:10.328 Could you elaborate a little bit on that before we get into what you expect 00:24:10.328 --> 00:24:15.048 from the EU and the US regulations going forward? 00:24:16.128 --> 00:24:21.348 So admittedly, GDPR actually probably had a larger extraterritorial scope because 00:24:21.348 --> 00:24:23.528 there was a fundamental rights law. 00:24:23.788 --> 00:24:26.708 It was enumerating certain privacy rights that you citizens had. 00:24:26.848 --> 00:24:29.728 And they're claiming that those rights exist for those citizens, 00:24:30.048 --> 00:24:33.248 regardless of how and where that they're operating on certain stuff, 00:24:33.508 --> 00:24:38.408 which is why you can even use citizen can send a takedown request to other companies. 00:24:38.408 --> 00:24:42.008 And it had a large impact in the US. the AI act might actually be a little bit 00:24:42.008 --> 00:24:46.068 more narrow in scope because it's really taking this product safety and commercial law. 00:24:46.368 --> 00:24:49.828 So admittedly, you're only really in scope of that if you are actively trying 00:24:49.828 --> 00:24:55.448 to sell your product inside the EU or if anybody in the EU is buying your product, um, 00:24:55.748 --> 00:24:59.228 through any of those mechanisms, the danger there is that there's entire companies 00:24:59.228 --> 00:25:02.808 that might say, Hey, we completely do not sell to any entity in the EU. 00:25:03.108 --> 00:25:07.868 We don't allow anybody operating or working in the EU to use our stuff and firewall it off completely. 00:25:07.868 --> 00:25:10.968 That'd be one of the bigger fears around like the innovation the 00:25:10.968 --> 00:25:13.928 market development part of that because it will be a 00:25:13.928 --> 00:25:18.148 little bit easier to actually avoid the ai act by simply not doing business 00:25:18.148 --> 00:25:23.388 with ai systems inside the eu than it was for probably gdpr my understanding 00:25:23.388 --> 00:25:32.188 right now as i said i'm neither lawyer nor coder is the regulations from right now enacted in the us. 00:25:33.268 --> 00:25:36.828 Speaking about memo, executive order, and so on and so forth. 00:25:36.988 --> 00:25:43.888 And the basic ideas of the AI Act are more or less in sync right now. Is that true? 00:25:44.008 --> 00:25:48.628 That's true in terms of several of the legislation proposed at the state level 00:25:48.628 --> 00:25:51.788 in the U.S., including one that passed earlier this year in Colorado, 00:25:52.188 --> 00:25:56.068 are fairly similar in that they identify certain high-risk use cases of AI. 00:25:56.248 --> 00:25:59.028 They say, we're going to impose certain requirements on them. 00:25:59.028 --> 00:26:03.008 Then the requirements that they're imposing are fairly similar to some stuff in the AI act. 00:26:03.408 --> 00:26:07.228 Usually the number one thing is that they say organizations need to be accountable 00:26:07.228 --> 00:26:09.028 for certain risks of their systems. 00:26:09.188 --> 00:26:11.408 So they need to implement a risk management system. 00:26:11.608 --> 00:26:15.388 They need to say, Hey, if this system hallucinates, here's what we think could happen. 00:26:15.688 --> 00:26:19.408 Here's how severe it is. And then document the mitigations that they have in 00:26:19.408 --> 00:26:23.068 place and then get somebody in the org, particularly their legal team to sign 00:26:23.068 --> 00:26:27.188 off and say, those mitigations are sufficient. and a big thing as well is actually 00:26:27.188 --> 00:26:30.248 having proper responses to incidents that are happening. 00:26:30.568 --> 00:26:35.088 The ETH is called post-market monitoring. In the US, it's usually just called incident response. 00:26:35.268 --> 00:26:39.148 So making sure that customers are able to report any problems they're having 00:26:39.148 --> 00:26:42.808 with the systems, that those reports are being looked at in a timely manner, 00:26:43.228 --> 00:26:45.468 that the system is being updated in response to that. 00:26:45.848 --> 00:26:49.128 Because we don't know everything that could happen from AI, it's more important 00:26:49.128 --> 00:26:52.928 to make sure that orgs are being diligent and responsible in responding to that 00:26:52.928 --> 00:26:56.528 kind of stuff rather than being able to know about that all up front right now. 00:26:58.430 --> 00:27:06.830 So we're now just at the beginning of the AI age, at the beginning of the regulation of AI. 00:27:07.010 --> 00:27:13.950 And I was wondering, you kind of live AI and AI regulation here. 00:27:14.830 --> 00:27:19.850 We also do have a lot of early stage entrepreneurs here. 00:27:20.010 --> 00:27:23.930 We have people responsible for larger organizations and so on and so forth. 00:27:23.930 --> 00:27:27.830 And many of them will, there's no question, 00:27:28.050 --> 00:27:37.390 will in the future implement some kind of true library API on anything else 00:27:37.390 --> 00:27:41.350 into the products related to AI. 00:27:42.290 --> 00:27:47.030 Is there kind of an approach you could recommend to them? 00:27:47.130 --> 00:27:52.570 For example, a very abstract do this first, then that, then that, 00:27:52.690 --> 00:27:57.050 then that. And of course, as always, talk to your lawyers. 00:27:57.270 --> 00:28:05.150 And of course, as always, I think more or less the AI testing sounds like auditing 00:28:05.150 --> 00:28:06.650 of some kind of software. 00:28:06.790 --> 00:28:12.790 You also have to do that. But before you get into this, some more very general 00:28:12.790 --> 00:28:19.950 thoughts about when and how to apply and how to work with the potential risk involved here. 00:28:20.969 --> 00:28:24.289 Yeah, so actually the first thing I recommend to a lot of teams is to think 00:28:24.289 --> 00:28:30.689 about documenting justifications of decisions made around your AI at every step along the way. 00:28:30.969 --> 00:28:34.709 In any sort of regulated context, having a paper trail of the decisions being 00:28:34.709 --> 00:28:36.509 made is one of the most important things. 00:28:36.569 --> 00:28:39.609 And if you do that from the beginning, you're saving yourself a lot of trouble 00:28:39.609 --> 00:28:43.209 at the end where some organizations try and like we say retrofit compliance 00:28:43.209 --> 00:28:45.289 to try and justify things after the fact. 00:28:45.289 --> 00:28:48.129 Oftentimes responsible ai isn't just having a 00:28:48.129 --> 00:28:50.949 product that's like unbiased and tested it's also about how you 00:28:50.949 --> 00:28:54.009 developed it did you engage with the right people along the ways 00:28:54.009 --> 00:28:56.789 um were you testing the system did you say like 00:28:56.789 --> 00:28:59.649 hey we're going to here's our criteria for selecting which 00:28:59.649 --> 00:29:03.549 model to use determine that criteria first do the evaluation then you can make 00:29:03.549 --> 00:29:07.349 a justification around that oftentimes implementing those things is what actually 00:29:07.349 --> 00:29:10.909 creates a lot of legal protections that you're doing what we say is like good 00:29:10.909 --> 00:29:14.909 faith efforts to comply good diligence around that and And the second part is 00:29:14.909 --> 00:29:17.749 realizing what some of the limitations of these systems are. 00:29:18.009 --> 00:29:21.449 There's a lot more work that needs to be done for transparency about these systems, 00:29:21.509 --> 00:29:23.289 and the AI Act does require that. 00:29:23.489 --> 00:29:26.789 You've done some studies and have evaluated which models are more transparent, 00:29:26.809 --> 00:29:30.329 but it's actually really important to understand maybe this model is not good 00:29:30.329 --> 00:29:34.469 at translation, or maybe I don't even know how good it is at translation because 00:29:34.469 --> 00:29:37.409 they didn't post a benchmark about its capabilities there. 00:29:37.789 --> 00:29:41.609 Maybe understanding this data set doesn't have information about maybe medical 00:29:41.609 --> 00:29:44.869 papers that went into it. And so because I'm doing something related to the 00:29:44.869 --> 00:29:47.269 medical space, that might be riskier to use. 00:29:47.629 --> 00:29:51.149 All of those different things are understanding the limitations of systems. 00:29:51.729 --> 00:29:54.969 It's going to become really important because oftentimes the first thing entrepreneurs 00:29:54.969 --> 00:29:56.949 are looking at is like, what model to use? 00:29:57.069 --> 00:30:00.149 How am I going to set this up? How am I going to sell this to my customer? 00:30:00.429 --> 00:30:03.969 And then making sure that you are accurately describing what the limitations 00:30:03.969 --> 00:30:07.569 of your own system are that's embedded with that. It's going to be the number one thing. 00:30:09.269 --> 00:30:15.949 And how to how to avoid risk for example i can now see in the future that you 00:30:15.949 --> 00:30:21.929 can simply just use the api of open ai and everything should be fine because 00:30:21.929 --> 00:30:27.829 it has strength there and weaknesses there and so basically you would need, 00:30:28.606 --> 00:30:37.306 to use a good enough or best tool for the respective duties that you assign 00:30:37.306 --> 00:30:43.146 to this software to this library to this api right precisely um we talk a lot 00:30:43.146 --> 00:30:48.046 about ai ethics one of the hard things about that and people always talk about oh we want unbiased ai. 00:30:48.606 --> 00:30:52.086 It's probably not possible for there to be unbiased ai because 00:30:52.086 --> 00:30:54.846 bias can be very subjective I lived for a while 00:30:54.846 --> 00:30:59.226 in France actually growing up and they actually constitutionally don't allow 00:30:59.226 --> 00:31:02.946 you to collect information about race or ethnicity they actually deny that those 00:31:02.946 --> 00:31:06.466 are constructs everyone should be equals and whatnot but then that creates a 00:31:06.466 --> 00:31:10.386 lot of challenges there where you couldn't measure necessarily like the selection 00:31:10.386 --> 00:31:13.686 rates of algorithms against those categories versus in the U.S. 00:31:14.006 --> 00:31:17.526 Now according to certain laws you have to do that definitely use cases in insurance 00:31:17.526 --> 00:31:20.806 there's some stuff in New York City around hiring you have to look at whether 00:31:20.806 --> 00:31:24.846 you're accidentally selecting in your AI certain races or ethnicities above 00:31:24.846 --> 00:31:26.386 others and look for bias there. 00:31:26.646 --> 00:31:29.566 So that's what's going to make it really hard for a single model possibly to 00:31:29.566 --> 00:31:31.426 be optimized against all of those. 00:31:31.566 --> 00:31:34.466 And either there's going to be different models that have different ethical 00:31:34.466 --> 00:31:35.866 principles built into them. 00:31:35.966 --> 00:31:39.766 You need to decide which ones are appropriate for your use, or we'll end up 00:31:39.766 --> 00:31:43.646 with a whole ecosystem of almost different ideologies, different philosophies, 00:31:43.746 --> 00:31:44.806 different ethical principles. 00:31:45.166 --> 00:31:48.346 We'll have a very competitive marketplace around that. I 00:31:48.346 --> 00:31:51.326 personally do believe you'll never 00:31:51.326 --> 00:31:55.186 have any unbiased ai 00:31:55.186 --> 00:31:58.746 tools because every human being has any 00:31:58.746 --> 00:32:01.706 type of bias and may maybe just be 00:32:01.706 --> 00:32:05.466 that they are fanatic for chocolate every human 00:32:05.466 --> 00:32:08.166 being has this and as long as we are 00:32:08.166 --> 00:32:12.006 coding those ais there'll be never an unbiased 00:32:12.006 --> 00:32:15.006 ai so always keep that in 00:32:15.006 --> 00:32:20.806 mind andrew there would be like a lot of more questions i would have for you 00:32:20.806 --> 00:32:28.546 but um we are running uh short on time at first like to thank you very much 00:32:28.546 --> 00:32:32.186 for being here secondly we will um. 00:32:33.450 --> 00:32:38.630 Link your company website and your LinkedIn profile down here in the show notes, 00:32:39.170 --> 00:32:48.810 can you give us like a very short pitch of what would be the use case for European 00:32:48.810 --> 00:32:51.310 companies using your tool? Yeah. 00:32:51.650 --> 00:32:55.150 So one of the hardest things actually for a lot of large organizations in particular 00:32:55.150 --> 00:32:59.190 with AI regulations is that they don't even know where and how AI might be used 00:32:59.190 --> 00:33:02.810 inside their organization, which means you don't know your exposure to the different laws. 00:33:02.990 --> 00:33:06.570 So the first thing our tool helps you do is build up an inventory of your AI. 00:33:06.810 --> 00:33:11.730 That's your AI use cases, your AI models, your vendors, some of the data sets being used. 00:33:11.890 --> 00:33:16.050 We can actually help evaluate what documentation you need for each of those, 00:33:16.370 --> 00:33:18.850 help you conduct some of the risk assessments to understand. 00:33:19.310 --> 00:33:20.930 You know, what are the risks of my AI? 00:33:21.210 --> 00:33:26.690 We combine a lot of insights on top of that, the whole idea being that AI is a fast-changing space. 00:33:26.690 --> 00:33:29.750 There might be new risks that emerge whether it's 00:33:29.750 --> 00:33:33.250 new cyber attacks new exploits or also new recommended mitigations 00:33:33.250 --> 00:33:36.950 against some of those risks We deliver that straight into our platform to help 00:33:36.950 --> 00:33:40.450 organizations actually be aware of hey I'm using a chat bot and there's an incident 00:33:40.450 --> 00:33:44.570 over here where chat bot unfortunately convinced a teen to commit suicide Maybe 00:33:44.570 --> 00:33:48.070 you should run this test to look for this implement this mitigation in order 00:33:48.070 --> 00:33:51.470 to reduce this kind of risk Then we track all the different regulations around 00:33:51.470 --> 00:33:53.230 the world to help identify hey, 00:33:53.330 --> 00:33:55.930 maybe there's a new requirement that's coming out of Canada that includes you 00:33:55.930 --> 00:33:59.650 to look at the environmental impact, and you haven't done that for some of these AI systems. 00:33:59.750 --> 00:34:02.090 So we can highlight that, keep you aware of that. 00:34:02.270 --> 00:34:06.490 The whole goal is to make staying on top of maintaining your AI systems as easy 00:34:06.490 --> 00:34:10.230 as possible, especially as either you have a lot of AI deployed in your org 00:34:10.230 --> 00:34:12.190 or you're having to comply with a lot of different laws. 00:34:12.390 --> 00:34:15.730 Our tool adds a lot of value. We're doing all of the heavy work on, 00:34:15.730 --> 00:34:18.630 I'll say, the mental workload of figuring out what you need to document, 00:34:18.630 --> 00:34:22.990 how to do So we can integrate into your system to help keep things in sync as well. 00:34:23.470 --> 00:34:26.670 Really just to help anyone comply with all of these emerging regulations. 00:34:27.370 --> 00:34:31.230 Great. Andrew, thank you very much. It was a pleasure having you as a guest 00:34:31.230 --> 00:34:35.410 and everybody who'd like to learn more, go down here in the show notes and there 00:34:35.410 --> 00:34:41.910 will be the links to his personal LinkedIn profile as well as the company website. Thank you very much. 00:34:42.330 --> 00:34:44.510 Thank you so much. Have a good day. Bye-bye. 00:34:49.290 --> 00:34:58.590 That's all, folks. Find more news, streams, events, and interviews at www.startuprad.io. 00:34:58.910 --> 00:35:01.070 Remember, sharing is caring. 00:35:01.840 --> 00:35:14.602 Music.