Latest / Agentic Conversations (formally mlops.community) / Security and Privacy // Day 2 Panel 1 // AI in Production Conference

Transcript

Security and Privacy // Day 2 Panel 1 // AI in Production Conference · Agentic Conversations (formally mlops.community)

• 0:05
  Ohh, look who it is. Mr Diego Oppenheimer.
• 0:13
  I conned you into being part of one of these conferences Yet
• 0:17
  another time. I'm always happy to do it.
• 0:21
  Can you hear me alright? I hear you loud and clear, man.
• 0:25
  It's great to have you. We've also got some other
• 0:28
  amazing people joining this next panel that we're about to do.
• 0:31
  We've got Mr adds Dawson. Great great to have you back.
• 0:35
  We've also got Catherine. How you doing?
• 0:39
  And of Gandalf fame Mr. David Abair.
• 0:45
  So. I get to jump off stage and let
• 0:50
  Diego run the show on this one. We're going to be talking all
• 0:54
  about security, privacy in the LM land.
• 0:57
  I'll let you take it over Diego. Sounds good.
• 1:02
  Thanks, Dimitris. Hi everyone.
• 1:04
  Super excited to have to talk with this panel.
• 1:05
  Have some amazing folks. I I do laugh about the data kind
• 1:11
  of privacy and security kind of talk about it in a good way.
• 1:14
  Because I remember talking with Dimitrius a long, long, long
• 1:17
  time ago about how, you know, there's all the security
• 1:22
  attached to models, yet we still allow our data scientists to
• 1:25
  kind of essentially import all of pie pie, you know, in every
• 1:29
  single time of their scripts that they use.
• 1:31
  And like one day somebody's gonna exploit that.
• 1:32
  And then I think over the last two years we've been sharing
• 1:35
  back and forth text around like, hey, here, it happened again, it
• 1:38
  happened again. So that was kind of a pretty
• 1:42
  funny too. I'm Diego Oppenheimer.
• 1:43
  I'm a co-founder at Guardrails AI, which is an organisation
• 1:47
  that focuses on helping get the behaviour you want out of
• 1:53
  workflows. I have an amazing panel today of
• 1:57
  absolute experts and I'm gonna let them introduce themselves.
• 1:59
  So I'm gonna start with ads and let you introduce a little bit
• 2:02
  about your background and yeah, to the audience.
• 2:06
  Hey, Yvonne, pleasure to be here.
• 2:08
  Thanks for having me. I'm a Senior Security Engineer
• 2:11
  at COHERE. I primarily work on our RED team
• 2:16
  operations, our application security as well as protecting
• 2:19
  our foundation models. My background, I started as a
• 2:22
  network engineer, kind of moved into network pen testing and
• 2:25
  then towards sort of REST APIs and our thoughts into like the
• 2:28
  crazy LM generally. I space over to our popcorn, IT
• 2:34
  to David. Hey guys, good.
• 2:37
  Good to see you all and it's great to see you again as well.
• 2:43
  So David, co-founder of Lacura, we work with some of the largest
• 2:49
  organisations out there and most exciting organisations out there
• 2:52
  and helping them adopt Gen AI securely.
• 2:54
  And as Demetrius mentioned, we also the company banned Gandalf,
• 2:58
  which I'm sure some of you have enjoyed playing over the last
• 3:02
  couple of months. Catherine.
• 3:05
  Catherine. Thanks.
• 3:08
  I'm Catherine Jermall. I'm a Principal Data Scientist
• 3:11
  at ThoughtWorks. I've been working and then help
• 3:14
  for more than 10 years. So I saw the rise of LM coming a
• 3:17
  long time, and I moved to privacy in machine learning
• 3:21
  privacy, particularly natural language processing, about seven
• 3:24
  years ago. And I'm the author of the book
• 3:27
  that got released last year at well Received called Practical
• 3:30
  Data Privacy, about building data privacy into machine
• 3:34
  learning. Amazing.
• 3:36
  Well I'll get right into it and I'll pick on you Catherine.
• 3:39
  First like help, help the audience like been defined for
• 3:43
  us security versus privacy, right, in this context, right.
• 3:46
  And in a I like you know what A, when we think about certain
• 3:48
  practises around security what should we think about when we
• 3:51
  think about certain practises and privacy what should we think
• 3:54
  about giving your book? This seems to be very relevant.
• 3:58
  Yeah, yeah. I mean, I've worked in the field
• 4:00
  of cryptography as well. So working on encrypted machine
• 4:03
  learning, which is where kind of like you can think of security
• 4:06
  and privacy meeting. But essentially we can think of
• 4:09
  privacy as both a social or political and a legal concept
• 4:13
  that is quite separate from security.
• 4:16
  GDPR is a privacy law, and it's probably the most relevant law
• 4:20
  for most of the things that we do around machine learning right
• 4:23
  now at scale. And it's by default isn't a
• 4:26
  cybersecurity law. So it's a privacy law because
• 4:29
  there are special legal rights to privacy are both within
• 4:34
  Europe and and internationally. Sometimes privacy and security
• 4:39
  support each other. That's the world I want to be
• 4:41
  in. Sometimes they actually don't
• 4:43
  support each other. So we can think of security
• 4:46
  surveillance systems like a doorbell camera as something
• 4:50
  that we might argue supports security, but in this case it
• 4:55
  would be a huge violation of privacy.
• 4:57
  And we have kind of some of the same things happening in LM I
• 5:00
  think, where depending on how you surveil prompts and how you
• 5:03
  want to say, you know, dangerous behaviour could be a violation
• 5:07
  of privacy. Of users to to get really
• 5:11
  tactical on like you know a couple of the privacy so you
• 5:14
  mentioned one with around the Proms like when thinking privacy
• 5:17
  in the world of ellams which is you know kind of what we're all
• 5:19
  looking at. What are the some of the you
• 5:21
  know kind of top two three things that like you know people
• 5:23
  need to be aware of like today you know from a from a
• 5:26
  practitioners perspective. Yeah, I mean, I would expect
• 5:31
  that there might be some changes in the LM landscape based on the
• 5:35
  New York Times case. In case you haven't heard of it,
• 5:37
  the New York Times is suing for copyright abuse.
• 5:40
  Of course, it could also be any type of ownership abuse as well
• 5:43
  as private data abuse for Open AI.
• 5:48
  If you've been following any of the work of Google Research,
• 5:51
  it's been long proven. Now over the course of four or
• 5:54
  five years that LM's memorise part of their training data,
• 5:58
  about 30% of repeated text is memorised.
• 6:01
  This is a feature, not a bug. It has to do with the way that
• 6:05
  the kernels in particular areas functions.
• 6:07
  So if you want to think of it as a large multi dimensional space
• 6:11
  we can say that the LM, especially the over
• 6:14
  parameterized models of any type end up memorising some of the
• 6:18
  training data in order to generalise for sparse areas of
• 6:23
  the data set. And so this by default means if
• 6:26
  you're using a large over parameterized model, LM or
• 6:30
  otherwise it probably memorised some of the training data.
• 6:34
  There's some ways we can combat this is not always that we can
• 6:39
  combat this, but you should be aware that there may or may not
• 6:42
  be changes to the landscape because of that.
• 6:45
  And then of course there was some very interesting research
• 6:50
  released today by Google Research saying that they had
• 6:54
  received very good with differential privacy training of
• 7:00
  on device text data. So I think thinking through how
• 7:04
  you actually train models from the beginning with privacy built
• 7:06
  in, which might mean differential privacy, it might
• 7:09
  mean Federated learning, might mean encrypted computation or
• 7:12
  encrypted machine learning, which are all of course covered
• 7:15
  in the book. That might be something that
• 7:18
  becomes quite relevant for this space.
• 7:22
  I'll open this up anyway. This is something that I've, you
• 7:24
  know, I've struggled not struggle, but I kind of like
• 7:27
  thinking through it philosophically, which is, you
• 7:29
  know, I really like that you bring up like, you know, kind of
• 7:32
  memory. You know, anybody who signed an
• 7:33
  NDA or actually have worked with kind of like contracts with
• 7:36
  mutual Ndas knows that essentially part of the
• 7:40
  exclusion is like something that you can learn like that you can
• 7:43
  learn and like learn into memory while you're doing it.
• 7:45
  So you can't take papers, You can't like take pictures.
• 7:48
  But if you learn into memory, there's this exclusion almost
• 7:51
  from legal frameworks around like, you know, kind of like,
• 7:54
  you know what what that looks like.
• 7:55
  Should we have a different paradigm for computers?
• 7:58
  Like should I be outside of that?
• 8:00
  Like, I mean like like should I kind of as humans, we expect
• 8:05
  humans to be able to learn and use that information in every
• 8:09
  day. It's actually really hard to
• 8:10
  unlearn something, especially if you're in a room or kind of
• 8:12
  exclude yourself from it. Should we have a different
• 8:15
  paradigm for AI? If so, why?
• 8:20
  And if not, again, I'm kind of curious and I'll let it to
• 8:22
  anybody. But like, I realise it's a
• 8:24
  little bit of a philosophical, get a lot more tactical after
• 8:26
  this, but I've always kind of had that in the back of my head
• 8:28
  since you brought up that memory thing is a really interesting
• 8:30
  problem. I think, David, you're smiling,
• 8:36
  so I think you have something to say about it.
• 8:40
  Just I'm just thinking through this and I wonder what ads would
• 8:42
  say. So I mean I mean the short
• 8:52
  answer should certainly be yes, we probably need a new new
• 8:55
  paradigm for for what we are going through now.
• 8:58
  Some of the things that we are spending a lot of time thinking
• 9:02
  about is not only sort of the current systems that we are
• 9:07
  seeing out there, humans interacting with, you know,
• 9:10
  typical LM sort of application. But like how do things look like
• 9:14
  as we take this into a world of agents and we've got more and
• 9:18
  more as of machine to machine interactions.
• 9:20
  So now we're talking not a human sort of learning things and
• 9:24
  memorising things anymore. And sort of the analogy to the
• 9:27
  NDA's that we are assigning today that that's going to be
• 9:30
  factored out very quickly, right, like it's more faster
• 9:33
  than atoms and there will be more machine to machine
• 9:36
  communication. These things will not only make
• 9:39
  decisions on our behalf, but they will also, you know, I
• 9:42
  guess learn on our behalf and memorise things.
• 9:45
  So I think there there's definitely a need for not only I
• 9:51
  guess the the concept of an NDA, but more more importantly so in
• 9:57
  our question around how we can trust these interactions and
• 10:00
  trust the learning that happens there and the memorization that
• 10:03
  happens there. These things are going to make
• 10:06
  all sorts of decisions for us, right.
• 10:07
  And people typically, it depends who you talk with, but people
• 10:10
  typically overestimate the timeline towards a highly agent
• 10:18
  driven world. This is gonna come sooner than
• 10:20
  we think. And I love the NDA example and
• 10:23
  I'll still think of a better answer, but I think there's just
• 10:26
  many, many pieces that we have to revisit and put in place for
• 10:31
  a highly agent driven, driven world.
• 10:34
  I just wanna time in you're still legally liable if you
• 10:37
  repeat something under NDA to somebody that hasn't signed.
• 10:40
  The NDA does, in case anybody doesn't know.
• 10:44
  You're not allowed to do that. Yeah, I I think it practicality
• 10:47
  if you look at enough. This is just something I was
• 10:49
  always been curious about Practicality, if you look at it,
• 10:51
  it's very hard to to prove those like again, like like in terms
• 10:56
  of again, you're right. I'm gonna start with that.
• 10:59
  You're absolutely right where you're saying that.
• 11:00
  But it's like they become like there's not a lot of cases of
• 11:02
  that actually having like you know, played out or being able
• 11:05
  to have shown that going into kind of more of the practical
• 11:08
  too. This brings up this kind of
• 11:11
  their pursuit of privacy versus utility, right?
• 11:13
  Like, what are the potential benefits, you know, of LM's?
• 11:17
  You know, do they outweigh the privacy risks?
• 11:19
  You know that we are talking about, you know, like what is
• 11:22
  the concept of private? You know, I think there's a
• 11:24
  bunch of like laws and especially in Europe that have
• 11:26
  been like quite successful in terms of helping with privacy.
• 11:30
  But, you know, some would argue that the illusion of privacy on
• 11:32
  the Internet is an illusion, you know, to a certain degree.
• 11:36
  So. So how should, how do we weigh
• 11:37
  the utility versus kind of like you know, benefits versus kind
• 11:41
  of risks of that of what we're doing right now with AI, Any
• 11:45
  frameworks that you've thought about or any interesting kind of
• 11:47
  like work that you can put the obvious to?
• 11:52
  I mean I think Apples approach is genius.
• 11:55
  So Apples just been going around and signing licence agreements
• 11:58
  with a bunch of content providers and several other AI
• 12:03
  startups are doing the same thing.
• 12:04
  So obviously if you have licence to use content and the licence
• 12:08
  is extended to whoever produces the content, so you find
• 12:12
  yourself in a very different legal standing.
• 12:14
  And I wouldn't be surprised if the LMS that Apple tries to
• 12:19
  introduce, which are also a little bit more task specific
• 12:22
  and content specific, end up outperforming many of the others
• 12:26
  when they release them. Sorry, I seem to have lost my my
• 12:31
  headset here. I can't hear anything.
• 12:33
  You guys still hear me? We can hear you perfectly, yeah.
• 12:38
  Hello, can you hear me so? Yeah, yeah.
• 12:43
  We lost the stop. Working.
• 12:46
  Ringleader. But that's the problem.
• 12:50
  Yeah. Ohh while he's deep debugging
• 12:56
  that. I love what you're talking
• 12:58
  about, Catherine, because the idea here is that you're saying
• 13:01
  ohh you know what? Apple is making a play not just
• 13:06
  for these bonds to be between Apple and selling New York Times
• 13:13
  on like your iOS subscription, but also now if they can deepen
• 13:18
  that relationship, it can be used in their models.
• 13:21
  And so in case Diego, you missed that.
• 13:23
  Are you back with us, Diego? I have back.
• 13:25
  Sorry about that. I don't know what happened.
• 13:26
  My like Bluetooth headset decided that I think you're on
• 13:29
  top of it. Thanks for jumping in.
• 13:30
  Like right there, like that. Was what I do.
• 13:32
  That's what I do. You caught me in the middle of
• 13:34
  eating a passion fruit, so I'm going to go back to eating the
• 13:36
  passion fruit. I'll let you guys get back to
• 13:38
  this. That's great.
• 13:41
  Yeah. Yeah.
• 13:42
  I think thanks for that too. I'm going to one more thing on
• 13:46
  kind of privacy and then I'm going to switch over to kind of
• 13:48
  like some security questions. Totally agree with you on the
• 13:53
  kind of like licencing content and taking your approach and you
• 13:55
  know really taking a a serious kind of look at like look this
• 13:58
  is your content generated, there's authors.
• 14:01
  We need to kind of like give that now what about, you know,
• 14:05
  we see a lot of like data being shared across, you know,
• 14:08
  especially here in the US you know, like you get a credit
• 14:10
  card, they're sharing your data with everybody.
• 14:12
  You sign up for this, you're sharing your data with
• 14:14
  everywhere. So in those cases where you know
• 14:18
  it's a little bit hard for you as a individual to kind of like
• 14:23
  you know control where all your data is being shared and how
• 14:25
  it's being shared. Does that change somewhat your
• 14:29
  answer Catherine, because I mean like these people are there is
• 14:31
  people licencing that data. I would argue it's technically
• 14:34
  legal right for them to to to to licence that they have because
• 14:37
  there's all the permissions but there really like de anonymizing
• 14:41
  you from the Internet and you are starting to show up in a
• 14:43
  bunch of things. So kind of like this, how does
• 14:45
  this tie like the right to be forgotten when these models get
• 14:47
  trained right and like how difficult that is to be achieved
• 14:51
  or David or anybody like wants to kind of answer that this is
• 14:54
  the question clear a little bit like kind of like.
• 14:56
  How I get what you're asking about.
• 14:58
  So there's like a few different paradigms at play here.
• 15:01
  So like 1 be right, right, Which is little bit separate than
• 15:05
  privacy. But we tend to use the same
• 15:07
  arguments in machine learning because with generative AI at
• 15:11
  least it's it's like the the arguments align quite well and
• 15:15
  then you know you have the copyright, you know owner and
• 15:19
  the copyright owner legally has a copyright to the material.
• 15:22
  And that's why if you read through the New York Times
• 15:24
  lawsuit that was filed, it's quite interesting because they
• 15:26
  were able to retrieve full article text from open AI models
• 15:31
  and so forth. So the argument here is like, if
• 15:35
  I can retrieve the full article text from a model that and that
• 15:38
  article is copyright underneath somebody's thing, you either
• 15:41
  need to pay for that content or you have to untrain.
• 15:45
  From what I've heard, Copilot has put some philtres on the
• 15:48
  same type of problems. So copilot has the same
• 15:51
  problems, right? Many of these over parameterized
• 15:55
  models will have the same problems because if you follow
• 15:57
  Vitally Feldman's research then you will see that it's
• 16:01
  mathematically impossible actually to not memorise data if
• 16:05
  your model is over parameterized.
• 16:07
  So that's fine. OK, what does that mean?
• 16:11
  Some of the research, the research I referred to today was
• 16:14
  about Google's research and Federated learning.
• 16:16
  Google's been doing Federated learning on device for keyword
• 16:19
  prediction also Apple does the same and they received today
• 16:24
  they announced, the research team announced they had received
• 16:27
  like within 10 percentage points accuracy using differential
• 16:31
  privacy during Federated training which has kind of been
• 16:35
  unheard of. I haven't read the paper yet, I
• 16:38
  haven't yet. So it was just today.
• 16:40
  But one thing that I advise folks, and why I wrote the book,
• 16:43
  is to start to familiarise yourself with privacy
• 16:48
  technologies and also with how they can be used in combination
• 16:51
  with stuff that you already do. And this is because when you use
• 16:56
  them correctly, and with the swag Google and Apple do it,
• 17:00
  you're often still compliant under strong privacy laws.
• 17:05
  Now we could have a whole other argument about do you want to go
• 17:07
  above and beyond compliance? But that's probably for a
• 17:10
  different time. Awesome.
• 17:13
  Thanks very much. I'm Richard.
• 17:17
  I'm going to switch it over to a little bit of the kind of on the
• 17:19
  security side. So from a governance and kind of
• 17:23
  you know side I've I've I've seen personally because I've
• 17:26
  worked in this space for quite a bit in terms of how governance
• 17:29
  and kind of risk management of ML models from what's called
• 17:32
  classical ML has switched over to kind of like the new
• 17:35
  generation. What I'm not because I'm not an
• 17:38
  expert on this like is how has the security landscape changed.
• 17:41
  So as I was going to ask you like kind of this type of stuff
• 17:44
  that you were working on from a security perspective on kind of
• 17:47
  like let's I guess we call it classical now is that, is that,
• 17:50
  is that the official term of to make us all feel old?
• 17:53
  But you know, has that like you know, has that shifted, What has
• 17:57
  changed? Is it mostly the same?
• 17:59
  Do you look at it in the same way, Like really curious to
• 18:01
  understand like kind of like that landscape and since you're
• 18:04
  right in the middle of. Yeah, it's, I'd say it's at
• 18:08
  least twofold to be honest. You have like what you would
• 18:12
  define when you say traditional machine learning.
• 18:15
  You've got like your traditional kind of mill Secops,
• 18:18
  vulnerabilities, threats, risks. But like we've been seeing this
• 18:24
  is kind of becoming agent driven.
• 18:27
  Large language models are being embedded within applications at
• 18:30
  the speed of light right now, like a common one for example is
• 18:34
  chat bots. So whilst we have ICAR
• 18:38
  traditional application security controls as well as you know
• 18:41
  like deep down in our ecosystem, our infrastructure and network
• 18:44
  controls, we also have what's kind of developed in terms of
• 18:50
  what we've been developing on the side of the OS new R top 10
• 18:54
  for OEM applications project is where those two kind of overlap.
• 18:58
  So where like classic machine learning vulnerabilities may
• 19:01
  introduce and maybe a lot more prevalent when embedded within
• 19:05
  an application, helping us define those like additional
• 19:09
  trust boundaries and areas for risk where that could be
• 19:12
  exploited where it can't be in the traditional environment.
• 19:15
  There are like overlaps between the two for sure, but there are
• 19:21
  there they do share like common unique, they do show share
• 19:25
  common common vulnerabilities within them.
• 19:28
  Example right, let's take like let's take like model data
• 19:32
  poisoning, something like front running or split view data
• 19:35
  poisoning is more like the ML Cyclops perspective but then
• 19:40
  that's not relevant to like LM application security per say,
• 19:43
  right? Because this is before the model
• 19:45
  was even produced and before it's put in the application.
• 19:47
  This is way in the kind of machine learning lifecycle, but
• 19:52
  then obviously they share their obvious common things, right?
• 19:54
  Like we always try and monitor and try and remediate dependency
• 19:58
  vulnerabilities on our in our SDLC.
• 20:01
  That's like really obviously even more prevalent now when
• 20:04
  we're considering our machine learning life cycle as well.
• 20:07
  So they they definitely do overlap, but is creating a new
• 20:13
  new landscape. New companies AI Red teaming is
• 20:17
  now kind of becoming its own thing.
• 20:19
  Penetration testing. So yeah.
• 20:23
  Yeah. Awesome.
• 20:27
  Yeah. So I I think it's, you know, I
• 20:29
  think in the audience we have a lot of people building stuff
• 20:32
  with AI. And it's interesting because if
• 20:34
  you're using a third, you know like for any of us who've built
• 20:39
  SAS products and launched him into the world that we've all
• 20:41
  gone through a security review, right.
• 20:42
  We've all had to like kind of like present ourselves as
• 20:45
  saying, hey look, this is the stuff that we've done and you
• 20:47
  have socked to or ICE or whatever someone in the
• 20:49
  standards and all these compliance and controls and it's
• 20:52
  really interesting. It's like you know what do we,
• 20:55
  you know what should we expect from our model provider vendors,
• 20:58
  right, so that we can actually go sell you know like if we want
• 21:02
  again assuming we're consuming models from other folks, right.
• 21:06
  Like if I am a, you know, if I'm a you know building in my own
• 21:10
  company and building my SAS like what should I be asking from us
• 21:14
  from my, you know from a security perspective from my
• 21:16
  vendor so that I can provide that to my clients, right, so
• 21:19
  that they get happy. Like I think that's a a core,
• 21:22
  core question that we all have as as we do that.
• 21:26
  And so I don't know as you have some thought, I know David you
• 21:28
  can have a lot of thoughts on this, but so I'm going to ask
• 21:32
  answer that if he has his specific questions then I'm
• 21:35
  going to move over to David. But no for sure I I think and
• 21:39
  and part of like what we've tried to do with Ross project is
• 21:41
  that there is definitely like a gap traditional machine learning
• 21:47
  doesn't you know count for things like traditional
• 21:50
  applications security whereas all these kind of like model
• 21:53
  based vulnerabilities and and not.
• 21:55
  So there's definitely like a kind of a a merge that's that's
• 21:58
  that's happening here. There's definitely a gap to
• 22:01
  educate one another. But yeah, I'll let David if he
• 22:05
  has anything to say, I'm I'm not.
• 22:08
  OK. So David, you're you're building
• 22:09
  out an area that's near and dear to my heart, which is, you know,
• 22:13
  I've said for a long time now that the number one barrier for
• 22:18
  AI adoption is that people don't understand how to measure and
• 22:22
  understand the risk associated with it.
• 22:24
  Like, you know, in general, like you know, building businesses is
• 22:27
  about taking risk in financial services is about taking risks.
• 22:29
  It's all about taking and understanding risk.
• 22:32
  But measuring risk here is really hard.
• 22:34
  You've built a company that's attacking the space to a certain
• 22:37
  degree. So maybe tell us 2 seconds about
• 22:40
  your company, but also like you know, how how should people
• 22:43
  think about, you know, and privacy and security go into
• 22:46
  that risk vector. Like how should we think about
• 22:47
  AI risk and what are the things that, you know, you know, we
• 22:50
  could do about it? Yeah.
• 22:53
  I think just to to um first of all everything that said and
• 22:58
  I'll add to that. I think we see a lot of
• 23:00
  companies now that are looking to the work that adds and and
• 23:07
  the group over at Owls have done to to ultimately understand what
• 23:11
  are what are first and foremost the the main risks that we
• 23:14
  should be looking at. It's from our perspective one of
• 23:17
  the most most helpful resources out there that companies use to
• 23:24
  to educate themselves. And then I would say sort of the
• 23:29
  traditional risk management side is in a transition phase right
• 23:32
  now where they're used to working with traditional
• 23:35
  software, non general I software.
• 23:38
  Things are well oiled sort of on that end.
• 23:41
  And all of a sudden they have 10 to hundreds of product teams
• 23:45
  that are running faster than anyone can think and they're
• 23:48
  putting the only security teams under pressure, but they're also
• 23:51
  putting traditional risk management under pressure,
• 23:53
  right. What does it mean now?
• 23:55
  What does it mean to pull in an open AI model into our stack and
• 24:00
  work with that? What does it mean to start sort
• 24:02
  of fine tuning our own models or even build our own models from
• 24:05
  scratch? I don't think we have templates
• 24:09
  for how to execute on that at this point.
• 24:11
  There are sort of universally applicable.
• 24:14
  We fortunately get to speak to some really exciting companies
• 24:18
  out there and we see some mature sort of concepts evolve and then
• 24:23
  you know it really comes down to ultimately what kind of
• 24:26
  application are you building. More and more we see companies
• 24:31
  in the news that I guess somewhat prematurely deployed
• 24:34
  there Ji application into the world and you know they're now
• 24:39
  sort of paying for that. Some companies are restricting
• 24:44
  use and sort of building Janai applications to internal
• 24:47
  applications to mitigate some of the risks that comes from the
• 24:49
  outside world. Others are building publicly
• 24:52
  facing applications. So they all come with different
• 24:56
  risk profiles. And again just coming back to
• 24:59
  what I said initially OWER specific is a great resource to
• 25:02
  learn more about this. I I literally point every person
• 25:05
  to that and there's a lot more work in the pipeline.
• 25:09
  I know on multiple fronts to to bring more sort of boilerplate
• 25:16
  execution templates into into the world as it comes to risk
• 25:20
  management and obviously of course as it comes to securing
• 25:23
  these generi applications prior to deployment but then also in
• 25:27
  in operation. I also wanted to sorry to
• 25:32
  interrupt. I also wanted to ask like David
• 25:35
  said nailed it. One of the real things that is
• 25:39
  is key primitive and is that education and is just basic
• 25:43
  elements that we go back to. Like threat modelling for
• 25:45
  example is like at a high level is a very, is a, is a is a is a
• 25:51
  way that you kind of use a framework or develop your own
• 25:54
  framework to evaluate and assess risk by looking at your
• 25:57
  environment. Like David said, like a lot of
• 25:59
  these vulnerabilities are not even relevant to your scenario.
• 26:03
  So about identifying through there's just through basic
• 26:07
  things like threat modelling where you can identify risk,
• 26:10
  identify trust boundaries and then look at you know your
• 26:13
  strengths and weaknesses of that particular trust boundary and
• 26:17
  effectively like go back to even old school networking
• 26:20
  infrastructure. Security controls can can really
• 26:22
  make key differences here. Yeah, I agree.
• 26:28
  Go ahead please. Ohh, sorry me just to follow on
• 26:29
  thought because I I fully agree and I also think it's
• 26:32
  interesting to to take a step back and think about what what
• 26:35
  is really sort of changing the landscape here and I think
• 26:38
  that's where it really pays off to go back to the basics that
• 26:42
  adds alluded to just now. But fundamentally what we're
• 26:45
  doing here is we're yeah we're bringing universal capabilities
• 26:49
  with universal interfaces into the world that literally like
• 26:52
  every 12 year old can play with. Gandalf is the best example of
• 26:56
  that and you now you know push these products out there like
• 27:01
  there's no tomorrow. So I think what what what you
• 27:04
  know our advice typically is to to sort of take a step back and
• 27:08
  think about what are the trust boundaries what does the threat
• 27:10
  model look like. Go back to the basics and you're
• 27:14
  you're in a good position to to to push applications out into
• 27:18
  the world. Yeah, like a lot of a lot of the
• 27:21
  a lot of the vulnerabilities that have been that have been
• 27:25
  you know, made public from what you can see from the public
• 27:28
  reports on like LM application security for example.
• 27:31
  A lot of them are traditional things like cross site scripting
• 27:34
  which have resolved through things like content security
• 27:36
  policies or even like API to API authorization.
• 27:40
  So you know, like implicit trust, human in the loop
• 27:44
  intervention as well. Sorry, that's the last time I
• 27:47
  interrupted. That was.
• 27:49
  Great. And one additional things go for
• 27:54
  it like has been helping some large companies deploy and
• 28:00
  production settings. And one of the things that we've
• 28:02
  found that I'll just give us a tip for some of these that went
• 28:05
  live in in big systems is like develop your evaluation criteria
• 28:11
  early and then update it quite often with your beta testers.
• 28:15
  Because a the model behaviour will change even on small
• 28:19
  version changes. And be if you don't have a
• 28:22
  valuation criteria for unexpected behaviour, users will
• 28:28
  find ways to create unexpected behaviour and will teach you to
• 28:32
  build your evaluation sets. So yeah, evaluation is extremely
• 28:37
  important and you should probably define your own rather
• 28:39
  than use a generic one. I love, you know that a lot of
• 28:44
  this stuff is like extremely practical.
• 28:46
  So one of the kind of like frustrations in this space,
• 28:48
  which I think is just a normal of like how it works is like
• 28:51
  it's very like philosophy driven for a while, right?
• 28:54
  Like we were talking about like you know I safety was mostly all
• 28:57
  defined about ethics and everybody wanted to talk about
• 28:59
  ethics. And then when she double click
• 29:00
  into that you're like well you know ethics change like you know
• 29:05
  based on population and civilization and who you are and
• 29:08
  what like you know these are not like concrete things that we can
• 29:10
  actually like like address. But now we're actually applying
• 29:13
  real security principles to you know kind of like same work
• 29:16
  ultimately. You know these these are
• 29:18
  probabilistic workflows, right. That's kind of like the change,
• 29:20
  right. But it is code and is machines
• 29:22
  and we are working with them and we have a lot of like the
• 29:24
  traditional stuff that's being applied.
• 29:27
  I think we're also hit a bunch of problems.
• 29:28
  I, I, I will always remember being on a call with a CIO of a
• 29:33
  very, very large insurance company and I'm telling that
• 29:36
  like you know we they would not be able to deploy our software
• 29:38
  or unless the scans came back with 90 percent 95%
• 29:44
  vulnerabilities addressed from the generic scan.
• 29:48
  And I was like well like I hate to like you know break this to
• 29:52
  you but like this means you need to shut off all your software
• 29:54
  like across the entire company because like no software comes
• 29:57
  back with like you know that that that kind of like you know
• 29:59
  level of you know here's here's your framework and here's how
• 30:02
  you address each one of those risks and what you can be
• 30:04
  exploited versus what you can't be exploited.
• 30:05
  But if you're just like purely scanning around this.
• 30:08
  And so I'm glad that a lot of these, you know, security
• 30:10
  engineers, like odds are kind of coming in and you know, kind of
• 30:13
  bringing that knowledge of like how this is done because I think
• 30:16
  like some of the same exact principles are gonna apply.
• 30:21
  I'm going to kind of like a the kind of like, you know talk a
• 30:26
  couple. We only have a couple minutes
• 30:27
  left so this is gonna be a bit of a crossfire.
• 30:29
  So when we think about kind of the black box nature of a lot of
• 30:35
  these, well, neural Nets in general, right.
• 30:37
  But like on top of that and like the fact that we're consuming
• 30:39
  models via an API, like is there things that we need to be like
• 30:44
  you know what what, what do we think that the Golden State
• 30:46
  where we need to get to right for Essentially ultimately I
• 30:49
  think the objective function is to be comfortable with risk.
• 30:52
  I'm not even going to say reduce risk.
• 30:53
  I think it would just be comfortable with the risk that
• 30:55
  we're taking. So like what do we think as an
• 30:58
  as an industry needs to happen for that to happen, right.
• 31:00
  Like, you know, for that to be true like this, we're going to
• 31:02
  assume that neural Nets will continue being black boxes to a
• 31:04
  certain degree. We are gonna win to continue
• 31:07
  using them because the the, the, the, the kind of the positive
• 31:12
  nature of them is like like, I mean the benefits, we're seeing
• 31:14
  it every day. What do we need to do to make
• 31:17
  ourselves comfortable with the risks that we're taking them?
• 31:20
  Yeah, I I I have very strong opinions here and I will not
• 31:23
  hijack the complete. Conversation you.
• 31:25
  Have one or two minutes here, but I I would even add to, I
• 31:28
  wouldn't even add to what you said, Diego.
• 31:30
  I think it's about controlling risk.
• 31:32
  It's not, it's it's not about like mitigating or like
• 31:35
  measuring. It's about controlling being in
• 31:37
  control. And I think one thing we've
• 31:39
  learned over the last decades, buildings on the most complex
• 31:44
  systems out there is that almost none of them are fully
• 31:47
  explainable. So this conversation around
• 31:49
  black box versus not black box and explain ability, that's
• 31:54
  good, but it's also distracting in many ways.
• 31:58
  And so I think ultimately what we need to get get to here is a
• 32:02
  world where the world of AI meets systems and safety
• 32:06
  engineering to to ultimately bring in principles into our
• 32:10
  development workflow and operational workflows that allow
• 32:13
  us to control the risk that is out there.
• 32:16
  If you, you know, think of a rocket that takes us to the
• 32:19
  moon, you know you've got a rocket that is like, you know,
• 32:24
  not explainable at all, but it's flown a million times and you
• 32:27
  know sort of what it's doing, that's a pretty good rocket to
• 32:30
  get on versus rocket that is fully explainable, has never
• 32:32
  flown at all. That's probably not the one you
• 32:35
  want to get onto. So, you know, I think there's a
• 32:38
  lot to be said here, but I think it's really about controlling
• 32:42
  risk. So I I love what you said.
• 32:44
  There you go. Catherine adds to any last
• 32:48
  thoughts because we got to start wrapping this up, sadly.
• 32:53
  No 100% control risk except the residual risk that is left over.
• 32:58
  Whether that's an accepted risk for whatever reason, 100%, I
• 33:04
  would definitely agree with that statement which you identified
• 33:08
  through things like threat modelling.
• 33:11
  Now I'd say privacy engineering is another thing to look out
• 33:14
  for. Still kind of in the infancy
• 33:16
  compared to security engineering, but it's an
• 33:19
  exciting field full of lots of fun maths.
• 33:22
  So if you got into machine learning for math, I welcome you
• 33:26
  to the field of privacy engineering.
• 33:29
  Well, I want to thank you all for being on this panel.
• 33:32
  I think it was awesome. I also want to thank you
• 33:34
  personally like adds Catherine, David.
• 33:35
  Like like I personally, I believe this deep in my heart
• 33:38
  like you're working on really the problem that needs to be
• 33:42
  worked on because you know I think this is the gateway to
• 33:45
  adopting what the greatest technology we will see in our
• 33:47
  lifetime is, right. I I very much deeply believe
• 33:50
  that like I think like we're we're working on this problem
• 33:52
  and it's about how do we adopt that technology and you're all
• 33:55
  working on it. So thanks for working on that.
• 33:57
  This was awesome. Thanks for having me.
• 33:59
  And Demetrius with floor is yours again.
• 34:02
  Incredible stuff, everybody. That was super cool.
• 34:06
  As you know Diego, I love just geeking out about this and I
• 34:10
  really appreciate that. We got all of you on a virtual
• 34:15
  room to geek out about it all at the same time.
• 34:19
  So it's really fun and thanks again, I'm going to keep it
• 34:22
  moving. So that means you all got to say
• 34:25
  goodbye and we will talk later. Hopefully.
• 34:29
  I'll get to see you.