Latest / Agentic Conversations (formally mlops.community) / Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines // Chris Van Pelt // #192

Transcript

Enterprises Using MLOps, the Changing LLM Landscape, MLOps Pipelines // Chris Van Pelt // #192 · Agentic Conversations (formally mlops.community)

• 0:00
  So my name is Chris Van Pelt. I am cofounder and Chief
• 0:04
  Information Security Officer at Weights and Biases.
• 0:08
  I like my coffee as a good old French press.
• 0:13
  Maybe with ohh, Look, I've got my little emoji too.
• 0:19
  French press, a little oat milk lately has been more of a jam.
• 0:22
  Sometimes half and half. But yeah, just a a good French
• 0:27
  press. But it's like good beans, you
• 0:29
  know, Like the fancy hipster coffee shop, for sure.
• 0:33
  This is another ML OPS community podcast.
• 0:35
  I am your host Dimitrios, and what an episode we have today
• 0:39
  with Chris, who is the CISO at weights and biases.
• 0:43
  Weights and biases. Of course, you've probably heard
• 0:45
  of them. They are huge.
• 0:47
  They are killing it. They've been in the game for so
• 0:50
  long, helping researchers, engineers, and now even more so.
• 0:57
  The developers of the world use and.
• 1:01
  Take advantage of machine learning.
• 1:04
  It started with the predictive mill, went into the deep
• 1:06
  learning, now we're all in that LMS world and how they are
• 1:11
  tailoring their product to. Capture all this energy that's
• 1:18
  out there in the LLM world, Chris.
• 1:21
  Talk to me about his. Actually his first iteration
• 1:26
  which was a company and many of you who know the.
• 1:31
  Story of machine learning, probably.
• 1:34
  Know that Chris has been in the game for a long time, I mean
• 1:37
  since 2007. He talks about his first company
• 1:41
  that he built with Lucas, the cofounder of Weights and Biases,
• 1:45
  and that is Crowd Flower. And he goes through what they
• 1:48
  were doing, the pain that they were trying to alleviate when
• 1:51
  they built Crowd Flower, and then how him and Lucas
• 1:55
  transitioned out of that. They spent some time just
• 1:58
  getting back into the machine learning world after that.
• 2:01
  Startup and they realised. You know what?
• 2:05
  I think there's a possibility for another company and if they
• 2:09
  did it again, we might as well just call these folks serial
• 2:14
  entrepreneurs. So Chris then goes into talking
• 2:18
  to me about building products and building 1DB and how the
• 2:22
  product has evolved and what some of these big features that
• 2:27
  have come in over the years, they've come in over the years
• 2:32
  into the product and how they decide which features.
• 2:36
  Are relevant and then how they AB test them and get them.
• 2:39
  I I love this phrase that he used which is like.
• 2:43
  Demo driven development. So he they go out to customers
• 2:47
  and they say you know here's a demo of this minimum viable
• 2:52
  product that they have and they tell the story around it and
• 2:55
  they see what the interest is with customers and what they
• 2:57
  would want and how they would change it.
• 2:59
  And then he was very honest that once you get something out there
• 3:03
  that customers are using, that is really day zero and that is
• 3:07
  when you need to start the true process of iterating it and
• 3:11
  making it something that customers.
• 3:13
  Really love. So I appreciate Chris being very
• 3:17
  open with me and transparent on how they're building product
• 3:20
  also on his security lifestyle these days or I mean he's the
• 3:26
  SISO, right. So he gave us some hints on what
• 3:29
  to do to make sure that you are secure when it comes to building
• 3:35
  products and using products. And we then later talked about
• 3:42
  personas and how you build for different personas in this ever
• 3:46
  rapidly changing environment. Let's get into the full
• 3:50
  conversation with Chris and a huge shout out two Weights and
• 3:53
  biases for sponsoring this episode.
• 3:56
  If you would like to find out more about any of the stuff that
• 3:59
  we talked about in the episode, especially the evaluation and
• 4:02
  monitoring part of Weights and Biases, you can click the link
• 4:06
  in the description or just go to 1B dot AI and you can see it all
• 4:12
  for yourself. Now let's get into this episode
• 4:14
  with Chris, and if you enjoy it, please share it with one thread.
• 4:25
  wanted to talk all about. The inception, basically I
• 4:29
  wanna, I wanna talk about you and your journey and I also want
• 4:33
  to talk about weights and biases, because I think we
• 4:37
  cannot. They're very intertwined, I
• 4:39
  guess we could say. And I I think there's a lot of
• 4:44
  places that we may go off track and probably talk about
• 4:48
  information security or talk about the new LLM landscape and
• 4:53
  all that stuff. But let's just start with this
• 4:56
  first question. Which is what is Crowdflower?
• 5:02
  Yeah, Crowdflower Blast from the past.
• 5:06
  That was my first startup. So Lucas, wait, somebody is the
• 5:12
  CEO and my cofounder. He and I started a company in
• 5:15
  2007, so just before that recession, it was like the
• 5:20
  perfect time to start. To start a company?
• 5:23
  Yeah, yeah. Lucas and I met at a startup
• 5:27
  called Powerset in San Francisco.
• 5:32
  And Powerset was a technology company making a search engine,
• 5:38
  but they were using natural language processing to to make
• 5:43
  that search engine better. And Lucas was actually doing
• 5:49
  machine learning at the company. He was responsible for the
• 5:51
  ranking algorithm. So how we chose to sort the
• 5:54
  results coming back for any given query and we needed a lot
• 6:00
  of labelled data to train that algorithm.
• 6:02
  So at Powerset we started using Amazon Mechanical Turk which a
• 6:07
  lot of people don't know this but it was actually the first
• 6:09
  Amazon Web service. So like even before, like S3, I
• 6:14
  believe we might want to Fact Check this, but it was a very,
• 6:17
  very early Amazon Web service and we used to do Powerset and
• 6:23
  we were blown away by the ability to get labelled data
• 6:26
  very quickly by just sending it out to people on the Internet
• 6:29
  that were willing to do these micro tasks or basically a short
• 6:32
  labelling task At Powerset it was, here's a query, here's a
• 6:35
  result. Rate this result on a scale of
• 6:37
  one to five and we started building some tools internally
• 6:42
  power set to help us with quality control and how to
• 6:46
  ensure that we were getting good labelled data for the algorithm.
• 6:49
  And we kind of stepped back and thought, hey, maybe there's a
• 6:51
  company here like we could help other companies do this.
• 6:54
  We think machine learning is cool and that's that's how it
• 6:58
  all kind of began, yeah. Dude, so ahead of the curve it
• 7:04
  was. We were two ahead of the curve.
• 7:05
  That was. We were way too early on the ML
• 7:08
  train for sure. But we We learned a lot, Yeah.
• 7:11
  Yeah, 2007 doing stuff that I think people in 2023 are still
• 7:17
  trying to figure out how to do consistently and well.
• 7:20
  And you were like, OK, there's a company here.
• 7:23
  So how does Crowd Flower differ from what then?
• 7:27
  And what was the trajectory of it?
• 7:30
  I mean, we were young, we were like 26 years old.
• 7:33
  We started the company and really we we're both
• 7:36
  programmers, so we love like using our computers to make cool
• 7:41
  stuff. But we had no idea how to run a
• 7:45
  business. So we we had to learn a lot of
• 7:50
  lessons the hard way but we learned you know a tonne of of
• 7:54
  valuable lessons for sure and and like what it takes to build
• 7:58
  a a sales organisation and and to to hire good talent.
• 8:03
  And you know one of the first big mistakes we made was we had
• 8:09
  built this kind of tooling around micro tasks and then we
• 8:12
  thought alright, let's just go to to companies and offer to
• 8:18
  solve their labelling problem almost as like a consultant.
• 8:23
  And suddenly we find we found ourselves like in a consulting
• 8:26
  business, which is not a good business.
• 8:29
  We we quickly found out that especially in Silicon Valley
• 8:33
  it's all about that subscription revenue.
• 8:35
  You need to somehow find a way to get subscriptions which is
• 8:39
  kind of it's hard with a a problem like data labelling
• 8:42
  because people want to to buy just a set of labels and not
• 8:48
  necessarily kind of an ongoing service.
• 8:51
  So I mean that's just one of like many many lessons we we
• 8:55
  learned and and the Crowd Flower journey, it was like 10 years a
• 8:58
  lots of like ups and downs and we we thought we had really you
• 9:02
  know things really going and then we had to kind of step back
• 9:05
  and rethink what the product looked like how it worked.
• 9:10
  Yeah, but learned a lot for sure.
• 9:13
  OK, So what was the transition from Crowd flower to weights and
• 9:17
  biases? Yeah.
• 9:19
  So eventually, after about 10 years, Crowd Flower really
• 9:23
  started to hit its stride. It was, it was working well,
• 9:28
  revenue was growing and we had actually brought in another CEO
• 9:34
  to run the company to kind of free Luke and I up to well take
• 9:43
  a break from just running things for 10 years and and also think
• 9:46
  about maybe what was going to be, what was going to be next.
• 9:50
  So finally after we kind of ended our relationship with
• 9:54
  Crowdflower, we were working on a a few different things.
• 9:58
  So we had seen that deep learning was really it was like
• 10:02
  working, it was taking off. We saw that at at Crowdflower,
• 10:06
  it was, you know a lot of the reason that the company finally
• 10:09
  started working. Like all of these organisations
• 10:11
  needed lots and lots of data and we saw that was especially true
• 10:17
  in like computer vision and autonomy.
• 10:20
  So at Crowbar we were working with a lot of the bigger self
• 10:26
  driving car MLB teams. So we did a couple of things.
• 10:33
  One, we were like, well we haven't been doing machine
• 10:36
  learning for a while. We should probably brush up on
• 10:39
  the fundamentals. So Luke started teaching classes
• 10:43
  on deep learning. We we'd run like events in in
• 10:46
  San Francisco to teach like regular software developers.
• 10:49
  Hey, what is deep learning? How does it work?
• 10:52
  And we also were playing with robots quite a bit because we
• 10:58
  thought, OK, well these these people making these big self
• 11:00
  driving cars, maybe we can make some like sick robots.
• 11:02
  And who doesn't like as an engineer just like working with
• 11:05
  robots. I was like so much fun.
• 11:07
  And also at the same time Lucas actually got an internship at
• 11:10
  Open AI which Luke and I are both based in San Francisco and
• 11:14
  the Mission district. Their their office was right
• 11:16
  there. O Luke was able to to go work on
• 11:20
  a robotics problem at Open AI. He was working on like a
• 11:23
  grasping problem. So to figure out, you know how
• 11:27
  the robots going to plan its actions to pick up an arbitrary
• 11:30
  an arbitrary object. This, this must have been early
• 11:33
  days of Open AI. It was very early.
• 11:35
  Yeah. It was like within a year of the
• 11:37
  inception. Yeah.
• 11:38
  So, you know, we kind of decided a hardware company working with
• 11:44
  robots was as fun as it might be.
• 11:47
  It could be like a pretty hard business to, um, to make work.
• 11:52
  And while Luke was doing his internship at Opening Eye and
• 11:56
  while we were like teaching classes, it became clear there
• 11:59
  just weren't great tools, especially when you're working
• 12:04
  with lots of other ML practitioners, to just keep
• 12:07
  track of all the stuff that you're doing.
• 12:09
  So the inception of weights and biases was like make a tool that
• 12:14
  enables loop to keep track of his grasping model.
• 12:18
  Um and as a is like a full stack software developer which I
• 12:26
  identify as. This is like, it's my biggest
• 12:31
  dream. Like you have this blank canvas,
• 12:33
  you have this exciting problem, you don't have the craft of like
• 12:37
  10 years of of developing a big massive product.
• 12:40
  You can just go and kind of create something new.
• 12:43
  So I really threw myself into it and we were able to make a very
• 12:48
  rough but functional prototype, I think within a a month or so.
• 12:53
  Yes, that's how it all began. Well, thank you for giving me
• 12:57
  your pronouns, which are full and and now I know how to
• 13:03
  address you. And so the cool thing about
• 13:07
  this, I feel like you probably saw Excel.
• 13:10
  Spreadsheets being. Asked around via e-mail on like
• 13:15
  hey look, this is how I'm tracking my machine learning and
• 13:20
  you saw it and or Lucas saw it and probably thought we could do
• 13:24
  way better than this. Well, like Excel spreadsheets,
• 13:27
  those were like the the more advanced experiment trackers,
• 13:31
  like most would just have this big text file with their own
• 13:33
  weird arbitrary format. Yeah.
• 13:37
  Yeah, it's so funny because I know even in 2019 when I so I
• 13:42
  went to ODBC then and I remember talking to somebody and he was
• 13:47
  like, yeah, we just pass around express, Excel sheets and that's
• 13:52
  how we know which models are working or not.
• 13:55
  And I was like what we're in 2019 and how is that possible
• 13:58
  and so? I am guessing that you yeah, you
• 14:01
  saw the evolution to Excel spreadsheets and you were like
• 14:04
  at least it's a step up from the just the text documents.
• 14:08
  Ohh yes. But if you're using like a
• 14:10
  Google sheet, it's collaborative.
• 14:11
  You're like, you know. Ohh that is killer dude that is
• 14:16
  so wild to think about and. To think like.
• 14:21
  How far we've come and it feels like you went out, you created
• 14:25
  weights and biases. You started with that single
• 14:27
  problem of like, hey, how can we track and how can we make it
• 14:32
  collaborative, right. So we understand.
• 14:35
  Or was there another thesis that was from the get go you you
• 14:38
  wanted to hit on? Yeah, I mean it was really just
• 14:43
  like keeping track of stuff. I do remember like Luke
• 14:46
  specifically for his grasping problem.
• 14:48
  He was like what I really want is just be able to load this
• 14:50
  model up like on my phone and try it Ohh like in the real
• 14:55
  world. Like take a picture of something
• 14:57
  and just see what it does. We haven't made the universal
• 15:02
  inference iPhone app yet, but I still that's that's one that I
• 15:05
  would love to make. But yeah, I mean it also
• 15:10
  originally like we weren't that visual right?
• 15:13
  It was mainly about just keeping track of what script was run,
• 15:17
  what was the version of the code, what hyper parameters were
• 15:19
  passed in, what model was was generated.
• 15:24
  And then we kind of quickly saw just by working with users that
• 15:28
  they really liked seeing all of the lost curves, accuracy
• 15:31
  curves. At the time we kind of felt
• 15:32
  like, well, tensor boards like good enough there.
• 15:36
  But yeah, it turned out, you know, to be like probably the
• 15:39
  defining feature of of weights and biases, the the kind of
• 15:42
  visualisations and charts, yeah. Well, talk me through how that
• 15:47
  change of the product, like what has the evolution been in your
• 15:52
  eyes, you've seen it go from? That basic one.
• 15:56
  And then you're talking to users and you realise, well, we need
• 15:59
  visualisation. And then what other iterations
• 16:03
  felt like big unlocks for you along the way?
• 16:08
  Yeah, you know, in the beginning we had an idea of of what we
• 16:13
  wanted to make like for ourselves, but we have no idea
• 16:16
  if anyone else wants it. And getting people to just like
• 16:20
  listen or care is it's it's daunting, it's very hard and
• 16:26
  there's a lot of of nose or don't care.
• 16:28
  So in the beginning it was you know a lot of just like
• 16:31
  iterating on the core tracking stuff and then just wondering
• 16:34
  like is this something that will ever find product market fit.
• 16:41
  One of the early features that we were a bit surprised by, but
• 16:45
  I think proved to be another really powerful feature with
• 16:48
  some biases was hyper parameter sweeps.
• 16:50
  So we were working with Toyota Research Institute and they were
• 16:56
  saying, like, hey, if you could make it easier for us to run
• 16:59
  sweeps, that would be awesome. And we were kind of thinking
• 17:01
  like, well, we're not going to make some magic algorithm that's
• 17:05
  going to search the hyperparameter space better.
• 17:11
  There's plenty of other like tools out there that can that
• 17:13
  can orchestrate these things. Everybody's infrastructure
• 17:17
  slightly different like like how could we really provide value
• 17:21
  here? And it was just like making it
• 17:22
  really simple to run agents wherever whether you're in
• 17:26
  Kubernetes, SLURM or some homegrown orchestration system
• 17:31
  and then provide like really nice visualisations and and
• 17:37
  simple you know kind of reading the literature and saying all
• 17:41
  right what hyper primary algorithms are working and just
• 17:44
  implement it open source it say it's here that proved to be a a
• 17:49
  really popular feature with folks.
• 17:51
  Then you kind of step back and you think all right well how big
• 17:54
  do we want to dream? Like what?
• 17:57
  What could we kind of ultimately build here that that makes us
• 18:02
  the kind of de facto collaboration visualisation tool
• 18:06
  for for Mel. And it it was pretty clear from
• 18:10
  the beginning. It's like, alright, well we need
• 18:12
  really good data set and model management functionality.
• 18:15
  So we created Artefacts which is kind of a version big data
• 18:19
  store. We need really good kind of
• 18:24
  evaluation functionality to be able to for any given ML use
• 18:28
  case have nice rich kind of tables and to be able to run
• 18:34
  analysis on top of them. And then you know most recently
• 18:39
  we after gosh it's been four years now since we kind of laid
• 18:44
  out this big vision we released the production monitoring kind
• 18:48
  of on the inference side. Historically we've been you know
• 18:50
  mainly on the the training part of the the pipeline and
• 18:53
  evaluation parts of the pipeline.
• 18:55
  Now we have offerings at inference time to be able to
• 18:58
  kind of get those evaluation results in and and connect to
• 19:01
  like CI CD and kick off reevaluation.
• 19:04
  So you know, in the beginning it was just like keep track models.
• 19:09
  Then it became clear as working where, like how big could we go,
• 19:11
  What are the biggest problem points?
• 19:15
  And yeah, today we're closer than ever at being like a truly
• 19:19
  end to end OS platform. Yeah, and it feels like the.
• 19:27
  The idea of like, OK, we've got this one thing, we figured it
• 19:31
  out and. Now we're we can spread like
• 19:35
  let's land and expand and see where else people are asking for
• 19:39
  help because they're having pain is a great way of doing it at
• 19:43
  some point, you. Switched into the CSO role,
• 19:48
  right? What did that look like and how
• 19:50
  did that transition happen? Yeah, So at Crowdflower I ended
• 19:56
  up often being the the technical representative going out with
• 20:01
  the sales team, especially for the like really large deals.
• 20:04
  And I became very familiar with working with the infosec teams
• 20:08
  about, you know, how our software works, what practises
• 20:13
  we have at the company to ensure that software is secure.
• 20:15
  And had had done this a lot at Crowd Flower.
• 20:23
  As a developer, I've also like shipped my fair share of like
• 20:28
  bad bugs. So if you're like viscerally
• 20:33
  feel the pain of, you know, doing something that you know,
• 20:38
  Crowdflower, we would do these, these we had these micro tasks,
• 20:41
  which means we pay people like cents to complete some labelling
• 20:44
  task. And there were like a handful of
• 20:47
  times and they weren't always my bugs, but one or two of them
• 20:51
  were by bugs where, you know, everyone on the Internet is
• 20:54
  trying to game this system. Like if they can just do the
• 20:57
  work without actually doing the work and get paid, then you
• 21:02
  know, that's just easy money for them.
• 21:05
  And they're, you know, sometimes be little bugs there that that
• 21:08
  cost us 10s of thousands of dollars.
• 21:09
  You can like really feel it. So I've always like cared a lot
• 21:13
  about security and originally my title IT with some biases was
• 21:18
  CBP, which is also those are my initials Chris Van Pelt.
• 21:22
  But it kind of sounds like Corporate Vice President in the
• 21:27
  early days of Crowd Flower. I think my title was like Chief
• 21:29
  Awesome Officer, which I thought was funny.
• 21:32
  But anyway, I needed a title that like made sense and given
• 21:37
  my my past experience we went with Chief Information Security
• 21:42
  Officer. Since you know, I've been able
• 21:44
  to build this like amazing team of of really passionate security
• 21:49
  folks. And we've seen the security
• 21:52
  industry at large really kind of mature as well with with better
• 21:57
  tooling for the compliance programmes that need to be
• 22:02
  around like sock two or or ISO 27,001 and a lot of the like
• 22:07
  code static code analysis and and those deeper tools which in
• 22:12
  this current kind of explosion of AI, I think there's going to
• 22:16
  be a lot of really cool security related functionality coming out
• 22:20
  as well. So yeah.
• 22:23
  There's so many things that I want to ask you on this security
• 22:27
  thread, and really it I think we could probably start with as a
• 22:32
  machine learning engineer who is tangentially related to the
• 22:37
  infosec or dev SEC OPS teams. What are some things that you
• 22:43
  can? Give us as advice as like easy
• 22:47
  wins to make sure that a if we are bringing all new tooling.
• 22:52
  We've got our cheques and balances and we're good on that.
• 22:56
  And then be just like in general how can we assure success.
• 23:02
  I think first like start thinking about it from the
• 23:05
  beginning. Like even when you are a small
• 23:08
  startup working on something, there are you know a few things
• 23:12
  that will go a long way and ensuring you're going to be able
• 23:16
  to develop software securely. So one is like least privilege
• 23:22
  which is it's such it's tricky but it's it's really important.
• 23:29
  So like the easiest thing when you're like giving new engineers
• 23:33
  access to, say, your AWS account is give them like some
• 23:38
  permissions that let them do basically like everything that
• 23:44
  is bad. You want to, you know, limit the
• 23:46
  permissions of what anyone on the team can do and hopefully
• 23:50
  get that into Terraform and make it really easy to maybe give
• 23:52
  people temporary escalations and then bring them down.
• 23:58
  The other pieces are just using good security tooling, so setup
• 24:03
  scanning of any Docker containers that are being built
• 24:05
  for for Cdes. If you ever deliver these to
• 24:08
  enterprise customers, they're going to find them before you
• 24:11
  do. If you aren't doing that, which
• 24:12
  is embarrassing and it should be easy to just update the
• 24:17
  dependencies there and have, you know, vulnerability kind of
• 24:20
  management system like depend about on GitHub will tell you.
• 24:24
  I think it's especially gnarly in the JavaScript ecosystem.
• 24:27
  There's just so many dependencies and so many PC's
• 24:30
  out there. But, but staying on top of that
• 24:33
  is important because the your customers can see it and their
• 24:37
  teams are going to be concerned about these things and what
• 24:42
  else? Yeah, Ohh, I mean keep track of
• 24:44
  vendors like from the beginning when you decide to like use some
• 24:49
  third party service, first step back and say, OK, what data are
• 24:52
  we giving them and what classification does it fall
• 24:55
  into. So you can think of like highly
• 24:58
  confidential, kind of secret, but not terribly confidential
• 25:04
  and then just low risk. And for anything high you should
• 25:07
  be like confident that they are a mature vendor that has like a
• 25:11
  sock two type 2 at the station or some accreditation.
• 25:16
  Because eventually you're going to have to tell your customers
• 25:18
  who your sub processors are and they're going to, you know, be
• 25:22
  pretty concerned if you're using this other, you know, brand new
• 25:25
  startup that maybe you won't have those accreditations, but.
• 25:32
  And so now thinking about like. This new paradigm, I feel like
• 25:37
  it's a good place to transition into LM's and how much it's the
• 25:43
  wild, Wild West in this space and almost like as you.
• 25:49
  Are looking at LMS and trying to think through security and data
• 25:56
  access or just like the potential problems that you get?
• 26:01
  Have you? Got any advice on that front?
• 26:06
  Like, have you figured out ways to go about it to cover your ass
• 26:10
  there? Yeah, I mean, it very much is
• 26:14
  the wild, Wild West. Every large enterprise that I've
• 26:18
  talked to like some team is working on this and it just like
• 26:24
  using LMS within their business and they're all concerned about
• 26:28
  the kind of data privacy and security and truthfulness of of
• 26:35
  these things. What we're seeing, you know
• 26:39
  enterprises do is instead of again this like vendor issue
• 26:44
  that I mentioned with security of like all right, well who do
• 26:47
  we trust the data with? Many are willing to say send it
• 26:51
  to the the chat TBT multi tenant instance and they'll get like a
• 26:56
  private Azure shut up to. To mitigate some of that risk
• 27:02
  you also have prompt injection which is an unsolved problem.
• 27:06
  Like you can you can get out of the system prompt and make the
• 27:09
  model do weird things. Which if you're then kind of
• 27:13
  using the outputs of the model to call functions or do other
• 27:17
  actions like just opens up this this big gnarly rats nest of
• 27:21
  possible injection attacks. So you have to really treat all
• 27:25
  of the data coming in or out of these models is untrusted and
• 27:30
  it's sure that you're they're executing that code in a really
• 27:34
  safe environment when you're doing it.
• 27:36
  Yeah, it's funny because I was just talking to a guy this
• 27:40
  morning and he was saying how? The idea of having a fine-tuned
• 27:45
  model on your data is like it's great in theory, but then how do
• 27:51
  you protect against these things?
• 27:53
  Like prompt injection can be really bad, or just data access.
• 27:57
  All of a sudden people have whatever data access they want
• 28:00
  because they can just prompt the model and it won't give them
• 28:03
  anything. And it's really hard to figure
• 28:05
  out how to deal with data access on a fine-tuned model on this
• 28:11
  potentially sensitive data. Yet often, I mean, some of the
• 28:15
  coolest use cases are like, alright, the model's going to
• 28:17
  generate SQL and then you can go and run that sequel.
• 28:20
  Well, you'd better make sure that whatever users running the
• 28:23
  SQL only has access to the data it can have.
• 28:26
  The model could generate code and execute code.
• 28:29
  All right, what? What can the sandbox the code is
• 28:32
  being run in do? All of these things are
• 28:35
  important to to think about. The other piece I'd say is just
• 28:38
  having visibility into this is I think the most important thing.
• 28:44
  So you know this comes back to the the feature that we just
• 28:49
  released a weights and biases or LM monitoring tool.
• 28:53
  We have a a proxy now that is like fully compliant with the
• 28:57
  Open AI API. So customers can kind of take
• 29:00
  any of their code that's hitting those APIs, change a couple
• 29:04
  environment variables and now everything goes through a proxy
• 29:08
  and everything is tracked. So we capture you know cost by
• 29:11
  user and the actual inputs and outputs that are coming from the
• 29:17
  API. This we like put into park files
• 29:21
  which are really easy to kind of ingest and some data pipelines
• 29:26
  to kind of detect. Maybe ohh is someone trying to
• 29:28
  inject prompts, Is there PPI being leaked?
• 29:32
  Is the model hallucinating kind of doing offline evaluations?
• 29:38
  These are other kind of mitigating ways to to solve for
• 29:41
  this. Excellent.
• 29:42
  So the other piece that I think is fascinating to dive into when
• 29:48
  it comes to LM's and just the journey of weights and biases in
• 29:52
  general. As you've seen it over the
• 29:56
  years, it feels like. There is.
• 30:00
  As weights and biases grows, you start to become useful for.
• 30:05
  New personas. And in the beginning it's like,
• 30:08
  Ohh yeah cool. Like the data scientist slash ML
• 30:11
  researchers are very interested in this.
• 30:14
  But then you start to add new functionality and it's like, ohh
• 30:17
  wait a minute, more engineers are interested and then LM's
• 30:20
  blow up on the scene and now software engineers are
• 30:23
  interested and so I'm wondering like.
• 30:28
  How do you feel? It's different slash the same to
• 30:33
  build for these different personas.
• 30:35
  Like what do you look at when you're trying to evaluate from
• 30:40
  the start, we really targeted 1 core persona and it was the the
• 30:45
  ML engineer and we really targeted like the the best ML
• 30:51
  engineers, right? Like we were working with the
• 30:54
  researchers at Open AI at companies like Toyota Research
• 30:58
  Institute in the very beginning because we knew you know the
• 31:02
  people solving some of the hardest problems.
• 31:04
  If we can really build tools that that help them, the the
• 31:08
  folks that are like just starting their journey with with
• 31:11
  ML are going to benefit from these things and we'll help them
• 31:14
  kind of get to the place where they can do as advanced stuff as
• 31:18
  as the the leaders in the field now is LLM's have kind of you
• 31:24
  know, really opened up the the personas.
• 31:29
  Floodgates, man, For sure. It's a It is a hard.
• 31:34
  It is a hard problem. I think there are.
• 31:35
  There are there's there's overlap between, say the ML
• 31:41
  engineer and the software engineer, and I think maybe some
• 31:43
  of the software engineers don't realise it.
• 31:45
  But you know, if you're interacting with these API's or
• 31:49
  using one of these open source models, you really need to have
• 31:54
  a formal way to evaluate it. Pops, you have to immerse
• 31:58
  yourself in the ML OPS content. The best way to do it is to
• 32:03
  subscribe to the Inner OPS Community Podcast.
• 32:07
  So good luck and keep learning. It was it was like the same the
• 32:17
  same problem we had at Powerset before we started Crowdflower.
• 32:21
  Like when you're using probabilistic engine and that
• 32:27
  engine might change over time and the way you interact with it
• 32:29
  might change, you need a way to like measure that and know
• 32:32
  whether or not the thing you're about to push to production is
• 32:35
  better or worse than it was before.
• 32:38
  So that is like very aligned with what ML engineers are doing
• 32:42
  while they're training and evaluating and and pushing
• 32:45
  models. So I think, uh, yeah, figuring
• 32:50
  out ways to really educate and empower those, those traditional
• 32:54
  software developers to start thinking a little more like ML
• 32:57
  engineers is, you know, it's something we did when we were
• 33:01
  first teaching classes as well. It's something we're we're
• 33:03
  familiar with. So that is often you know you've
• 33:06
  got this like really complex tool, How do you hide a lot of
• 33:09
  the complexity such that people can kind of on board and then
• 33:12
  slowly be revealed more of the power features and and
• 33:15
  complexity. I think it's also like a great
• 33:20
  opportunity maybe incorporate large language model
• 33:23
  functionality. So instead of needing to like
• 33:24
  click around and do 1000 things you could just ask the product
• 33:29
  what to do and it would it would do it.
• 33:32
  We're actually we're doing a we do a hackathon twice a year at
• 33:36
  weights and biases and the next week is our our hackathon.
• 33:38
  So I think I I just had my idea for the.
• 33:41
  Oh, there you go. Yeah, you you got it.
• 33:44
  No team up and you're in it and you envision it just like the
• 33:50
  onboarding is you asking can I do this, can I do that?
• 33:53
  How do I do this? What does this look like?
• 33:56
  And it's, it's basically taking the docks, ingesting the docks
• 34:01
  and then. Also ingesting all those
• 34:03
  onboarding and making sure that well that would be more like a
• 34:05
  chat. But I think my dream would be
• 34:07
  like for the really complex UI when we've got like 50 different
• 34:10
  settings for a chart or some way to like compare things.
• 34:13
  You could just like ask it, you know, show me a confusion matrix
• 34:17
  of wrong or right answers between the latest three runs or
• 34:20
  something and then it it generates the appropriate kind
• 34:24
  of UI elements and and queries to do it.
• 34:27
  But that might be a project that's a little gnarlier than a
• 34:31
  one week hackathon. But I don't know that these
• 34:34
  elements are wild man. It's a great start to get it
• 34:38
  rocking and then if there's value like well you can see and
• 34:43
  I guess that is that is another question just pulling on that
• 34:46
  thread a little bit is. How do you figure out if
• 34:49
  something has value you hack on that you're like, wow, I love
• 34:53
  this, but I don't know if anybody else will.
• 34:55
  Are you doing these kind of rolling roll or these
• 35:00
  incremental rollouts as we could say on your own products and
• 35:04
  seeing do the people that are using this, do they find this as
• 35:09
  valuable as I think they'll find this?
• 35:13
  Yeah. So I mean for this Sir we've
• 35:16
  we've done but I guess you could call like demo driven
• 35:21
  development. So when we have an idea for what
• 35:24
  we think is going to be useful to folks, we'll put together a
• 35:30
  prototype. And you know, pieces of it might
• 35:33
  not actually work, but we try to make it look as good as we can
• 35:37
  And then we go sit down with maybe an existing customer or
• 35:40
  someone that's like friendly and willing to engage and say hey
• 35:43
  let us tell you this story, what is and then gauge their reaction
• 35:47
  and and see if they have feedback and kind of guide us to
• 35:49
  the the next step. Then, you know, it's often like
• 35:55
  once we feel convinced enough that yes, this is, this is
• 35:58
  something that you know, good enough percentage of the markets
• 36:03
  gonna want, we get it into a functional state.
• 36:07
  But that's really when the work begins, right, because it's all
• 36:13
  about the, the details, the the many, many iterations on
• 36:18
  different aspects of it that make it great and and
• 36:21
  delightful. And that's just kind of
• 36:23
  relentless working with with customers and looking at what
• 36:27
  data we can get just from from actual usage and and being very
• 36:32
  data-driven and try to like get those numbers up.
• 36:35
  Yeah. So for example, let's take this
• 36:38
  new release that you're coming out with like the monitoring,
• 36:41
  right, or the evaluation? Like, what does that actually
• 36:44
  mean when you're iterating on it?
• 36:46
  Is it? Just UI things.
• 36:48
  Is it actual API calls? Like is it where it sits and the
• 36:53
  full on functionality? Like how much are you willing to
• 36:56
  just go in and butcher a feature if you're getting feedback from
• 37:01
  your existing customers that say, yeah, we would want this,
• 37:07
  yeah, I mean we, we try to be as, as open minded and
• 37:09
  experimental as possible. I think especially when you're
• 37:12
  running an enterprise software business, there is a danger in
• 37:17
  just doing every little thing that every company wants and
• 37:22
  then suddenly you you have this kind of a big disjoint, not
• 37:28
  unified platform. So we are careful to say like
• 37:32
  alright, well is this, is this reasonable?
• 37:34
  Do we think other customers will also benefit before we decide to
• 37:39
  like experiment with something? One thing we know is evaluation
• 37:44
  is that every use case is very nuanced and different.
• 37:49
  So you can't just make like an LM evaluation tool because
• 37:54
  depending on what you're asking the LM to do, it's going to you
• 37:57
  know be potentially very complex process to perform the the
• 38:03
  evaluation. So all of our, our new offerings
• 38:09
  that we've released in the last couple of months have been built
• 38:12
  on top of this toolkit we made called Weave that is open source
• 38:17
  and it's kind of like Streamlet but with actual types and
• 38:21
  extensibility. So it's like, yeah, I won't, I
• 38:26
  won't do anymore analogies, but we're building a lot of this new
• 38:30
  function functionality on top of it which allows us to experiment
• 38:35
  quickly and it even allows the customer to kind of customise or
• 38:38
  change these dashboards themselves.
• 38:40
  So it's not this hard coded like React dashboard that then we
• 38:44
  have to go have people go and change.
• 38:45
  It's something that is is similar to like our current
• 38:49
  reports functionality. We can kind of go in and tweak
• 38:51
  and move things around and try to get it you know exactly right
• 38:54
  for for your use case. And it gives people that
• 38:57
  flexibility and you're not super opinionated on it and so then
• 39:02
  they can say, alright, this is actually what's important to me,
• 39:06
  like you said, especially with evaluation.
• 39:10
  It's case by case basis. So you can't just pull on, say,
• 39:14
  this is the evaluation tool and this is how we're going to use
• 39:17
  it, because that's going to get. Stale real quick.
• 39:22
  Yeah. I mean, this whole space is
• 39:23
  moving so quickly. Like the, the state-of-the-art
• 39:25
  on how to evaluate certain use cases could very well, you know,
• 39:29
  change tomorrow so. Yeah, totally, totally.
• 39:32
  I mean, I know that we've been. So we put out an evaluation
• 39:36
  survey about a month ago and we've had over like 100, like
• 39:44
  120 people have written in and said what they're doing, how
• 39:48
  they're evaluating and what's funny to me is how.
• 39:52
  One thing that people are, the majority of people are still
• 39:56
  using these traditional metrics to evaluate and it feels like
• 40:01
  those aren't ideal, but it is something that people have just
• 40:05
  been trained to use and so that's like, well, I guess it's
• 40:09
  something. And it's very much you are
• 40:13
  seeing this current state of things where people are like I'm
• 40:16
  trying to evaluate, but I don't really know if what I'm
• 40:21
  evaluating is actually making sense.
• 40:24
  And so there's this huge question mark, at least from
• 40:27
  what I've been seeing in the data and that it's all open
• 40:30
  source. I'll send you the link in a bit
• 40:32
  because we made all the responses open for anyone to
• 40:34
  check out. And this data is fascinating to
• 40:38
  see like people are I I don't want to say foolish struggling
• 40:44
  but you know like a lot of people are like, ohh, we just
• 40:46
  have a thumbs up thumbs down and we're good and you're like dude,
• 40:49
  that's not good enough. That is not how you can call it
• 40:53
  quits after a thumbs up thumbs down.
• 40:55
  That's not going to be like the be all to end all.
• 40:58
  I just don't see that working that well.
• 41:01
  Yeah, I mean, I go back to the early days working on the search
• 41:05
  engine and the ranking algorithm.
• 41:07
  How do you evaluate that? Like how do you know, you know,
• 41:10
  we asked these these users how good a result is.
• 41:12
  It's very ambiguous. It's like pretty subjective to
• 41:15
  like take a query and say, OK, how good of a match it is.
• 41:21
  You'd see often to like big companies like eBay or a you
• 41:24
  know, a big. Ecommerce retailer is you have
• 41:29
  the MLT team trying to make the results like really relevant
• 41:33
  using this like relevance metric but then you have the business
• 41:35
  trying to sell product. So it's like ultimately like
• 41:39
  which product sold now do you want the relevancy or do you
• 41:42
  want to optimise for you know selling more product and how you
• 41:46
  how you align everyone is is is a real challenge.
• 41:49
  But what's important is you're like.
• 41:51
  Tracking it so that you don't in a month push a button to deploy
• 41:55
  some new prompt template or a new model and it like tanks,
• 41:59
  whatever. You know, the core business
• 42:01
  problem. Dissolving.
• 42:04
  Yeah, exactly. It's like almost.
• 42:06
  You're just gotta make sure of the huge dangers and then the
• 42:12
  other stuff, Alright, we'll tweak it, we'll make it a little
• 42:15
  bit better. But it's super subjective.
• 42:17
  Yeah, we can deal with that and we're trying to figure that out.
• 42:21
  But yeah, cover your back first and foremost and make sure.
• 42:25
  These bugs, like, the bugs, are so subtle with these ML models
• 42:28
  because they're like, they're not going to air.
• 42:30
  They're not going to throw an exception.
• 42:31
  They're just going to, you know, just.
• 42:33
  Like, subtly wrong, yeah. And it's so hard to detect
• 42:38
  oftentimes. Yeah, that's that's so funny
• 42:40
  cause Rohit, who I was just talking to all in the podcast,
• 42:43
  he was like maybe we need a new type of HTTP number, you know
• 42:50
  it's not like a 404, it's not a two or two, it's like a 224 or
• 42:54
  something that is like yeah you got a response but it's not
• 42:58
  really accurate response or it's not like to be trusted, you know
• 43:03
  and that's that's kind of the the http://that.
• 43:06
  You'll get. Already there, I forget.
• 43:07
  I forget the exact number, but there's an I'm a teapot HTTP
• 43:11
  code that's in the official spec.
• 43:13
  No way. Yeah.
• 43:14
  Wow, should look it up. There we go.
• 43:16
  That's a nice goal. What is it?
• 43:19
  Breadcrumb. So as we're moving forward
• 43:22
  through the landscape of MLM's and the evolution of it and you
• 43:27
  traditionally kind of been ahead of the curve on things.
• 43:31
  You started an MLM company in 2007.
• 43:34
  So that gives you a bit of a track record, I think.
• 43:37
  Where's your head out right now? What are you thinking about in
• 43:41
  the landscape and how things are evolving and where they're
• 43:44
  going? Yeah, I mean, the space is so
• 43:47
  exciting and I'm just like. Always pleased to see, you know,
• 43:53
  every day something trending on Hacker News that's related to to
• 43:57
  all this stuff and a lot of the interesting papers coming out of
• 44:00
  the innovations. You know where all of this is
• 44:03
  going to end up? I don't know.
• 44:04
  I think a space that's especially interesting is the
• 44:07
  open source model space and kind of running these things on the
• 44:10
  edge and. You know, selfishly, I'd also
• 44:13
  love it if lots and lots of companies start fine tuning
• 44:17
  these smaller models and they use weights and biases to do it.
• 44:20
  I think the multimodal stuff is is really interesting, like just
• 44:24
  playing with chats, UBT with images is like pretty magical
• 44:27
  and I think we'll just continue to see more and more modalities,
• 44:31
  being able to be intertwined to do novel things.
• 44:38
  Yeah, and it's, it's exciting. I think the other areas like
• 44:41
  these agents, I think they don't work great today, but they're
• 44:46
  like interesting enough and kind of magical enough that.
• 44:51
  People are, you know, continuing to kind of experiment and work
• 44:54
  with them. I think as these models get
• 44:55
  better the the power of the kind of like do anything agent is
• 44:59
  going to be really interesting and a little scary.
• 45:03
  But yeah, yeah, the Agent one definitely has captured, I think
• 45:08
  everyone's attention And in a way it is like you're saying,
• 45:11
  like, it's not good enough yet, but there's enough of a spark to
• 45:18
  where we realise this could be something and maybe it's in a
• 45:22
  few months, maybe it's in a few years, but there's something
• 45:25
  here and we should pull on that thread a little bit more.
• 45:28
  Yeah, totally. So I see you creating the
• 45:33
  future, making it happen. Chris, this has been awesome.
• 45:35
  I really appreciate you coming on here and chatting with me.
• 45:38
  And last question I have for you.
• 45:42
  What was Conan O'Brien like when you were an intern on a show
• 45:45
  back in the day? It was.
• 45:48
  It was pretty awesome. It was, you know, it was an
• 45:52
  unpaid internship. We paid ourselves with access to
• 45:55
  the soda fridge and then when we would order dinner for the
• 45:59
  writers, we could also order dinner for ourselves.
• 46:01
  So that was our payment. That and like, getting to see a
• 46:04
  bunch of cool movie stars and bump into to Conan, who was very
• 46:09
  tall. He's a very tall human being,
• 46:12
  but also very kind. And it was, you know, super
• 46:16
  exciting to to get to work the writer.
• 46:18
  Were you interning as a writer or just as a cameraman?
• 46:22
  What was your? Insurance.
• 46:24
  I was a production assistant, which means basically an errand
• 46:27
  boy. I would get writers coffee.
• 46:30
  I would go to the Virgin Megastore and buy a CD that they
• 46:33
  wanted to use in one of the skits.
• 46:35
  I would like help the prop guys with stuff.
• 46:38
  It was like anything and everything.
• 46:41
  Yeah, Was Andy Richter around on the show back then?
• 46:45
  At that time, he wasn't. Yeah.
• 46:47
  Ohh OK Ohh, awesome. Dude.
• 46:49
  What, what an experience. And thank you so much for coming
• 46:53
  on here. I appreciate this and I love
• 46:55
  everything that you're doing. It waits and biases.
• 46:57
  I'm excited for the new features that you're putting out and I
• 47:02
  encourage everyone that's listening to go check out this
• 47:06
  new monitoring and evaluation tooling that you've got.
• 47:10
  They now have access to. And there's this weave platform
• 47:13
  that people can go and mess around with, which is totally
• 47:15
  open source. So thanks man.
• 47:19
  I think we'll cut it there. I love it, man.
• 47:21
  Thanks for having me. This is fun.
• 47:27
  This is Skyler. I lead machine learning at
• 47:29
  Health Rhythms. If you want to stay on top of
• 47:32
  everything happening in ML OPS, subscribe to this podcast now.
• 47:38
  Now now. Now now.
• 47:44
  Now. Now.