Latest / Agentic Conversations (formally mlops.community) / MLOps Build or Buy, Startup vs. Enterprise? // Aaron Maurer & Katrina Ni # 157

Transcript

MLOps Build or Buy, Startup vs. Enterprise? // Aaron Maurer & Katrina Ni # 157 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey, I'm Aaron our, I'm a senior engineer manager and slack and I
• 0:04
  take my coffee, no cream but brown sugar.
• 0:08
  Hey, I'm Katrina. I'm a mushroom and engineer a
• 0:11
  nice lat. So I don't.
• 0:13
  Yeah, I don't drink coffee but I drink tea.
• 0:17
  I have like the student key beside me at the moment.
• 0:22
  Yeah, welcome back to another ml.
• 0:25
  Ops Community podcast. You are here with none other
• 0:29
  than Host Demetrios. And today, I am joined by an
• 0:34
  incredible special guest, Jake, what's up dude?
• 0:39
  Hey, glad to be back on the podcast.
• 0:41
  So as, you know, I cannot think of the two words recommender and
• 0:46
  system together without thinking about you.
• 0:49
  Why is that man? Why can I not get bad
• 0:51
  Association out of my head? Yeah, so for every, what does it
• 0:55
  know? I'm a baby.
• 0:57
  I'm a tech lead at a tectonic feature platform.
• 1:00
  Buddy now, but prior to that, I was at YouTube for six years and
• 1:04
  it YouTube I worked on video recommendations particular.
• 1:08
  I focus on like long-term satisfaction metrics and to this
• 1:13
  day I end up talking through my current company to a lot of a
• 1:17
  lot of smaller companies, doing recommendation systems.
• 1:20
  And I've become Dimitri wanted to meet your host is go to guys
• 1:24
  on how on recommendation systems and occasionally co-hosting a
• 1:28
  recommendation systems Focus. Podcast, the accept our YouTube
• 1:31
  videos still. Don't get more than 20 views.
• 1:33
  So I don't know what the hell is going on there.
• 1:35
  That recommendation system on that platforms broken, man.
• 1:39
  It's gonna stop look, and maybe where it may be working as a
• 1:41
  tinted. I don't know to be sure that
• 1:43
  work under your thumb nail polish, maybe it's also it's
• 1:45
  broken. Shit, don't tell me that.
• 1:47
  So we got another wreck cyst tangentially, Rex's
• 1:52
  conversation, and Jake did what he does best and he dove into
• 1:57
  all the nitty-gritty details, which I'm sure.
• 2:00
  Everyone out there loves as they have told me before in the past.
• 2:03
  Jake, I imagine they've reached out to you to know what were
• 2:06
  some of your key takeaways here from this conversation, man.
• 2:10
  The the B2B considerations on data privacy and feature,
• 2:15
  engineering for slack. We're really, really interesting
• 2:18
  and I love there. It seemed very thoughtful about
• 2:22
  the team seemed very thoughtful about how they build Their
• 2:26
  existing infrastructure in a way that will meet all their needs
• 2:30
  and scale out. It seemed like they had a good
• 2:32
  balance of like shipping for all of you use cases and not over
• 2:36
  building and I think my favorite one made me.
• 2:39
  Why another favorite aspect was, they have you resounded,
• 2:43
  internal customers. You need to think about how to
• 2:45
  sell the product managers. So how do you how do you prove
• 2:48
  value in quick value of the product managers?
• 2:51
  It's it felt like you know if they felt like they were selling
• 2:54
  a product. Mmm, that's so good.
• 2:58
  I mean you said it perfectly where it's just like, how can we
• 3:01
  make this API? Like how can we hit the nail on
• 3:05
  the head with the API to give it just what it needs to do and
• 3:09
  make it very clear to these product managers what you're
• 3:12
  trying to do and how they can get value out of it.
• 3:14
  So then they can adopt it real quick and start using it.
• 3:17
  Put it to use, get that value. And then start adding more of
• 3:22
  that land and expand deal for everyone that has been on the
• 3:25
  Sales side of things. So I've really enjoyed that.
• 3:28
  I also enjoyed knowing that they both came from data science but
• 3:33
  now the way that slack structures their teams is that
• 3:38
  ml Engineers are everything, so they're a little bit of data
• 3:43
  Engineers there, a little bit of data scientists, they're a
• 3:45
  little bit of machine learning Engineers where we would
• 3:47
  typically have fall under machine learning Engineers, so
• 3:51
  you can be expected to do anything from the data
• 3:55
  engineering. Ringside of things to the model
• 3:57
  building to the actual deployment.
• 3:59
  If you are an ml engineer at slack, which I thought was
• 4:03
  fascinating because coming from a data scientist to like, having
• 4:07
  to know all of that is, it's not the easiest of Feats.
• 4:13
  Yeah, I think Aaron Aaron perspective as an engineering
• 4:16
  manager was interesting. I did you focus on, you know,
• 4:19
  because you can teach software engineers and say, hey we can
• 4:22
  grow you into a ml Direction. And you can take data scientist.
• 4:26
  See, we can grow you into a software engineering Direction.
• 4:29
  He said it's really easy to hire for his team and he's able to
• 4:32
  bring in really good talent. So I thought that was a great
• 4:35
  perspective. I think that might be a really
• 4:37
  good. I really effective way to
• 4:38
  structure ml teams at companies, you know, if you can staff a
• 4:42
  full ml engineering team, there you go.
• 4:45
  Jake. I can't thank you enough for
• 4:47
  coming on and I also want to just mention something you can't
• 4:50
  see it really? But oh what do I got?
• 4:52
  I'm wearing my car weather here. I got some tech tons, whack, I
• 4:57
  conference I did it just for you, man.
• 5:00
  I'm just showing it people that are listening.
• 5:02
  You can't see this right now but I am just slightly just a little
• 5:08
  bit of sweat moving like that Stover.
• 5:10
  So you can see right over my heart, there's a tecton sign and
• 5:15
  we want to give them a huge, thanks for sponsoring this
• 5:19
  episode and sponsoring the community.
• 5:21
  Tecton has been an amazing sponsor for the three years
• 5:26
  we've been around. So without further Ado let's
• 5:30
  just get into this conversation man.
• 5:38
  So one thing that is going to be confusing for the whole time
• 5:42
  that we're talking whenever you all refer to slack, I am going
• 5:46
  to instantly think about the MLS Community slack because whenever
• 5:50
  I say slack, that's instantly what I think about.
• 5:53
  However in this His specific appearance.
• 5:58
  We have two of the employees of slack.
• 6:01
  So when you talk about slack, you're talking about working at
• 6:03
  slack so Katrina and Aaron. It is a pleasure to have you
• 6:06
  both on here. How you all doing?
• 6:09
  Yeah. Thank you, Demetrius.
• 6:10
  I'm doing great. And then super excited to join
• 6:13
  this podcast. Like I've I've known about this
• 6:17
  community for a while and then I like you're doing a great job.
• 6:21
  Just, you know, educating people about Emma cops.
• 6:24
  Oh, You're too kind and Aaron is joined by his little offspring.
• 6:31
  That just appeared in the world. Yeah, it's a great to join you
• 6:35
  all. I'm on the four month stint of
• 6:37
  Parental leave right now, so it's nice to take a break from
• 6:40
  Sydney to Caleb's diapers to come chat about ML.
• 6:44
  And Kayla probably has lots of opinions to and you made him
• 6:47
  cry. Melt once in a while.
• 6:49
  Yeah, I love that. Well, if so I think This
• 6:55
  conversation originally got kicked off because you all wrote
• 6:58
  an incredible blog post around recommenders and search.
• 7:02
  I think when it comes to slack and how you all are doing that
• 7:06
  at slack, maybe we can just go over that real fast and then
• 7:11
  we'll dive into some of the specifics.
• 7:14
  Yeah, I can give like a brief introduction to it.
• 7:19
  So that whole sort of article and Technical stack and grew out
• 7:24
  of what it is server, recommending Pi where I think
• 7:28
  you know a lot of times I companies there's maybe a couple
• 7:31
  really high value recommenders that you have sort of bespoke
• 7:35
  things. Like you know, when I was at
• 7:37
  back at Airbnb like we had a recommendation page for
• 7:40
  listings, the Civil War was things, where was Sir 2001 often
• 7:43
  you didn't have to make repeatable and for our anything
• 7:48
  for it but it's like sir. But we found ourselves doing is
• 7:51
  there's a whole slew of different situations where you
• 7:54
  want to recommend. Users are channels in some
• 7:57
  countries like there's a, there's a hundred different
• 7:59
  products, you can kind of set up that followed basically that
• 8:02
  framework. So we kind of started thinking,
• 8:04
  well how can we make this generic both in terms of like
• 8:08
  how we present it to other teams of the company and how we kind
• 8:10
  of serve the stuff on our back? So we started out, we serve
• 8:14
  having an API where other teams could start calling these
• 8:17
  recommenders and it's generic way.
• 8:18
  And over time, the team built out this like really, I think
• 8:22
  pretty complete suit of tools to like build.
• 8:25
  View recommenders train models, get trainee, did it sort of like
• 8:29
  the full stack of recommendation and that blog post serve
• 8:33
  describes all the pieces and how they got built?
• 8:36
  Yes. As I was reading that the block
• 8:38
  does that immediately jumped out that you all didn't have one
• 8:41
  tent pole. Most companies have a tent pole,
• 8:43
  machine learning to use case and Rex's use case that they build
• 8:46
  around and put smaller teams kind of gum and leverage that
• 8:49
  infrastructure. It's like have anything like
• 8:52
  that before or is it The having many different smaller racks,
• 8:57
  use cases, seems to kind of beg for an infra first solution.
• 9:01
  Yes. I mean, I think in terms of
• 9:04
  History, our tent pole was probably search.
• 9:06
  Like that's where we've invested the most historically around ml
• 9:11
  at slack. And I mean that's probably true
• 9:13
  of most places. But, you know, these things we
• 9:16
  you reuse from search for this recommend API like, you know,
• 9:21
  there's a bunch of data features we use in our models that are
• 9:23
  shared really It was kind of like the needs of
• 9:26
  recommendation. Actually dictated a somewhat
• 9:28
  different approach so I would say it wasn't really an
• 9:32
  outgrowth of search in that way. Yeah, I had surgery
• 9:34
  recommendations often ever. So my background is from YouTube
• 9:37
  and I were to be two recognitions.
• 9:39
  We shared a lot of components with search but it is
• 9:41
  fundamentally a different problem and so typically require
• 9:45
  really different Stacks. Yeah.
• 9:47
  You know it's a kind of think about them at the basic level
• 9:49
  I'd like you know, you build them the same way of recruiting
• 9:52
  Wolfies, you have a re-ranking phase, you have some role.
• 9:55
  Basically is at the end but, you know, for retrieval.
• 9:58
  Usually start with solar for something like surgery
• 10:00
  elasticsearch. And for recommendation, it's
• 10:03
  usually some other data source like we, in fact, we had serve
• 10:07
  set up. So you could use a bunch of
• 10:09
  different sources for where you Source, your base
• 10:13
  recommendations from. Can you talk a little bit about
• 10:15
  the Genesis of this framework, or of this system?
• 10:18
  So was it you all had searched and you had all these other
• 10:21
  ideas for smaller, recommendation, use cases, but
• 10:25
  No Solutions in production and you wanted to build out one
• 10:29
  piece of common in fraud, so there was a Grand Vision or did
• 10:31
  at the story play out some other way.
• 10:33
  The original Genesis of it was an engineer neighborhood out who
• 10:36
  is somewhat since left. But any is basically, he had
• 10:39
  sort of Solved the same problem. Basically the same way a couple
• 10:45
  times, he's like how we should really make this generic and
• 10:47
  that was kind of when he set up this originally Pi.
• 10:51
  So the API itself like the You know, it's basically you have a
• 10:55
  query, which is some context of users and channels, it spits out
• 10:59
  some users and channels and behind.
• 11:00
  That was like, very, very served simple tools, kind of three
• 11:05
  competitions of some things we had done before, but that was
• 11:07
  sort of the basic version of it. And I think we've kind of built
• 11:12
  an extended on that because like one thing we found, we need to
• 11:15
  do is when partnering with other teams in the company.
• 11:17
  We need to make serve the interface with which they
• 11:20
  interact with us. Like, really simple.
• 11:22
  Like, you know, if you start throwing Around talking about
• 11:25
  embeddings, or she learning models that can kind of throw
• 11:29
  people off. But if you just say, hey, here's
• 11:31
  an API. Here's a basic idea of what you
• 11:33
  send us. This query will will do the rest
• 11:36
  and send you back good recommendations.
• 11:38
  I think that kind of they made it very easy to start onboarding
• 11:41
  other teens and that allowed us to serve start building back
• 11:44
  from that basic API. Okay?
• 11:47
  So that the special, the Special Sauce here and made me for other
• 11:50
  MLM proteins. Having a simple The Firm API and
• 11:55
  then by now you start simple and then sounds like you all built
• 11:58
  up complexity and sophistication behind that very simple API over
• 12:01
  time. Yeah.
• 12:02
  Like we'd had this product called parsing which is served
• 12:08
  in Bedding, nearest neighbors look up.
• 12:09
  So you could likelihood embeddings of users and channels
• 12:12
  based on their semantic content and like you could say like,
• 12:15
  hey, here's a channel. Give me some antique week
• 12:17
  simpler channels and like, demo people.
• 12:20
  I think that's like fairly straightforward, but like, when
• 12:23
  you start talking about, Cosine similarity, then beddings to
• 12:25
  like, your average like back end engineer or front energy or
• 12:29
  they're like, I don't know. What the hell this is so, you
• 12:31
  know, team's kept coming to us saying like, hey, we want this
• 12:35
  sort of MLS use case and be like, oh yeah, well here's
• 12:38
  parsec. Here's embeddings.
• 12:39
  Here's you know, how you do nearest neighbor search in
• 12:42
  there? Like, I'm just going to go do
• 12:45
  something else that seems hard and just like, when we distilled
• 12:48
  it down to this like recommend API, I think it became a lot
• 12:52
  easier to Yeah, partner with other teams and deliver stuff.
• 12:57
  Wait, can we pause there for a second because if I'm
• 12:59
  understanding this correctly you had teams that were coming to
• 13:03
  you with asks, and they didn't realize it was really machine
• 13:07
  learning. And then if you would go back to
• 13:10
  them and say like, yeah, we can do all this complex, jargony
• 13:14
  stuff, they would be like, I don't know what that means and
• 13:18
  it does that even accomplish my problem, I don't know.
• 13:21
  But then when you go back to him and you say, hey look here's
• 13:24
  Here's the simple way of fixing it with a recommender system
• 13:28
  that we all understand what that is more or less, then they were
• 13:33
  on board, and they were interested.
• 13:35
  So it was almost like that translation layer or you both
• 13:38
  had to be speaking the same language before anything could
• 13:40
  get done. Yeah, I mean, I think that was
• 13:43
  it, honestly, I mean, you know, it's often team, you know, maybe
• 13:46
  an engineer, who was trying to prototype something out and had
• 13:49
  a generic idea, but it was definitely like You know,
• 13:53
  exposing the complexity of our profession kind of made it a lot
• 13:58
  harder to partner with people. Well, you know, distilling it
• 14:01
  down to the thing to actually care about that fling that, that
• 14:04
  helps yeast of the partnering. I kind of just wanted to add
• 14:09
  that. We also worked with p.m. from
• 14:12
  other teams, a lot and then with this recommend API, it's really
• 14:16
  it's just really fast to boost job in your recommenders.
• 14:18
  So like, waiting a week we can just add a new recommender and
• 14:22
  then we are Also have a like a test page where people can just
• 14:25
  test the results, so like p.m. can't like even though they
• 14:28
  don't know about ml, like inviting like all those kind of
• 14:31
  things, they see the results and then it's much better than like,
• 14:35
  you know, heuristics. Like they may have some
• 14:37
  heuristic in their mind and then like it's just much better than
• 14:41
  they expected. Then they are kind of like
• 14:43
  having more trust with working with our team.
• 14:45
  It's just like this iteration and then it's also very easy
• 14:49
  just if they are not happy with some hope, the results we can
• 14:52
  just Iterate really fast and then just by changing feature
• 14:56
  ways and stuff like that. Yeah, yeah.
• 14:57
  Just like, better collaboration with other teams I guess.
• 15:00
  Greater. Oh well, just gonna so like,
• 15:03
  having an API was also sort of branding like apis sound like
• 15:06
  they're entirely Self Serve and that makes it mean like, oh
• 15:09
  yeah, they can just other teams can just come and be ml people
• 15:12
  sort of about, it's not quite that.
• 15:14
  But like over time we've iterated to, you know, it's not
• 15:17
  quite so surf but it's pretty damn close.
• 15:20
  And that I think like Katrina said, is really good.
• 15:22
  So I'm Point. What else is?
• 15:24
  Sounds like by having an API and for maybe I like this.
• 15:26
  You make the integration cheap for the other team so it sounds
• 15:30
  so three things. Tramping it on right there were
• 15:32
  so it easy integration allows users to allows your potential
• 15:37
  customers, your internal customers to try this with like
• 15:39
  minimal risk, your iterated, your iteration cycle.
• 15:44
  That's kind of more of an internal thing to your team,
• 15:46
  right? So your ability to iterate and
• 15:47
  try other ideas. And then you mentioned that
• 15:51
  like, it's pretty quick for you own start.
• 15:53
  I think you call these sources. So whenever it's a new use case,
• 15:55
  it's a you call it a source. Can you go in like how long does
• 15:59
  it take for a slag to like come up with a new source like the
• 16:02
  new team comes to you? How long, how long does it take
• 16:04
  you to get them going? Yeah, the can like talk a bit
• 16:08
  more about more about this. So usually like slack because if
• 16:12
  you think about the current recommendation use cases, in
• 16:15
  fact, like there are different entities but the most common
• 16:18
  one. It's really like channels.
• 16:20
  And then, you know, users, right?
• 16:23
  These are the two, most common entities, we were recommend, and
• 16:26
  then we have a bunch of features like Arrow actually build out
• 16:29
  this signal service that we can use.
• 16:32
  So, we have all the features about users and channels.
• 16:35
  And then for use cases, like, for a new source, which is a new
• 16:39
  use case. We usually just need to adjust
• 16:42
  like what features we are going to use how much weights like we
• 16:46
  also start with a hand to model. We can't dive into.
• 16:49
  Later and how, what's for the ways we assigned to each
• 16:51
  feature? And then also like the just, the
• 16:54
  fetching, like, we're the first stage is always fetching the
• 16:56
  candidates, and then maybe we are going to use different
• 16:59
  factors. So these are the factor and then
• 17:02
  the features. These are two most common
• 17:04
  things. We just need to think a bit more
• 17:07
  about each use case. All your feature engineering is
• 17:09
  already done you just and your Cana and I believe this is the
• 17:12
  scene sound like the slack can be the generation problem is
• 17:14
  relatively simple. Usually because you should just
• 17:17
  be able to hit like it. Operational database and get a
• 17:21
  list of users in this account. Yeah.
• 17:23
  It sort of varies like very common.
• 17:25
  One is like, hey, take all the channels.
• 17:28
  I'm a member of, you know, that's like a pretty simple one.
• 17:31
  But, you know, in some cases like you want to do Discovery
• 17:33
  and then it's like, okay we could start with every channel
• 17:36
  on the team but that's impractical for a lot of bigger
• 17:38
  teams. So that's like when we go back
• 17:40
  to embedding store to get nearest neighbors you know.
• 17:43
  So I think it's we have a bunch of plug-and-play ones available
• 17:46
  and there's a lot of very standard ones where you read.
• 17:49
  It's not like we do exactly the same one every time.
• 17:52
  So on the future engineering, like how many, like how many
• 17:55
  features are you? So I'm going to get a lot of
• 17:57
  reuse between of use cases, right?
• 17:59
  I guess if it's a youth, if a person or a channel features
• 18:02
  useful for one model is probably useful for the next.
• 18:06
  How many do you know roughly at your main model?
• 18:09
  Your main CTR models of thanks for some of these services.
• 18:11
  Like how many feet have any user and channel features?
• 18:13
  Do you all have available in our inner using for ranking?
• 18:17
  There's probably like And hundreds, thousands, hundreds of
• 18:20
  made outside there's hundreds but in practice.
• 18:24
  You know I think it's usually most models.
• 18:27
  Maybe need a couple dozen and we could probably get down to just
• 18:32
  a couple bells of across. All our bottles that are like
• 18:34
  really making a big impact. Does that sound right to you
• 18:37
  Katrina? Yeah I agree and a lot of them
• 18:40
  ran into being redundant. So something we want to put a
• 18:44
  bit of me that the hand toward starting a band to model and
• 18:47
  iterating from their comeback. Super interesting about your
• 18:50
  revenge, I love that. It makes a lot of sense and I
• 18:52
  think more team should adopt. But first, we talked a little
• 18:56
  bit about before the show about data privacy and how that's
• 18:59
  important the slack is that relevant to the feature
• 19:02
  engineering, and the feature serving component or is that
• 19:05
  more relevant on the back end and make model Training Academy?
• 19:08
  Where how do you all think about that?
• 19:10
  Yes, I mean it's important to both, you know, I think the
• 19:14
  biggest one is like you have to keep in mind that the police
• 19:17
  like slack. It's it's hard to imagine how
• 19:21
  the value we can create with ML. Would outweigh the value.
• 19:25
  We could destroy. If we leak customers, they
• 19:27
  dining teams like their most important data is in slack, like
• 19:31
  business Communications everything.
• 19:32
  So, like things we do, which would leak that data are not
• 19:36
  even worth considerate. So, kind of starting from the
• 19:39
  ground up, when we think about designing features, we have to
• 19:43
  think about white privacy. So the first of those is like
• 19:47
  anything where we're dealing with like, Summer text Data, the
• 19:49
  things you write into slack. We have to be like, really,
• 19:52
  really careful run. And the short answer is we
• 19:56
  usually avoid building from features from those inside sort
• 19:59
  of focus on like your interactions and slack will
• 20:03
  focus on like, hey, you know, what's the web of channels and
• 20:07
  users who work together more than like, this is a man
• 20:10
  toccatas. For instance, that the big wand
• 20:13
  up common factors and or feature creation And then in model
• 20:17
  training, you know, similarly we need to make sure any models, we
• 20:21
  train art somehow learning the actual content of your data in a
• 20:26
  way that might lead to another team.
• 20:29
  So like usually we serve have this level of separation between
• 20:32
  like the underlying like message data for instance and the models
• 20:36
  we train so that. Yeah we're not you know we don't
• 20:39
  have to risk a model. Additionally memorizing
• 20:41
  something you shared in slack and then regurgitating it
• 20:44
  elsewhere. Yeah like a real man.
• 20:46
  Example of that might be if you all had like an autocomplete
• 20:49
  function and that was able to memorize some other, two other
• 20:52
  companies jargon or internal messages.
• 20:57
  Yeah. That is that is that, is that
• 20:58
  the right way thinking about those?
• 20:59
  Yeah, that's exactly right. So, you know, like usually like
• 21:03
  the way we've done all complete suggestions so far is we kind of
• 21:07
  parse out what we think are likely suggestions from just
• 21:09
  your team's text and then maybe we have some like high level
• 21:13
  features about that like, oh, how many times did this phrase
• 21:15
  in Peter? How often have you said, things
• 21:18
  like that? That factors into, whether it
• 21:21
  appears as a suggestion for you, we don't directly use some light
• 21:25
  generative model, make those because out involve like
• 21:27
  training on directly on team text in a, that's really a
• 21:32
  dress. That's probably the most unique
• 21:34
  and might be the most unique thing about this slack
• 21:36
  recommendations problem. But I've heard because you all
• 21:38
  can't, you just don't, you don't feed raw tax or even your very,
• 21:41
  very sensitive about what goes into your directly in your model
• 21:44
  because you don't want the model like leaking.
• 21:46
  Between the organization's. Yeah, I mean I think probably a
• 21:50
  lot of other B2B software companies have that concern in
• 21:54
  some regard. You know.
• 21:55
  I don't know how plate like Google for instance with Gmail
• 21:58
  deals with it but it's at least should be the concern.
• 22:02
  Most people are dealing with the stuff extent but it's it's less
• 22:04
  of a big deal in terms of the business to Consumer world than
• 22:07
  in the business to business world.
• 22:09
  The thing that I think is fascinating going back to what
• 22:12
  you were saying on these points that you were talking about with
• 22:15
  like the autocomplete. Complete, you can't use these
• 22:18
  generative models. At least not what I think you
• 22:21
  can use. Or you can't use specially can't
• 22:25
  like fine-tune the generative models.
• 22:27
  Yeah, I remember. I also know about this fella but
• 22:29
  actually talk to us like my other teammates about actually
• 22:33
  training, you know, like fine-tuned like large language
• 22:36
  models and using our like slack data but Like, at the moment,
• 22:41
  it's like a no nose like a weekend.
• 22:44
  Like probably, like in the future.
• 22:45
  We can just use it, right? Like news, those models like
• 22:48
  even Just use it but still no. The whole to like actually fine.
• 22:53
  Tune, those models in Stacks. Yeah, I think we should put a
• 22:57
  bed in the back as I think is really interesting and probably
• 22:59
  applicable and probably under talked about a lot of space.
• 23:02
  I think people whenever they think recommendations that I
• 23:04
  could focus research, they like to focus on like the Twitter's
• 23:07
  with a Facebook and how they do it, but the B2B space is does
• 23:10
  have very, very different constraints.
• 23:11
  In fact, I've talked with some people in the medical community
• 23:14
  about that are doing separate. Recommendations is a funny word
• 23:17
  to use for it but it does boil down to a similar problem of
• 23:20
  stations and like the constraints that they hover over
• 23:22
  data privacy or crazy. So that feature Engineering in
• 23:26
  the world of data privacy and regulation is it looks it looks
• 23:30
  really can look really different.
• 23:31
  Did the other topic I want to talk about is the cold start
• 23:34
  problem and how slack gets around that?
• 23:36
  So Katrina - and say, you will do login, wait future engineer,
• 23:42
  so your future, your future. Your future platform is an
• 23:46
  online system. And then your computer features
• 23:49
  online. You log what features were
• 23:52
  computed and then you can use those later for training so that
• 23:55
  means that you need to start with something.
• 23:58
  That'll make a reasonably good prediction.
• 24:00
  So, that before you have any training data, it is to be late
• 24:04
  and that's, and that's your cold.
• 24:05
  That's that's lacks version, the cold start problem.
• 24:07
  So can you dig into a little bit of that and why you chose that
• 24:09
  direction and what you like about it?
• 24:11
  What you don't like about it? Yeah.
• 24:13
  I'm actually like like I'm also like quite Impressive.
• 24:18
  When I joined slack like this is how the team is dealing with
• 24:21
  co-starred problem. It's just like with a hand to
• 24:24
  the model and then I don't see it mashing like I read like a
• 24:28
  bunch of like blogs or like another engineer books.
• 24:32
  Another all lot of T measure that all like, they kind of mash
• 24:35
  it implicitly. I think we can start with a huge
• 24:39
  risk models and then just use that to gather the training
• 24:43
  data. I just never thought of this
• 24:45
  before I joined stack. And then after I Slack and see
• 24:48
  how people actually, like, how the team is actually doing this.
• 24:51
  It's actually pretty impressive. But when you think about it,
• 24:54
  it's pretty also like pretty intuitive, right?
• 24:57
  It, we can always start with a handheld model and then why is
• 25:02
  that we can gather training there on the other is that like
• 25:05
  people are just like when we explained that the model like
• 25:08
  the hand to the model 2pm's they are like how often it for
• 25:11
  example we are recommending users like the what future is
• 25:14
  always like how often the The user interact with each other
• 25:19
  and then we explain legs to pee. And then this, this this feature
• 25:23
  is important. So we assigned a like a larger
• 25:25
  number like a bigger weight to this feature.
• 25:28
  And when we explain this to p.m. they are just like they
• 25:31
  understand it. We don't have to explain like
• 25:33
  how many requests a largest regression Works.
• 25:36
  How extra boost works. You know they just understand
• 25:38
  the hand to model and then they can see the results.
• 25:41
  Also is already making sense and that we will explain.
• 25:44
  This is just our first iteration and we will iterate upon that.
• 25:48
  Once we have the training, day.so, the results is supposed
• 25:51
  to get better. Yeah.
• 25:53
  So mainly I think two points like why it's just we get
• 25:56
  Trinidad the other just like it's easy to explain 2pm's with
• 26:01
  this approach. So Jake you mentioned that this
• 26:05
  is underrated and I want to hear your heart take on that why
• 26:09
  people should be doing this more well so in this approach it you
• 26:12
  start with sort of the energy models nice because it also gets
• 26:16
  your PM's confidence of the Well version and then it gives you a
• 26:19
  reasonable Benchmark to beat what I saw.
• 26:21
  From the slack. Its stats was actually a male
• 26:23
  models, would come in and wipe the floor with a lot of his hand
• 26:26
  to models. And that gives you a lot of
• 26:28
  confidence that it's not. Some, you don't know what this
• 26:30
  model is doing. You know, it's better than the
• 26:32
  Sand. Dune thing that is very
• 26:33
  understandable and then the other thing, so for the record,
• 26:36
  YouTube does this. So I think an industry this
• 26:39
  broad approach is not as login weight in terms of feature
• 26:42
  engineering. So, you are doing all the
• 26:44
  feature engineering online? You are logging.
• 26:47
  It features that are computed and then you have to wait a week
• 26:50
  or two, depends on how much training data.
• 26:52
  You need for a very large company like slack that in a
• 26:56
  recommendations case. I get the ton of examples.
• 26:58
  This is right really feasible and it could be great because it
• 27:01
  allows you to just focus on the online problem and you can have,
• 27:05
  you never have to worry about problems, like featuring
• 27:07
  training serving, skew how to construct backfill queries Etc
• 27:11
  like a none of that you sidestep all of those problems.
• 27:15
  So, I think for a lot of recommendations teams that have
• 27:17
  relatively short windows for like for training data
• 27:20
  collection, like they only need 2 weeks or less of like training
• 27:24
  to identify changing data. This is a really good approach.
• 27:28
  Yeah. Do we agree?
• 27:29
  I'm as you stay out of it if like the you can get into like a
• 27:32
  future engineer. Hell, if you try to like
• 27:34
  reproduce features you calculate all I'd offline ahead of time.
• 27:37
  It's like, I mean there's probably times where it's really
• 27:40
  worth the time to go do that, but in most cases is just save
• 27:44
  you You so much time to just, why did it work off that?
• 27:47
  It's hot. So also like yeah, like a jqu
• 27:50
  point out. Like how it like that, like the
• 27:53
  hand to model, give us a nice Benchmark, exactly.
• 27:57
  Like, it gives me confidence about machine learning because
• 28:01
  like, people talking about it as a black box.
• 28:03
  And then, but like, seeing, like we're on experiments when we,
• 28:07
  you know, switch from hand-tuned to like other.
• 28:10
  Like, a male based models, we always get our like, more than
• 28:13
  10% Improvement. When we switch in two largest
• 28:17
  regression. Like it just gives me confidence
• 28:20
  that damn resemble is working. Yay.
• 28:22
  Like so do you want to keep around the hand to model as like
• 28:26
  a as a baseline for like as a kind of longitudal longitudinal
• 28:31
  metrics? We don't we keep it around not
• 28:34
  so much to do like you know some holdout group that keeps doing
• 28:37
  it but like if nothing else it's a great way of like if are
• 28:40
  serving service fails we can just fall back to this hand to
• 28:44
  Which takes you know? No special.
• 28:46
  I met Lantern serve. And so that's a way where we
• 28:48
  could feel over very cleanly. Not like a total degradation of
• 28:52
  the experience. Okay cool.
• 28:56
  Anything interesting. Katrina Katrina Aaron about
• 28:59
  that, we want to like kind of touch on in this, my kind of
• 29:02
  cold storage space. What it takes you to get up and
• 29:03
  running one of them? The thing I like about into
• 29:06
  models, besides the things we've mentioned already is I think
• 29:11
  there's a lot of PMS who can be a little put off by.
• 29:14
  I like making the switch from like some heuristic the designed
• 29:18
  to like, and she was like system, like, it feels like this
• 29:20
  big jump where they're giving up a ton of control.
• 29:24
  And I mean, I think the way to think about it is these are all,
• 29:26
  it's all a spectrum of light complexed, a heuristic that they
• 29:30
  can describe in three sentences or eristic.
• 29:32
  That takes three paragraphs, a hand to model a model and made
• 29:37
  by machine. I think like having that hand to
• 29:39
  model as a step, where they still have lost control but it's
• 29:42
  the same form as a machine learning model.
• 29:44
  L can often, he's he's teams into the idea that they're using
• 29:48
  machine learning and relying on something that's a little bit
• 29:51
  more out of their control. My, I feel like my product
• 29:54
  manager, is always a YouTube, the most rewarding thing for
• 29:56
  them, as they come up with a feature idea, for something that
• 29:59
  just intuitively made sense and we went and implemented it and
• 30:02
  stuck it in our model and then all the sudden saw a big
• 30:05
  performance job. I don't know about, like, it
• 30:07
  kind of fits into that, like, product managers love to have
• 30:09
  the intuitive features and understanding, how they impact
• 30:12
  user experience like they go, they go crazy.
• 30:14
  That all right everyone. I just want to jump in real fast
• 30:18
  because I have a huge announcement.
• 30:20
  We will be having a part two of our large language models in
• 30:24
  production. The first conference was such a
• 30:26
  success that we had to follow it up because there were so many
• 30:29
  incredible speakers that reached out after the first one.
• 30:32
  I couldn't help myself. So it will be on June 15th, and
• 30:36
  it is going to be online. So anyone from anywhere can join
• 30:41
  as long as you can stay awake, if you're on the far East, were
• 30:43
  doing it in West. Coast times.
• 30:45
  So you might have to take a few extra coffees that day, but if
• 30:50
  you thought the first one was good, then boy, are you in for a
• 30:54
  treat? We have some confirm speakers
• 30:56
  already. I'll just go through a list of
• 30:58
  them mact a from data bricks, you know, the founder of data,
• 31:01
  bricks and Spark and all that good stuff.
• 31:03
  We recently just had him on the Pod.
• 31:05
  He's coming back to Keynote. You've got chip, Yuan, who is an
• 31:10
  amazing person and incredible, follow on social.
• 31:14
  Sharing so many insights and has been dropping knowledge bombs
• 31:18
  when it comes to large language models recently, you've also got
• 31:21
  Scott McKay from M.A I you've got the Deep speed team is going
• 31:28
  to be there from Microsoft that is pretty epic and Sahar more
• 31:32
  from stripe. Another stripe alumni is
• 31:35
  Emmanuel Emison. He's formally of stripe.
• 31:38
  Now he's working at one of these large language model companies
• 31:41
  that you've definitely heard of, but I don't think I can spill
• 31:43
  the beans on yet. Yet and we've got Lance Martin
• 31:47
  in as from number station. George Matthews from Insight
• 31:51
  Partners. This is just a glimpse at the
• 31:54
  whole list is even more incredible than this.
• 31:57
  So go sign up. You've got all the details in
• 32:01
  the link in the description. Huge shout out to the sponsors
• 32:04
  redis for showing us some love and making sure that this event
• 32:09
  happens. Our guila any scale continual
• 32:12
  chroma. Thank you all and You can expect
• 32:16
  all kinds of bad musical improvisation, the brakes, like
• 32:19
  I normally do it. And who knows?
• 32:22
  I may make that I hallucinate more than chat GP t-shirt
• 32:25
  available for sale again. Check it all out.
• 32:29
  Links in the description, will see you soon.
• 32:32
  Okay, so both of you are data scientist, turn ml, engineer's.
• 32:36
  That's like other separate roles for data scientists and ml
• 32:38
  engineers. And do you all see a lot of a
• 32:41
  lot of data scientist making that, that transition within the
• 32:44
  company? So we do have data scientist but
• 32:47
  the data scientist that slack, I mean maybe very occasionally,
• 32:50
  they'll train some sort of regression model offline or
• 32:52
  something like that. But in general like our machine
• 32:56
  learning engineer is on our teen are vertically.
• 32:59
  We do the whole thing so like Katrina might one day be working
• 33:03
  on our model serving service. I don't know their daylight the
• 33:07
  you know, training and actual model that will go into it and
• 33:09
  on the other day might be like program enough to integration
• 33:13
  where that model where are Monterey Jazz called from serve
• 33:17
  our main monolith. So, it's like here, it's like
• 33:21
  MLA cheers. Do the full thing.
• 33:23
  Also, you know, data Engineering Building feature pipelines,
• 33:27
  training data, pipelines, all that.
• 33:30
  So, like rml engineer flaw vertically integrated.
• 33:35
  So, this is crazy to me, you telling me that unicorns do
• 33:38
  exist, basically, because how are you able to, for the both of
• 33:44
  you that We are in this position, like, keep up to date
• 33:47
  with all of the surface area that you're supposed to be held
• 33:52
  accountable for. I mean, I think well, Katrina
• 33:55
  probably has a lot of thoughts about this.
• 33:56
  I would say, you know, from the team's perspective, you know,
• 33:58
  people do I think, within that mold specialized to varying
• 34:02
  degrees. There's some people will realize
• 34:03
  metalwork time on the in front of some people spend more time
• 34:06
  on bottle training. I think it's realistically.
• 34:08
  I don't think you can stay up to speed on every single part of it
• 34:11
  as an engineer. Exactly like like, I guess, Me
• 34:14
  because like I was a data scientist so I usually do more
• 34:18
  like about exploring the models and then just doing something
• 34:22
  like data scientist stuff. And I've always wanted to just
• 34:25
  dive into like infrastructure a bit more but our team has like
• 34:30
  someone who's super great at like setting up, you know,
• 34:33
  starting a micro service or like, you know, just dealing
• 34:36
  with GPU, give us structure. It's like super hard to just
• 34:40
  know, you know, like know everything.
• 34:42
  So so nominally everyone Your team is a demo engineer, but it
• 34:46
  sounds like maybe people are coming out of from both
• 34:48
  directions. So you have some software
• 34:50
  Engineers that want to get more into the data science space and
• 34:54
  the amount of vacations. And then you have ml Engineers
• 34:56
  who want to run infrastructure and get into production.
• 34:59
  And then so, within a team, all integers, but there is still
• 35:03
  some diversity of skill set. Is that, is that kind of how you
• 35:05
  all are? Brazilian get up and start the
• 35:08
  call. So from a career development
• 35:09
  perspective, this offers everyone on the team, something
• 35:11
  out of valuable to. Yeah, everyone has Correction to
• 35:15
  grow it. Yeah, I got forever for the
• 35:17
  record. This is actually, this is this
• 35:19
  memory years, my experience, this is what we did it YouTube
• 35:22
  to I came in from the software engineering side.
• 35:24
  So I was stuck with all these days, you know, former data
• 35:28
  scientist that were really good at machine learning and I would
• 35:30
  help them with like, logs in front over time, I would learn
• 35:32
  learn more about the data science behind it, and then, you
• 35:36
  know, I was kind of help them in the reverse Direction.
• 35:38
  I think this is a fascinating way to put it together because
• 35:41
  you have the ability to pay pair with people on your team that
• 35:46
  have the strengths where you're lacking.
• 35:48
  I definitely learned a lot from like the other team member who
• 35:51
  is doing a lot of, like the infrastructure stuff and then
• 35:56
  like Ops stuff. Yeah, he's the obstacle in our
• 36:00
  team, definitely learned a lot. I'm wondering about because
• 36:03
  Katrina, you and I were talking about this before, we jumped on
• 36:06
  the call and it was the idea of the future of ML and how you saw
• 36:12
  it. And I have a Strong thoughts.
• 36:15
  But you mentioned in your eyes where is it going?
• 36:19
  And maybe you can reiterate what you were thinking and I know
• 36:22
  Jake's probably got some in Aaron's probably got some
• 36:24
  thoughts on it too. Yeah.
• 36:26
  Honestly like I'm not sure. So the question is, what you?
• 36:30
  I guess I'm interviewing you to like because I just like you I
• 36:34
  just teed it up for me to be understood.
• 36:36
  You like how I did that? I didn't give it to you.
• 36:38
  To ask me the question. Yeah, yeah.
• 36:43
  Like so Like, exact, I see like we have so like everything.
• 36:48
  I'm not sure if we can talk about this, but we do have like
• 36:51
  a worked with a third, like a, well that, well, the called
• 36:56
  third party, you know, you know, the kind of like to serve our
• 37:00
  models. We do use external service
• 37:02
  before, but there are like comes with a lot of troubles, like
• 37:05
  they have to adapt their infrastructure to slacks
• 37:08
  infrastructure and then like just like, especially if I Karen
• 37:12
  deal with that a lot, especially when we met Edge with
• 37:14
  Salesforce. There's just tons of I guess
• 37:18
  policies to deal with was using external services and then, and
• 37:21
  then we ended up just like replacing it to our internal.
• 37:24
  Like, we, we build a, like, an internal serving infrastructure,
• 37:28
  and then we end up just replacing the, the external
• 37:31
  service. And then not to mention like
• 37:33
  that. Now, I'm procedure to just, you
• 37:36
  know, stop the contract. It's like like half a year on
• 37:39
  wine year kind of process to even do that.
• 37:42
  So, and then I see we are like with lack, they already like
• 37:49
  this kind of scale of tech company.
• 37:51
  They already have like a well-maintained warehouse and a
• 37:54
  lot of infrastructure to support us.
• 37:56
  So we kind of like we kind of inclined to just use our
• 38:01
  internal infrastructure as much as possible.
• 38:04
  And yeah, that's what I see in slack but I think for startups
• 38:09
  that might be a different story but I just cannot, I still
• 38:12
  cannot imagine like a A day like we're like this kind of B2B
• 38:17
  company like was large skill. They can use like, external
• 38:22
  services like without any concern, like, you know, like I
• 38:26
  just still cannot imagine that they come like in next, I don't
• 38:30
  know, five years old, I don't know.
• 38:32
  Yeah. So Aaron's thinking about what
• 38:36
  part of that one is going to get cut out by PR and what part is
• 38:39
  actually. Yeah.
• 38:40
  Okay. For us to have in your yeah.
• 38:45
  And it was definitely like there's a, there's a whole host
• 38:50
  of issues with, like bringing in, you know, outside companies
• 38:54
  to provide like ml infrastructure of things like
• 38:56
  that. You know, it's kind of like the
• 38:59
  question I'm always asking is like, is it more work to
• 39:01
  integrate with someone or is it more work to build it ourselves
• 39:04
  and like with model serving, we kind of concluded at the end of
• 39:08
  the day it was more work to integrate that was to build it
• 39:11
  ourselves which you know, he kind of As a manager, it
• 39:16
  frustrates me. Because like, for my point of
• 39:17
  view, I always want people working on my call, like
• 39:20
  differentiated engineering effort.
• 39:23
  I don't want to like generically if there's like, a generic way
• 39:26
  to sailor. Serve, like or song ml Ops
• 39:30
  that's just available in works for everyone.
• 39:32
  You know, that's not what I wanted years workout.
• 39:34
  I want them to work on something that's special and unique to
• 39:37
  slack, its context. And it turns out in for is still
• 39:41
  getting the infra to work is still special.
• 39:42
  You need to slack to the point where that's where worth us
• 39:44
  doing. But, you know, like, I want us
• 39:48
  work out specialized stuff and, you know, for the moment it's
• 39:50
  sort of like like obviously bespoke models for our use cases
• 39:53
  as one but you know, I can even act like the bill been a eyes
• 39:57
  and gbts of the world suggest like, hey maybe even bespoke
• 40:00
  models isn't differentiated work and future at some point.
• 40:04
  I don't think that's ever really gonna fully happen but it is
• 40:07
  kind of, it's an interesting question.
• 40:10
  What will forward what I think Bezos gave that famous talk
• 40:14
  about how you want to Do only what makes the beer taste better
• 40:19
  and then not worry about the rest right?
• 40:21
  When he was talking about why everybody should be using AWS.
• 40:24
  And so the thing that I think about when this question is like
• 40:30
  if you're at a certain maturity level like slack is at, I see
• 40:35
  that at this point in time. It feels like it is a big
• 40:40
  question mark But then if you're at a smaller scale and you want
• 40:44
  to go from like that, Zero to one.
• 40:47
  I definitely think there is some huge plus, to being able to just
• 40:55
  grab the tools off the shelf and run as fast as you can with it.
• 41:00
  And maybe that's using a full platform, or maybe that's using
• 41:05
  just the best of breed and then incorporating in some open
• 41:08
  source stuff. And what I really am interested
• 41:12
  to see, like you were talking about Katrina in the next five
• 41:15
  years. Is if we look at like the
• 41:18
  kubernetes space, it has just blown up more and more each
• 41:22
  year. There's more tools and more
• 41:25
  projects that are coming onto the kubernetes ecosystem.
• 41:29
  I think what we're seeing right now are at least from what I can
• 41:34
  gather from the smart people that I've talked to none of my
• 41:37
  own abilities is just that it's were really still very much at
• 41:43
  this maturity level to where Where people aren't comfortable
• 41:48
  with the fully outsourced version or the data still has
• 41:53
  like we talked about earlier. There's still a lot of big
• 41:56
  question marks around data and security and if we're going to
• 41:59
  be passing this data off to a third party, then how much of a
• 42:03
  headache is that to onboard that supplier or tooling company, and
• 42:08
  is it worth it for me to do that?
• 42:10
  And then, if you think about okay, well, if we're at that
• 42:14
  maturity level, And then you also look at everyone's got
• 42:18
  their own stack and there's not necessarily the most mature
• 42:22
  Integrations between the tools yet, because it's not super
• 42:27
  clear that like 90% of people are using XYZ stack, then you
• 42:33
  get this headache and that's kind of my view on it.
• 42:36
  I just feel like if we give it more time, we'll start to see
• 42:40
  the maturity will happen. And like, I feel like we're
• 42:43
  seeing right now with the data engineering space, right?
• 42:45
  Like data engineering. I feel like was very much at
• 42:48
  this level that we're at right now, maybe 34 years ago and it
• 42:53
  feels like people are picking up in their wants and needs for
• 42:58
  using data engineering tools. That's kind of my view on it but
• 43:02
  we saved the best for last. What are you got to save others?
• 43:05
  Jake. Yeah, full disclosure I work at
• 43:08
  it in our laps, God work at tecton which is a feature
• 43:11
  platform, you know, ml W company.
• 43:13
  So my take on this is definitely Third eye.
• 43:16
  That I think the AWS analogy is very apt.
• 43:19
  We're, you know, in the lab space is very, it's still pretty
• 43:22
  immature. So this is this would be like
• 43:24
  going on a BOS when a to be else was brand-new and didn't have
• 43:27
  all of the bells and whistles that has today.
• 43:29
  So, you know, you're debating between going on Prem, or on
• 43:34
  this very immature Cloud product and it probably still to do all
• 43:38
  the things you want to do. If you're sophisticated company,
• 43:40
  probably still makes sense to build on-prem, right?
• 43:43
  To do it yourself. But as the cloud provider, in
• 43:48
  this case, is ml Ops gets better and the tools give more mature.
• 43:52
  You're going to start having been harder and harder trade-off
• 43:54
  to say hey I just want to focus on my business out.
• 43:56
  I just want to focus on my business problems like our
• 43:58
  differentiators. I don't want to have to
• 44:00
  reinvent. This thing that's been done
• 44:02
  multiple times. I also want to be able to hire
• 44:05
  and people that have expertise and this framework now my does
• 44:10
  bespoke slack version or you know whatever or you know,
• 44:13
  on-prem version of whatever this is But I think right now for a
• 44:17
  lot of companies, if you have a good strong engineering team,
• 44:19
  and you have the resources that like you already mentioned, you
• 44:23
  already had all these things that you were just layering on
• 44:25
  that sand layer on top of, not all those companies need ml, you
• 44:29
  know to buy go out and buy ml Ops top to bottom in a few
• 44:33
  years. I might look really different.
• 44:34
  Once, these products all really mature and they can they're easy
• 44:37
  to integrate and do all the things that you want and they
• 44:40
  can allow you to To build everything there, just focus on
• 44:44
  your problems. Now, for some companies that
• 44:48
  don't want to have to go do that reintegration or ready to
• 44:50
  integrate and pay that cost and now and don't have to rebuild
• 44:53
  things later, and are thinking they're small.
• 44:55
  Now they're going to what they want to grow into this, into a
• 44:57
  grow into that investment. Those are early adopters on ML
• 45:01
  Ops, I think it starts making a lot of sense for or even even
• 45:04
  today. The thing I think about a lot is
• 45:06
  like, unless we're at a very weird company, like ml is like a
• 45:10
  very small portion of the total, like engineering at a company.
• 45:14
  The difficulty we found is we had to have have our tools be
• 45:17
  able to layer on top of all our other ng-switch were built
• 45:21
  without much interesting thought towards what made sense to ml,
• 45:24
  and that's what it felt like, the primary the primary
• 45:28
  difficulty. So if we can serve, solve that
• 45:29
  problem where like these things layer in cleanly, I think, you
• 45:34
  know, like gummies, we'd love to spend less time on ML off some
• 45:38
  more time, on solving business problems with machine learning.
• 45:41
  But that's that's the sort of The river that needs to be
• 45:44
  crossed from my point of view and it's him like one, I always
• 45:48
  think about that. It seems to pop up.
• 45:50
  A lot is like experimentation from from our point of view.
• 45:54
  Like running experiment is measuring the business impact of
• 45:58
  a model change through like a generic experimentation platform
• 46:03
  for our company as a whole mostly used by, not about people
• 46:05
  but then I think I should preface here, I am in closer to
• 46:09
  one of the Company's trying to build experiment platform but
• 46:12
  then I always see Like ml Ops companies come in and like offer
• 46:16
  their own brand of experimentation management that
• 46:19
  sort of separate and that's one of these areas where it feels
• 46:22
  like again, it's difficult to layer and cleanly on top of the
• 46:26
  more generic engineering and product problems across the
• 46:30
  company. Yeah, they really don't think
• 46:32
  about it. Experimentation platform in
• 46:33
  particular that it's such a deep integration into your system.
• 46:36
  Yeah. Like it has to touch production
• 46:38
  and data metrics and you know the whole Stacks.
• 46:41
  I think that's I think that's it.
• 46:42
  That's all. A Google did bespoke and it was
• 46:45
  still a great. And I think every company I
• 46:48
  think almost I almost companies that I'd like to kind of roll
• 46:51
  their own version of experimentation platform.
• 46:53
  Yeah. So we can me.
• 46:54
  I'm advising Ethel, I've actually been kind of impressed
• 46:56
  with how well, Dave, and it people to manage that
• 46:59
  integration. So I think that gives me some
• 47:01
  hope for like, dental office space to, but it's it's still
• 47:04
  it's a ton of touch points and it's a ton of places where you
• 47:06
  need to integrate people yet. It's it's not an easy problem,
• 47:09
  it may be to tie it, tie it back to the beginning of the Talk,
• 47:13
  you know what? You talked about, how your
• 47:15
  internal service, the strong simple API that, people just get
• 47:19
  the right thing from. I think a lot of ml W
• 47:21
  Integrations will boil down to people that can figure out where
• 47:24
  those API boundaries are and how to make a strong simple easy to
• 47:29
  integrate with API that you can have an excellent ml,
• 47:33
  engineering team behind that actually, like levels things up
• 47:35
  and that'll look different depending on what part of the
• 47:37
  stack, your end. But I think that's kind of the
• 47:38
  key secret sauce. I think there's a lot of hype
• 47:41
  around generative. Yeah.
• 47:43
  I am like you know I don't think it's gonna totally replace like
• 47:45
  something more. Bespoke feature-based things.
• 47:48
  I think that's like fundamentally when open-air.
• 47:50
  I got, right? It's like here's this API where
• 47:52
  you just sending you tent, you send us text.
• 47:54
  We send you back text and you can put the complexity and text
• 47:56
  and we'll deal with it and that's like Almost more than
• 48:00
  just how cool the models are just that they distilled it down
• 48:03
  to that is like really really appealing dude.
• 48:07
  So good. Well I'm going to wrap it up
• 48:10
  here and last question for you because I tried to ask Katrina
• 48:12
  while you were putting the little one to bed.
• 48:15
  Can my boss? See my direct messages on Slack?
• 48:28
  Privacy is a big deal like, you know, it's not only like do we
• 48:31
  make sure we need to make sure like our models are not somehow
• 48:34
  leaking, you know, obviously we don't want build features, we're
• 48:37
  just people can view other people's private messages.
• 48:40
  We want to make sure our models even within a team aren't
• 48:42
  leaking that data and we don't even want to, like, we want to
• 48:45
  design our futures, where we don't even hint at the existence
• 48:48
  of a DM between you and another person through a model like we
• 48:51
  actually, we're testing out something internally and we
• 48:55
  scrapped it because we realized like, This.
• 48:58
  It could indicate that like two people were messaging each other
• 49:01
  without getting directly at the content.
• 49:03
  And we said blood that was a promise.
• 49:05
  So we're very, very serious about like, sort of the Privacy
• 49:09
  balance of the product and slack, both as a company, as a
• 49:12
  whole and then kind of respecting those.
• 49:14
  Well as we build machine learning stuff, like in
• 49:17
  practice, the usual like that, you'll see this.
• 49:19
  We build a lot of features exclusively from public Channel.
• 49:23
  Messages are public Channel, interactions to make sure we're
• 49:26
  not leaking that stuff. Good.
• 49:28
  So keep us posted when slack comes out with the Community
• 49:32
  Edition. Everybody is waiting for that.
• 49:34
  I am sure. And this has been an awesome
• 49:37
  conversation. Thank you both for coming on
• 49:39
  here and talking to us about everything behind the scenes of
• 49:42
  the tool that I spend way too much of my life on.
• 49:46
  Let's go talking to y'all, that's great.
• 49:48
  Talking to y'all.