Latest / Agentic Conversations (formally mlops.community) / MLOps.community #4 - Building an ML platform @SurveyMonkey with Shubhi Jain

Transcript

MLOps.community #4 - Building an ML platform @SurveyMonkey with Shubhi Jain · Agentic Conversations (formally mlops.community)

• 0:06
  Hey everybody, thanks for tuning in to another episode of our ml
• 0:10
  Ops Community. Meetups I wanted to let you know
• 0:13
  about a few things. Before we start off this
• 0:14
  episode, we have a slack community that you can join.
• 0:19
  You just need to go to ml Ops dot community, and you'll find
• 0:22
  the link to that there. We talked about everything and
• 0:26
  anything, ml Ops related. And the last thing is that we
• 0:30
  are going to be opening up office hours.
• 0:35
  This is a place where anyone can come in and talk about what
• 0:39
  they're doing around ml Ops and get help from the community if
• 0:43
  anyone needs it. So, jump in the slack, you'll
• 0:46
  see all the updates about our office hours.
• 0:58
  Alright, here we go with another episode of our ml Ops community.
• 1:04
  Podcast for this installment, we're joined by Shuba Jane a
• 1:09
  machine-learning engineer at Survey Monkey.
• 1:12
  And as you'll hear in the show, he was an integral part of the
• 1:17
  team that helped build out the machine learning platform at
• 1:21
  SurveyMonkey. And if you like the way that he
• 1:24
  breaks things down and explains all of these Concepts, I invite
• 1:29
  you to check out some of the content that he has on YouTube
• 1:33
  and I'll leave some links to To his videos in the show notes,
• 1:38
  just another thing. Before we get started, we had
• 1:41
  some great questions from the audience on this last meet up.
• 1:46
  And if you'd like to be in the live recording so that you can
• 1:51
  ask questions and hear it firsthand.
• 1:54
  I'll leave a link to where you can register for that in the
• 1:59
  show notes. We'd love to have you.
• 2:02
  The last thing is that if If you want to be a guest you feel like
• 2:08
  you got something to say or you know, someone that would be a
• 2:11
  great guest, please, reach out to us and we'll make something
• 2:16
  happen. Alright, without further Ado,
• 2:20
  here is shoe BJ. And thank you for joining us
• 2:26
  today. We've got a very special guest
• 2:31
  on. This is our fourth time, our
• 2:33
  fourth go around, and I think it just keeps getting better.
• 2:38
  We were joined by Shuba Jane and I want to give you a big thanks
• 2:42
  for coming and talking with us. We originally chatted I think it
• 2:48
  was at the end of last year and we talked about what you've been
• 2:52
  doing a survey monkey with the ml plan.
• 2:55
  Form there and how how you set things up and I'm so when we
• 3:00
  started this ml Ops Community, I thought it would be great to
• 3:04
  chat with you again and get your opinion on just ml Ops and ml
• 3:10
  Platforms in general. Because I knew you would have
• 3:13
  good information to share and knowledge to pass on to the
• 3:16
  community. So, let's go ahead and jump
• 3:21
  right into it. We kind of or I've been Starting
• 3:26
  from the beginning and asking people are guests, how they got
• 3:32
  into technology where that came from.
• 3:36
  So if you could just tell us who beat a few things about how you
• 3:41
  started in into tech and and where it's gone from there, it
• 3:46
  sounds great. Thanks so much to meet Russ for
• 3:47
  having me on here today. And yeah, it's a great question.
• 3:51
  Start us off. So, I grew up in the bay.
• 3:55
  They area. So super fortunate to be
• 3:57
  surrounded by Tech my family, like my parents lunch, my
• 4:02
  cousin's much, my friends kind of grew up in this in this area
• 4:06
  as well, where we started working in Tech very early on
• 4:10
  and, you know, you're really surrounded by it in the Bay
• 4:12
  Area. You see it.
• 4:13
  Everywhere you drive by down some highways, you take any you?
• 4:20
  You're talking to folks and Tech is everywhere.
• 4:23
  And so how I really got started. It on this journey for myself is
• 4:27
  and back in school I started attending some hackathons so you
• 4:32
  know it was a really great space for me to learn new
• 4:35
  technologies. Learn what this building culture
• 4:37
  is like what things you can really build using software and
• 4:42
  from there I didn't and I never really stopped.
• 4:46
  I went and studied computer science in college.
• 4:49
  Did my undergrad and my bachelor's at Caltech?
• 4:53
  Then from there I ended up joining Very small start-up
• 4:56
  working as an analytics engineer, where I did a lot of
• 4:59
  what you would call classical data, engineering stuff.
• 5:02
  But given that it was a start-up.
• 5:04
  And, you know, we were just 20 people, I did it up doing quite
• 5:07
  a bit of machine learning and machine learning infrastructure
• 5:10
  as well. From there, I, my current role
• 5:14
  where I work at SurveyMonkey as a machine-learning engineer, but
• 5:18
  along the way, I've done a bunch of passion projects inside
• 5:21
  projects, some Consulting work where I had the opportunity to
• 5:27
  help build out, various aspects of ml platforms or help build
• 5:32
  that. For variety of customers and
• 5:37
  clients both within the financial space as well as in
• 5:40
  the tech space. So kind of going and looking at
• 5:44
  various organizations that are different stages within their
• 5:47
  journey of tech as well as data science and machine learning, I
• 5:51
  think has been really eye-opening and I've had the
• 5:54
  opportunity to understand what really I can do to make to
• 6:00
  support that organization to do well for whatever they're really
• 6:03
  looking to do. At that time and it's been
• 6:07
  exciting and I'm really looking forward to seeing what comes
• 6:09
  next. No, this is a rapidly changing
• 6:11
  field. You learn some new things every
• 6:13
  single day, every single week. So it's been great completely
• 6:17
  along those lines. When you joined SurveyMonkey,
• 6:21
  had they already had something in place for an ml platform.
• 6:27
  Or was that starting from scratch?
• 6:29
  Where did that come from? That's a great question.
• 6:31
  So, I joined about three months after After we decided as a team
• 6:37
  that, oh, yeah, we need to have like a platform in place.
• 6:40
  So, at that time, what we had was essentially microservices
• 6:46
  for each use case. So, whenever we wanted to
• 6:50
  develop a new machine learning use case, Some data scientists
• 6:55
  or ml engineer would start developing a model.
• 6:57
  And once we were ready with that model, we went and said, okay,
• 7:01
  let's let's productionize it. Like if we need something real
• 7:04
  time, let's let's wrap it around in a, in a micro service.
• 7:08
  Let's talk Horizon and let's deploy it and SurveyMonkey.
• 7:11
  Given that we're pretty old software company.
• 7:13
  We've been around since 99. We have pretty standard
• 7:18
  processes for setting up. Microservices, that's not very
• 7:21
  hard, but Around April, of 2018 is when we realized, hey, you
• 7:27
  know, there are some unique requirements for our machine
• 7:31
  learning microservices that other teams don't really have to
• 7:34
  deal with and that's where we decided.
• 7:36
  Hey, we need to have a platform for this.
• 7:38
  We have some special requirements that allow us to
• 7:44
  look at this in a greater scope and realize there are a lot of
• 7:47
  different aspects like retraining models feedback
• 7:50
  pipelines where data is coming. From various things like that,
• 7:55
  that made us sit back and say, oh yeah, we need something else.
• 7:58
  Besides the process that we have set up right now.
• 8:01
  And so when you were looking at what you needed, how did you go
• 8:06
  through this build verse by process?
• 8:08
  Because I know a lot of companies are looking at.
• 8:11
  Should we get a third project party or should we jump on it
• 8:15
  and start to build it ourselves? Yeah, always the first question
• 8:18
  you want to ask. I think we also asked ourselves
• 8:21
  the same question too. And I think what really helped
• 8:25
  us figure out whether we should build or buy is we turn to our
• 8:31
  team? We say like, hey, you know, if
• 8:33
  we had the ideal platform where every problem that we're facing
• 8:36
  today and maybe for the foreseeable future has taken
• 8:39
  care of what would that look like?
• 8:40
  What would the requirements for that?
• 8:42
  Look like. And so, we had, I remember
• 8:45
  something like a Multi-day brainstorming session where
• 8:49
  we're all just sitting down and writing like, okay, in the ideal
• 8:52
  world. I break this down into we, we
• 8:56
  broke it down into three groups or three pillars and I actually
• 9:00
  realized that these three pillars aligns really closely.
• 9:03
  What with what Phil had talked about last week on this, I meet
• 9:07
  up as well which is we talked about the data side, then we
• 9:10
  talked about the services and the serving side and last we
• 9:14
  talked about the management side.
• 9:16
  Although those three pillars are not the exact same words that
• 9:21
  say fill, it shows. I think he is where data
• 9:23
  software and governance. I think they match up pretty
• 9:28
  closely though. And for each of those, we ended
• 9:31
  up defining. Okay?
• 9:32
  What are our requirements? You know, for data, we need some
• 9:35
  way for our data, scientists to easily find data within our
• 9:39
  organization. We need some way for us to
• 9:42
  reduce training serving, skew. We need real time live updates
• 9:46
  for a E-learning features then on the serving side, we needed
• 9:50
  ways to have feedback pipelines, be built out either real time
• 9:54
  or, you know, in a slower batch. Moving way we needed to record
• 9:58
  every single prediction that took place or inference across
• 10:01
  all of our models, as well as be able to handle that within our
• 10:05
  a/b testing and and experimental platform.
• 10:09
  And then on the management side, being able to manage all
• 10:12
  versions of new models that are coming out all model, And then
• 10:17
  all sorts of data that's coming through this.
• 10:19
  And so, as we started writing everything down, we said, okay,
• 10:23
  you know, there is not one platform that can do all of this
• 10:25
  for us and then it becomes a question.
• 10:31
  We stitching things well given the way that development moves
• 10:36
  in this kind of space how it's so rapid, you know, we might
• 10:39
  choose that we're going to go with this platform today for the
• 10:41
  data side but then tomorrow we might see, it's not compatible
• 10:44
  with our upgrades on the serving size Side or on the management
• 10:47
  side. And so, that's where we sat back
• 10:50
  and said, okay, I think it makes sense to build here.
• 10:53
  And we had the expertise in-house, which was really great
• 10:57
  for us to convert some of our general software Engineers into
• 11:01
  machine learning engineers and data scientists that could help
• 11:06
  us build this out from within. And, you know, we did some
• 11:09
  hiring of course, but I think largely, we did have great
• 11:13
  experience within the organization and And I'm
• 11:16
  wondering how big was the team that embarked on this journey,
• 11:22
  which big question? I think.
• 11:23
  So we have we had like four or five data scientists and 526
• 11:32
  machine learning engineers and just to be really specific the
• 11:36
  difference between these two groups of people are data
• 11:40
  scientists and our team are primarily responsible for
• 11:43
  developing machine learning models whereas machine Earning
• 11:46
  Engineers are primarily responsible for developing
• 11:49
  machine, learning services. So all the deployment side of
• 11:52
  things, sometimes they'll be some overlap because people have
• 11:55
  some expertise as well and just because as projects grow and
• 11:59
  expand that could be something there.
• 12:01
  But largely speaking, that's the breakdown.
• 12:06
  And actually since then we haven't grown too much, I want
• 12:11
  to say we probably added one more one or two more data
• 12:14
  scientists or one or two more machine learning engineers.
• 12:16
  But our output capacity has definitely grown to the point
• 12:20
  where, you know, we had a handful of ml use cases back in
• 12:24
  2018 and that's exploded to the point where, you know, during
• 12:29
  the 30s and 40s where it would take us on the order of say, six
• 12:35
  to eight months to have one use case from inception or like
• 12:38
  ideation to out there and production.
• 12:41
  But now it's on the order of like Salem where we can get
• 12:44
  something out there and get the ball rolling.
• 12:46
  Which is really great. Yeah.
• 12:48
  So what you've what the work you put in paid off sounds like oh
• 12:53
  yeah yeah and there's definitely room for improvement, you know.
• 12:55
  It's an iterative process but we're really happy to see how
• 12:59
  we've been able to really accelerate or velocity when it
• 13:02
  comes to all these kinds of projects completely.
• 13:05
  So last question before we jump into the actual Frameworks and
• 13:10
  whatever, all of the infrastructure and how you're
• 13:12
  doing it as far as upkeep goes. Yeah.
• 13:17
  How, how does that play out and how is it?
• 13:21
  The machine learning Engineers that are doing that and how much
• 13:26
  of your time or the machine learning Engineers time is spent
• 13:30
  on just making sure everything runs properly.
• 13:33
  The yeah, I think that's definitely really important to
• 13:36
  know, especially for, like, teams that may not have like
• 13:38
  dedicated platform oriented teams or they don't want to be
• 13:42
  want to have folks that are dedicated necessarily to manage
• 13:45
  the platform. I want to say that in terms of
• 13:48
  upkeep, there's not a lot because we want our platform to
• 13:53
  work for us, you know, it's it's really the whole game is
• 13:57
  automate automate. Automate I think though that
• 14:01
  there is some time that's dedicated to adding new features
• 14:05
  or new capabilities to the platform, and that's really an
• 14:07
  iterative process. And, you know, each, we like to
• 14:10
  break apart our development Cycles into quarters and then
• 14:14
  Sprints, so, whatever. Our Early goals are like, okay,
• 14:17
  we're going to break this up into two weeks friends, but for
• 14:21
  each quarter, we generally have like a couple of goals like how
• 14:23
  are we going to increase the capabilities of our platform
• 14:26
  today? Does that mean that we're going
• 14:29
  to add in a completely new component?
• 14:31
  Does that mean just to just a new, some new upgrades here,
• 14:34
  there are small and so upkeep time can be relative based off
• 14:39
  of what our goals are for that quarter.
• 14:41
  And what we're looking to do, if we have in general, like, fewer
• 14:46
  machine, Earning use cases that are new to the organization or
• 14:49
  that we have made promises to say product managers or other
• 14:52
  product teams that are, you know, we have fewer of those,
• 14:55
  then we can maybe dedicate some more time to upgrades in our
• 14:58
  platform. And really we're always we're
• 15:01
  looking Less on the maintenance side and more on the improved
• 15:04
  capability side. So I think on maintenance, you
• 15:08
  know, I want to say it hovers between five to 10% of our
• 15:11
  engineers and data scientists time which is around the time
• 15:16
  that We roughly would spend on any service for that matter.
• 15:19
  Just, you know, if anything goes down, all right.
• 15:21
  Yeah, let's take a look. What's up with that or anything
• 15:24
  along those lines but generally we look to automate things.
• 15:28
  So now let's get into the Nitty Gritty.
• 15:31
  Yeah. Could you give us an overview of
• 15:34
  what you're working with and how it works?
• 15:37
  Definitely, I can definitely want to build an overview here.
• 15:42
  So, I like to break this apart into, you know, those three
• 15:48
  areas. You know, there's that data side
• 15:51
  that Services side and then the management side and under data
• 15:56
  side, what we've developed here. So just to give you a bit of
• 16:00
  background like back in 2018, will we would be doing is we had
• 16:05
  a data warehouse that was managed by By our data
• 16:07
  engineering team, you know, they would be in charge of developing
• 16:10
  the ETL that would do like a prod copy of all of our
• 16:15
  databases into our data warehouse and something that's
• 16:21
  unique about SurveyMonkey is that we're a hybrid platform.
• 16:24
  So some of our stuff is on premise and some of our stuff is
• 16:27
  in the cloud and we're actually in the midst of a transition of
• 16:31
  moving, everything to the cloud and putting that on.
• 16:34
  It'll be us for us. However, given, you know, we're
• 16:38
  pretty old company, we went with on premise, you know, many years
• 16:42
  ago, and we stuck by that for quite some time.
• 16:46
  And so all of our data that was used for machine learning model,
• 16:51
  training and development would sit in our data warehouses and
• 16:54
  our data scientists would call through these.
• 16:56
  Hundreds of tables, be really familiar with which data sits
• 17:01
  where what kind of quality checks they have to run what
• 17:03
  kind of validation in order for them to get.
• 17:06
  Raw training set and that would take a really long time, you
• 17:10
  know, sometimes in the order of months and then on the serving
• 17:13
  side, our data would be looking like okay.
• 17:17
  We know what tables this came from in training but sometimes
• 17:23
  we have already set up some services that serve us this
• 17:26
  table, the data from these tables already.
• 17:29
  So really, we should be using these service fees microservices
• 17:32
  because our engineering teams have worked hard for us to do.
• 17:36
  Deliver that sort of data in a nice clean format at certain
• 17:40
  latency is LA's and all those kinds of things.
• 17:44
  So we would go through a strenuous process of mapping,
• 17:47
  okay? You got so-and-so feature from
• 17:50
  this data warehouse table, so really this maps to this service
• 17:55
  and it's this feature but really got altered here somewhere.
• 17:58
  So we need to do some engineering on top of it.
• 18:01
  And then we have some really lengthy data validation process
• 18:05
  to make sure. Okay yeah.
• 18:06
  That's the Temperature, we're not, we're not having some
• 18:08
  training servings here that takes place and to do that for
• 18:11
  every single model for every single feature was a pain and we
• 18:15
  did it wrong so many times before we got it, right?
• 18:19
  So, we knew this wasn't the right way to go and our
• 18:22
  solution. To this was developing a machine
• 18:24
  learning feature store where we wanted to have, essentially, a
• 18:30
  One-Stop shop for all machine learning data.
• 18:33
  And we realized that we were essentially Starting to use
• 18:37
  cases here one is this kind of bulk training use case where you
• 18:41
  wanted data to be made available for all of our users, that use
• 18:45
  the Survey Monkey platform, but stolen an easy enough way that
• 18:51
  maps to the use case of our machine learning projects.
• 18:55
  So I gave a talk about actually, our machine learning feature
• 18:59
  store at O'Reilly strata data conference last month, and I can
• 19:06
  show you Good quick slide maybe so that it makes a bit more
• 19:09
  sense of yeah. Great.
• 19:11
  What what what this kind of looks like.
• 19:15
  That's something that I remember when we talked last you were
• 19:18
  telling me. Yeah, we're almost done with the
• 19:19
  feature store. Cool.
• 19:21
  So the way that we thought we took, we thought about this ml
• 19:23
  feature store was actually also do things like roast.
• 19:28
  We took a good hard. Look at our data patterns that
• 19:33
  existed within our organization and and we found a relationship
• 19:38
  that, you know, we had several one too many or one to one
• 19:42
  relationships that existed between our data and entities
• 19:46
  between them and then we, we also realized that our data
• 19:51
  scientists wanted point in time, window views into our data as
• 19:57
  well. So let me break down what that
• 19:59
  means. So if if I'm say like Airbnb I
• 20:03
  might be developing machine learning models, that are
• 20:06
  centered It around our guests or our hosts.
• 20:15
  Have Alders relevant feature data about them but then they
• 20:18
  might have some more in-depth data about, say, a guest can
• 20:22
  have many stays a host can have many homes.
• 20:25
  And these one-to-many relationships are then
• 20:27
  aggregated in view with respect to a certain entity.
• 20:31
  And so, at the end of the day, what data scientists really want
• 20:34
  is a super flat table, that contains all the information
• 20:38
  about that entity and all aggregated.
• 20:41
  Else other information about their sub entities or higher
• 20:44
  level entities and there at the end of the day, they also want
• 20:49
  this information with respect to certain time so how did you know
• 20:55
  that data? Looks like for a certain user on
• 20:58
  January 1st 2016 I should be able to know that so that I can
• 21:02
  develop say like a conversion model or churn model about this.
• 21:07
  And so we thought okay we really need to have a feature story.
• 21:11
  That serves this use case for our training purposes and
• 21:15
  ultimately, the structure that we were able to come up with
• 21:18
  that was super simple for our data, scientists to use and work
• 21:22
  with and easy for us to develop enough was creating a flat table
• 21:26
  for each entity where we have each entity ID broken down by
• 21:30
  each day. So each day's data and then all
• 21:33
  the features specific entity. I know this is like a really
• 21:37
  wide flat table, you know, it has hundreds of features and
• 21:41
  This made it super easy for data scientists to gets get up and
• 21:45
  running with any use case that they needed to work on.
• 21:49
  And so this machine learning feature store sits within we use
• 21:55
  AWS for all, for all the. So we leveraged EMR to do a lot
• 22:01
  of our data processing. We leverage AWS of theta, as our
• 22:06
  final tables, for entity tables, for our data scientists, to send
• 22:10
  a request to And it's backed by S3.
• 22:14
  So every, all the data is actually just sitting in that's
• 22:16
  three for us and it's it's really exciting to to see that
• 22:24
  work in in real life. And then, from there we had, so
• 22:31
  that's a really our batch use case, but then I talked about
• 22:33
  like are serving issues, right? So at the same time, we have a
• 22:38
  streaming architecture that reads off of our, uh,
• 22:41
  Application logs, from all of our services and creates a
• 22:45
  one-to-one mapping between the features that are in our feature
• 22:47
  story here and what spills off of our application logs.
• 22:52
  In real time, these values are then updated in a similar.
• 22:55
  Exact same structure feature store that can be read in, you
• 23:00
  know, at a single out of single layer, a single line.
• 23:05
  So, if I want to know, What is the data for a specific entity
• 23:10
  on a specific date? I hit my real-time feature store
• 23:13
  for that and my services hit that real-time feature store and
• 23:17
  the latency on that. As you know, just a few
• 23:19
  milliseconds. And so really, I'm able to get
• 23:23
  single line reads, super fast for my real-time feed restore,
• 23:27
  but I'm able to do my bulk reads against my trip.
• 23:30
  My bulk feature store as a data scientist, whenever I want again
• 23:35
  at a pretty fast speed. So I know that was a lot of
• 23:38
  information so I can maybe take a quick pause for you to ask
• 23:42
  some questions there, that was great.
• 23:45
  And I think this may be a good moment for us to open it up to
• 23:52
  everyone else that's in the in the chat right now.
• 23:56
  If anyone has a question that they want to ask juvie, feel
• 23:59
  free to unmute yourself and fire away.
• 24:03
  I have a question. Can you hear me?
• 24:06
  Yeah. Oh, hi.
• 24:15
  That today would have different
• 24:32
  considerations because they had so much.
• 24:35
  Of our platform that we might have decided to just buy.
• 24:54
  I agree with you but probably smaller aspects not like a huge
• 24:59
  data aspect or a huge serving aspect.
• 25:04
  Man out of the box and reason for that is even now you know,
• 25:09
  two years later. I we have the ability to look
• 25:12
  back and say, oh, you know, should we have done anything
• 25:14
  different and each time we do our retrospective and ask
• 25:18
  ourselves this question, what we end up realizing is, it doesn't
• 25:23
  end up being two things. It doesn't end up being super
• 25:26
  compatible with the rest of our, our, our architecture that we've
• 25:31
  developed it ends up being I'm like you buy, you need to buy
• 25:34
  into the whole thing and then you're limited by the
• 25:36
  capabilities of what you buy into, at least in the ml space.
• 25:41
  Since this is moving so quickly. What ends up, being released as
• 25:44
  a buy option takes some time. And then the second thing is,
• 25:49
  more of a Survey Monkey limitation is whenever we're
• 25:51
  working with anything. We often work with our
• 25:54
  infrastructure teams to make sure, okay, are we good from a
• 25:57
  security perspective? Are we good from any support or
• 26:01
  expertise perspective? Do we need to learn And anything
• 26:03
  substantially new from an infrastructure team.
• 26:06
  If anything were to go down because if anything were to go
• 26:09
  down where the first line of defense given were consumer
• 26:13
  platform. We have hundreds of thousands of
• 26:15
  people using our platform every day.
• 26:17
  It's very important for us. That things don't go down, you
• 26:20
  know, up time really matters, but the second line of defense
• 26:24
  is our infrastructure team, you know, they get paged next and if
• 26:28
  they don't have expertise, or they aren't able to learn the
• 26:30
  expertise to learn learn The tools effectively to be able to
• 26:34
  support us. That doesn't help us either
• 26:36
  because we don't have good backup in those scenarios.
• 26:38
  And so really, the first thing is realizing that we would be
• 26:42
  locked into certain things. And then the second thing is
• 26:44
  that more practical of we didn't have the ability to have a good
• 26:49
  support from like a devops perspective in terms of, like,
• 26:54
  if things were to go down or in terms of just uptime and such
• 26:57
  thank you. I'm seeing a few questions in
• 27:00
  the chat. One from d.c.
• 27:03
  Asking did you did you handle versioning bit temporarily.
• 27:09
  And if so, how did you express consistency requirements for end
• 27:14
  users? Yeah, great question DC.
• 27:18
  I so in terms of burgeoning so our ml Peter store, essentially
• 27:24
  we version by day and we realize that's not necessarily the most
• 27:29
  granular, versioning, that we that is possible technically We
• 27:33
  speaking, we can do it by every single change.
• 27:35
  That's taken place in a across for each user, you know, going
• 27:39
  back in time, but we really looked at what was really needed
• 27:44
  by our use cases and add a versioning, like versioning for
• 27:48
  every single day was more than good enough for our use cases
• 27:53
  and teams. We didn't have any issues with
• 27:56
  anything like that. So we're like, okay, you know,
• 27:58
  we know something greater is possible, but what do we really
• 28:01
  need? We can get away with Jay based
• 28:05
  versioning. And so we did and then in terms
• 28:07
  of when we need to version our training data for our machine
• 28:10
  learning models. So our machine learning models,
• 28:15
  they actually would have their own.
• 28:18
  They would read from our feature store to develop their training
• 28:21
  sets. And in each training set would
• 28:23
  be version based off of whatever data site would be version by
• 28:26
  time stamp based off of what the data scientist wants to use.
• 28:29
  So models are rarely developed by Saying like oh these are the
• 28:34
  features in the future store in. This is the date that I want
• 28:36
  things for and that's it. We actually do have a separate
• 28:39
  process of like. Okay.
• 28:40
  Yeah this was the exact data that was used to train version 3
• 28:45
  of this model and it's sitting here in S3.
• 28:48
  It's, you know, time locked. If we need to go retrain version
• 28:52
  3 of the model because if some corruption the model artifacts
• 28:55
  we can actually go do that and we're pretty happy with how that
• 28:58
  works. And so that allows us to have
• 29:00
  that consistency. With our model artifacts and
• 29:06
  model versions themselves. Let me know.
• 29:09
  Do you see if that doesn't answer your question fully?
• 29:14
  Yeah I think that that's that's awesome.
• 29:18
  Yeah then we just have one more question in the chat of read it
• 29:23
  out loud for everyone that's just listening in.
• 29:26
  Does the feature store also store updated results of other
• 29:31
  ml models or just plain features from the dwh.
• 29:37
  Great question. So when we first started, it was
• 29:39
  just plain features because that was you know, our Phase 1.
• 29:42
  We just wanted to get things, get the ball rolling.
• 29:44
  Now it does have updated results of other ml models and that's
• 29:49
  really useful because as we know the inputs of some models are
• 29:53
  the outputs of other models. And so all of our use cases
• 30:01
  really function. And the way this really works is
• 30:05
  we have this gets into more of the feedback pipeline side of
• 30:08
  things, we have feedback pipelines that are set up to Put
• 30:13
  results of models into others basis within our data warehouse
• 30:18
  or, you know, spit out things into our log stream that we can
• 30:23
  then put directly into our data warehouse.
• 30:26
  And then it gets right into our MLP twist or so different ways
• 30:30
  of making that happen. And the reason for both of these
• 30:32
  things is, sometimes we need real-time feedback and sometimes
• 30:36
  we're okay with slower-moving feedback where the actual true
• 30:40
  results or True Values. Of inferences of Mo models.
• 30:44
  We end up getting that to, those True Values.
• 30:46
  Same days later. Like, does it user convert?
• 30:49
  Well, we might end up finding that answer, say 30 days later,
• 30:52
  but is it user? A Spam user 0.
• 30:55
  We can actually find that out like, super fast.
• 30:58
  So, based off of when we can find out, the True Values of
• 31:01
  things, that's when values are updated and made available that,
• 31:05
  then our ml teacher store, Cool, very cool.
• 31:09
  And since you're giving us a few use cases, can you go into a bit
• 31:13
  more depth on other use cases that you've you've had and sure
• 31:19
  should we talk maybe more on just to like get to keep the
• 31:23
  ball rolling, let's talk more about use cases and then maybe
• 31:26
  the lady to the serving side of things.
• 31:28
  Yeah, yeah, so we developed essentially a machine learning
• 31:37
  This is Gateway that acts as our.
• 31:41
  One place for all machine learning services.
• 31:43
  So essentially you want to call any mo model you hit our machine
• 31:47
  learning service Gateway. We let us know which model
• 31:49
  you're looking to hit and if there's any A/B Testing, you're
• 31:52
  looking to do specific to model versions and such, you can pass
• 31:56
  that in and then we pass him. The the data needed for that
• 32:01
  model to make make an inference. And what we want to do is make
• 32:06
  it super easy for our partner services to integrate with us.
• 32:10
  So When I say passing the data needed to make an inference, I
• 32:14
  don't mean all of the raw features that are needed to make
• 32:17
  a model and friends. I might mean do they just give
• 32:20
  us the user ID or the survey ID? And then from there we know for
• 32:25
  each model what features are needed.
• 32:27
  We go, hit our real-time feature store get those features and
• 32:31
  then we have essentially model project pods which are just
• 32:38
  clusters of know. Roads that are all running the
• 32:43
  same Docker image or various different versions of Docker
• 32:47
  images. If we're doing proper versioning
• 32:48
  and a/b testing, and then they, they have received all the Grab
• 32:59
  that in be we go hit those nodes, get her in friends, back
• 33:03
  and then return the result to our to our whoever cult has
• 33:07
  called our service. So just repeat that last part.
• 33:10
  I think you may have broken up a little bit.
• 33:12
  Oh, sure thing. So basically, we have like a
• 33:15
  high level Gateway. We receive all requests.
• 33:18
  We go. Hit a real-time feature store
• 33:19
  and grab our features. Then based off of, whatever the
• 33:23
  API primer inputs are, which model we're looking to hit and
• 33:26
  which version we go ahead. We have a we have an nginx
• 33:30
  routing layer in between that helps us route face up to these
• 33:33
  programs, which nodes we should be hitting to get to make our
• 33:37
  inference. We go make our inference, and it
• 33:39
  travels back up our path to then we return our results to our
• 33:43
  users. And so there's essentially a
• 33:45
  load balancer that sits right at our Gateway layer and then we
• 33:49
  have a router that sits at right below or gateway to route which
• 33:53
  nodes we should be hitting to make our inference and we're
• 33:58
  able To do this. Now, both synchronously and
• 34:01
  asynchronously because some of our models are super heavy and
• 34:05
  say they might take on the order of seconds but our Our partners
• 34:15
  want to be able to receive a results or at least know that
• 34:19
  they made the right call, it made the right request pretty
• 34:22
  quickly. And so, what ends up taking
• 34:25
  place? There is a synchronous evil
• 34:27
  return. Like, hey, you know, we got your
• 34:28
  request, here's your ID Pingas, whenever you need us to display
• 34:32
  results of the user and then we can go ahead and do that.
• 34:35
  And so were able to overcome our her.
• 34:39
  And the reason why we did this as we need, we had some hurdles
• 34:42
  on the infrastructure. Aside, we couldn't support some
• 34:46
  models that needed gpus to run very quickly and this is mainly
• 34:51
  due to like Sarah infrastructure.
• 34:57
  To eat some meat. Like, we couldn't get those GPU
• 34:59
  enabled, clusters or nodes. And so, as a result, we're
• 35:03
  running some things on CPUs there, bit slower.
• 35:05
  But we're able to still meet our requirements for delivering
• 35:09
  really quick results to our to our users and a use case that
• 35:15
  kind of demonstrates synchronous versus asynchronous.
• 35:18
  Is we have a machine learning model feature called
• 35:24
  surveillance a genius. So this is a feature that's its
• 35:29
  when your sits on the create side of our Survey Monkey
• 35:32
  platform. So when you're creating a
• 35:33
  survey, you know, Oh, good. We, when we're creating a survey
• 35:44
  on our platform, we want to help our users create optimal
• 35:47
  surveys. So we provide them with an
• 35:50
  estimated completion rate, estimated, completion, time, and
• 35:53
  some recommendations for how they can better their survey so
• 35:57
  that it's more likely to be completed.
• 35:58
  And so these are backed by a couple of different machine
• 36:00
  learning models, but we want the results appear real time as the
• 36:05
  user is creating their survey. So that they can make the
• 36:08
  changes to their survey almost immediately.
• 36:11
  And so this is the case that's asynchronous in Prince
• 36:15
  architecture since synchronous inference call.
• 36:18
  But then there's a use case of asynchronous and Friends.
• 36:20
  Call say, when we're doing some heavy analysis, Upon Our surveys
• 36:25
  and want to return some outputs on say the sentiments of every
• 36:29
  response forever, for the survey that you're looking at, you
• 36:33
  know, you can get millions of responses on your survey getting
• 36:36
  the sentiment. For everything can be really
• 36:38
  quite a while so in those cases where like okay, yeah, let's use
• 36:42
  our asynchronous architecture. It'll take us a couple of
• 36:44
  minutes for us to get you, you know, the overall distribution
• 36:48
  of sentiments and such like, such things like that.
• 36:51
  And so let's use our asynchronous architecture and
• 36:55
  you know, our users are okay with that and we've been able to
• 37:00
  see if they deliver on that for them as well in an efficient
• 37:03
  manner, without bogging down or our infrastructure too much.
• 37:08
  That's so cool. Thanks for breaking that down.
• 37:11
  So Karthik in the chat was asking a question that I was
• 37:17
  actually also going to ask about the inference architecture and
• 37:21
  does it run on kubernetes? Great question.
• 37:25
  So it did not for us because we whatever we're looking to add in
• 37:32
  any new tools or technology to our stack at They like to go
• 37:38
  through, we have, we have to go through such processes.
• 37:41
  So while we would like to be using kubernetes down the line,
• 37:46
  it just hasn't gone through that process of approval by security
• 37:50
  approval, by infra making sure we have all the knowledge that
• 37:53
  we need to support it. Essentially, we given Republic
• 37:58
  company and we have, we're consumer-facing platform.
• 38:04
  We need to make sure we have all our lines of Defense there.
• 38:07
  All are checkboxes. So what we end up using instead
• 38:11
  is ansible, which we have been using for many years at sorry
• 38:16
  monkey and it works well for inference architecture, that's
• 38:20
  great and Ryan's got a great chat or a great question in the
• 38:24
  chat, I should say, he's asking, if you can go more into the tech
• 38:28
  of the async architecture. And if it's closely related to
• 38:33
  the synchronous architecture, or is it a totally different?
• 38:37
  Yeah, it has layers of similarity Ryan.
• 38:40
  And so I would say the aspect of making a request in our Gateway,
• 38:47
  a routing that all Remains the Same.
• 38:49
  I think the additional layer of complexity comes in essentially
• 38:53
  what we use to inform our clients like, hey, your your
• 38:57
  request has a response that's ready or for sometimes even pass
• 39:02
  in a request to us. Is we have we use a message bus.
• 39:07
  That's our second one key wide message bus, you created a topic
• 39:11
  for ourselves and that allows us to do passing around of requests
• 39:18
  and responses in an asynchronous manner effectively.
• 39:22
  And that's essentially our main differentiator, I would say,
• 39:25
  beginner async and synchronous architecture.
• 39:27
  And there are some things also getting more into the
• 39:31
  nitty-gritty about like priority of requests and prioritization
• 39:36
  cues And different things like that.
• 39:39
  Yeah. But that's super specific.
• 39:41
  And for, for each use case, we end up realizing we need a new
• 39:48
  capability. How can we architect?
• 39:50
  Are asynchronous architecture to work for this use case.
• 39:57
  Already. And we like to be driven by use
• 40:00
  cases rather than entering the whole thing that we can think
• 40:03
  of. And then try to make it work
• 40:04
  because otherwise we get bogged down by all these other things.
• 40:09
  So, DC had another question in here.
• 40:12
  It's do you run in? Do you run any human tests?
• 40:16
  IE. How long does it take a human to
• 40:18
  add a feature? Get it flowing through to the
• 40:23
  end point. Yeah.
• 40:26
  Ali always whenever we add in a new capability or whenever we're
• 40:30
  just like, hey you know, let's see how things improved.
• 40:33
  So like, at least once a quarter will be looking to say like
• 40:36
  okay, you know, when you use case, you new machine learning
• 40:39
  model, that we want to help develop, you know, go through
• 40:42
  the whole Flow by herself. How does it look how long did it
• 40:45
  take you? Let's record It All.
• 40:47
  We Have It All documented. We know how long things took we
• 40:49
  know where the pain points exist and you know that serves as
• 40:53
  information for what we can do better next time and what
• 40:56
  Movements, we can continue to make to our platform for it
• 41:00
  access. A learning experience for, for
• 41:02
  people, that are onboarding onto our platform as well.
• 41:04
  And so, both things and these human tests, we always have to
• 41:09
  run to make sure that our platform is working to serve our
• 41:13
  data scientists and the engineers.
• 41:15
  Well and you know we're really expanding to other teams now
• 41:19
  helping to democratize machine learning within the greater
• 41:22
  SurveyMonkey engineering org. Think that could be a nice segue
• 41:25
  into To talking about how you're doing monitoring with the
• 41:31
  machine learning models. And I know that there's a few
• 41:34
  things to keep in mind when you're monitoring machine
• 41:37
  learning models, you have to just make sure that everything's
• 41:39
  working properly, the system is up.
• 41:42
  But then you also want to make sure that your models are doing
• 41:46
  what they said they put and they they were going to do.
• 41:49
  And so are using a third-party just to monitor the system like
• 41:54
  data dog or in Spanish or Whatever it is.
• 41:57
  And then is that all in-house also or are you also did you
• 42:03
  build that out and how did you go about that?
• 42:07
  Yeah, great question. So yeah, when we look at
• 42:09
  monitoring we look at essentially three things, I
• 42:13
  would say one is the service monitoring go just like you're
• 42:16
  monitoring any service. Second thing is input data
• 42:21
  monitoring so monitoring like what kind of data is coming into
• 42:24
  your machine learning models? And then Monitor.
• 42:27
  And then third thing is, like predictions / like real true
• 42:32
  value monitoring. And so on the service side, we
• 42:37
  do, we do use some, some third-party tool to help us out.
• 42:43
  And this is used across our entire organization, we use New
• 42:48
  Relic, I was just, sorry, I was thinking to make sure that I'm
• 42:50
  allowed to say and we are so yeah how we use, we use New
• 42:53
  Relic which allows us to You and all the monitoring on in terms
• 43:00
  of like service side of things and it's really similar to data
• 43:04
  dog. We had used a dog in the past
• 43:06
  and it works quite the same and we and yes.
• 43:12
  Dan Sullivan from New Relic is going to be on here in about a
• 43:16
  month so I'll let him know. Yeah, definitely.
• 43:21
  Yeah, we're big supporters of New Relic and it's worked really
• 43:24
  well for us and we're really happy about Out that.
• 43:27
  And then on the input data monitor instead of things, what
• 43:33
  were essentially doing? Is we look through our log
• 43:36
  stream. We log all of our input data
• 43:39
  going into our machine learning models, so that can be logged to
• 43:46
  our our, our application logs. So that would be coming in
• 43:49
  through our log stream then. So, you know, we have some, we
• 43:52
  have different log streams for different use cases, so Kafka.
• 43:56
  For our message by certain other things like that.
• 43:59
  And then the second place where we might lock things, is we
• 44:02
  every prediction is also logged into a prediction DB, so it sits
• 44:07
  in a database. We can query against it.
• 44:09
  Real time allows us to do any sorts of like analysis, check on
• 44:13
  data drift, very things like that.
• 44:15
  And so, because it gets logged to our log stream, we can
• 44:21
  develop some dashboards that allow us to look into David
• 44:23
  drift. What is our data distribution?
• 44:25
  Look. I'm like for all of our features
• 44:27
  for the specific model, what is our prediction distribution
• 44:31
  looking like for this specific model for each version of our
• 44:35
  model that's sitting out there. How is our our our A/B test
• 44:38
  results looking like we can go also map.
• 44:42
  These two True Values later on because our machine learning
• 44:46
  service, Tain back and point. So, if we can get the feedback
• 44:54
  real time, we do a scarcer, our clients.
• 44:56
  Hey, hitter hitter feedback in point.
• 44:58
  Let us know what the real value was, and that allows us to the
• 45:00
  map. Oh, yeah, we made this
• 45:01
  inference. This is the input data.
• 45:03
  This is the true value. This is how well our model is
• 45:05
  performing, and that gives us a real time view into our model
• 45:09
  performance. In some use cases, we don't have
• 45:11
  that and that's where because we're logging everything to this
• 45:15
  table. Whenever we do get that
• 45:17
  feedback, we asked our partners. So, log that feedback into a
• 45:22
  table of sorts? And so, we can set up some
• 45:25
  feedback pipelines that essentially map these inferences
• 45:30
  to the True Values. We can then get some results of
• 45:35
  how well, our models performing usually, it's like daily or
• 45:38
  weekly views. Let me just post it into a slack
• 45:41
  channel. So whoever this model pertains
• 45:44
  to say, our product managers, our clients us, the ones that
• 45:47
  worked on the models themselves are aware of Oh yeah.
• 45:51
  You know this past week we saw we saw some down taking
• 45:54
  performance, probably we're good.
• 45:56
  Where we have to go look into this and see like what's going
• 45:58
  on, or then we can maybe kick off an auto retrain process to
• 46:03
  just Auto retrain, our model, if our performance dipped below a
• 46:06
  certain certain threshold and so we have all these systems
• 46:10
  working in tandem with each other that allow us to monitor
• 46:14
  all three aspects, effectively, very cool.
• 46:19
  And what lead? I see your question I just Ask
• 46:21
  one more about that auto retrain, because last week, when
• 46:25
  Phil was on, he made the case for this idea of good practice
• 46:29
  being just to Auto retrain, but he wasn't sure it was so worth
• 46:34
  it in some to most use cases or what he had seen with certain
• 46:42
  companies. It wasn't needed.
• 46:44
  Do you have Auto retraining going on?
• 46:46
  Or is that something that is not necessary?
• 46:49
  How does that work with? You guys.
• 46:51
  So I agree with Phil as well. I think there are some use cases
• 46:57
  where you do not need it. And I think that's the same
• 46:59
  thing for us. There are some use cases where
• 47:01
  we do enable it and some used to use cases where we do not
• 47:04
  however, we what whether we enable auto retainer, not is one
• 47:11
  thing but still being able to Auto retrain is I think another
• 47:15
  thing, all of our models if we need to retrain our And fresh
• 47:21
  data we can we can just click a button and it'll go retrain.
• 47:25
  And the reason for that is because if anything were to get
• 47:28
  corrupted, if anything, where we realize ago, we lost our model
• 47:33
  artifact, or whole like repository of our model
• 47:37
  artifacts went down. We know, we can go and do it
• 47:40
  again and we're confident in that and that is just no, our
• 47:44
  backup processes. We want to make sure everything
• 47:46
  is good there. And I think often, When we think
• 47:51
  like, oh yeah, everything's in the cloud, it's safe.
• 47:53
  But, you know S3, go down to it'll, be less goes down to and
• 47:57
  we like to just have our backups there for those keys.
• 48:01
  For the, you know, that less than 1% of the time when things
• 48:04
  go wrong. And it also allows us to have a
• 48:09
  level of accountability of making sure.
• 48:11
  Like, oh, we did what we did during training, and we can
• 48:14
  reproduce it. And then reproducibility aspect,
• 48:17
  make sure that if someone is leaving the company, And someone
• 48:21
  who is joining, it makes sense to them.
• 48:23
  What went on here and they can get the exact same model
• 48:26
  artifact once again, so can be used in any case.
• 48:30
  So, a couple of different smaller reasons there that Force
• 48:33
  us to have Auto, we train available still there, but it's
• 48:36
  still only enabled for certain use cases, where we see data
• 48:39
  drift occurring, or where we see that we are, our performance is
• 48:43
  varying over time. Yeah.
• 48:46
  Yeah, completely it makes sense. So while lead had a question
• 48:50
  here in the The chat. He was wondering he says I would
• 48:54
  like to understand more the role of ansible in your
• 48:58
  infrastructure. Could you please elaborate on
• 49:01
  how it's used and how your on-prem infrastructure looks
• 49:05
  like? Yeah, that's a great question.
• 49:08
  Why lead? So ansible is used for
• 49:10
  orchestration process. It's used for setting up like we
• 49:15
  have you know, hundreds of microservices.
• 49:18
  SurveyMonkey each micro. This has specific Docker images,
• 49:23
  they want specific setup for node flavors, like what kind of
• 49:27
  instance you're dealing with different things like that to
• 49:30
  ansible orchestrates, the setup for each specific node relative
• 49:36
  to which service it's going to fit under and that's essentially
• 49:43
  what we use it for there isn't much.
• 49:45
  There isn't any real of And that's a limitation of our under
• 49:52
  a structure as we moved to AWS. And some of our services are
• 49:56
  over there. In some of these cases are over
• 49:58
  there. That's where we can take the
• 49:59
  benefit of what AWS gives us. There are on-prem infra I can't
• 50:05
  talk too much more about what I said than what I've already
• 50:08
  said. Unfortunately, yeah, I apologize
• 50:13
  for that but it's, it's It will be rely upon ansible to setup
• 50:20
  our nodes, and influences, and sit in their perfect.
• 50:24
  And I was Sandra was asking, do you have use cases that involve
• 50:29
  personal identifiable information, if you do, how do
• 50:34
  you handle that? Yeah, great question Alessandro.
• 50:38
  We do a lot of we are HIPAA compliant because many hospitals
• 50:44
  or Health Care Professionals use SurveyMonkey.
• 50:47
  We also integrate with a lot of Enterprises that provide us Pi
• 50:53
  info and so a pii data. And so we need to be able to
• 50:58
  work with that and so the way that works is our real-time
• 51:01
  feature store is set up in a secure manner that only specific
• 51:06
  services and access it and it does contain the pi data.
• 51:09
  Since sometimes, we do use Pi data during inference time and
• 51:14
  that's known to our users. It's an And thing where users
• 51:18
  have to opt into it however for training.
• 51:21
  No pii data is available, we can to make that accessible for
• 51:26
  training ever. So you cannot train on any PI
• 51:30
  data. It can be anonymized in certain
• 51:32
  use cases depending upon these case and we have an entire
• 51:36
  governance process set up that I'm not responsible for.
• 51:39
  But we make sure to abide by it at each point to make sure that
• 51:44
  we are working there if Actively and I think, I think the the pi
• 51:53
  data that we use in the real time cases are also like we do
• 51:57
  have the ability to be able to go back in and delete it if
• 52:00
  needed. And the various other processes
• 52:02
  that are instead of been set up for to abide by it, like the,
• 52:08
  the data governance practices that we need to fulfill as as a
• 52:12
  server monkey org. Yeah, that makes sense.
• 52:15
  Gah. Has a question here about the
• 52:18
  line between the machine learning engineer and data
• 52:21
  scientists, he's wondering creating managing model
• 52:25
  artifacts using repository for this model.
• 52:28
  Versioning retraining, is this the role of a ml engineer or a
• 52:32
  data scientist? How to ml engineers and data
• 52:36
  scientists work together? And when does the ml engineer
• 52:40
  takeover or are they continuously working together,
• 52:45
  because Data scientists aren't often keen on software
• 52:49
  development or versioning. Yeah I know it's a great
• 52:53
  question. I think each organization Works
• 52:55
  differently so I can talk specifically from SurveyMonkey
• 52:58
  has few years ago and used to be like oh yeah data scientists do
• 53:03
  all the model development to give us model artifact and then
• 53:07
  they're like okay here you go and off that did not work very
• 53:12
  well for us in the long run and since then we work together
• 53:16
  intend on. From the very beginning and that
• 53:19
  allows our data scientists to pick up on some software
• 53:22
  development that allows their MO engineer's to understand the
• 53:25
  model at a very, very low level. So that makes sure that we can
• 53:31
  know from the get-go like oh actually you know we don't have
• 53:34
  this data available in our feature store real-time like it
• 53:39
  only gets populated once a day. So like are you okay with that?
• 53:42
  Is that fine and our data scientists know this.
• 53:46
  They requirements or limitations upfront or like oh yeah you
• 53:50
  know, we can't support a 15 layer neural net we can only
• 53:55
  really support 10 layer ones at the moment.
• 53:57
  So you know whatever you're looking to do, we can't do that.
• 54:00
  And because those conversations are happening earlier on our
• 54:03
  data, scientists know what the restrictions are, we know what
• 54:06
  the restrictions are. And from there on, we can really
• 54:10
  make sure that we don't have any like blockers that we didn't.
• 54:16
  See kind of appearing and as a result to be specific data,
• 54:21
  scientists still are probably the leads on the model
• 54:24
  development and them all Engineers are more.
• 54:27
  So, the leaves on the production izing aspects, but there's
• 54:31
  definitely some crossover and more and more crossover edit
• 54:34
  that, but it happens over time. And I would say that over time,
• 54:37
  I'd say, I've gone way more depth on the data science side
• 54:41
  of stuff, enter data scientists have become way more Adept also
• 54:46
  on On the software engineering side.
• 54:50
  There was no it's going on and if anything were to happen to
• 54:54
  our services, I'm sure they'd be able to, you know, be able to
• 54:57
  chip in and make sure things are running smoothly.
• 55:00
  Great. All right, so we got two minutes
• 55:03
  left. I want to ask you one final
• 55:06
  question. But before I do, what's the best
• 55:09
  way to connect with you? If anyone wants to continue the
• 55:12
  conversation, Yeah, I would say linkedin's probably the easiest
• 55:19
  thing. I look at most frequently, I do
• 55:21
  have a Twitter and I'll but I don't, I feel like I end up
• 55:26
  using it all too much. So yeah, perfect.
• 55:29
  So LinkedIn will put that in the follow-up email and my last
• 55:35
  question of part is can you talk to us just quickly about some
• 55:42
  obstacles that you face? When you were building the
• 55:45
  platform, something that you thought in the beginning was
• 55:48
  going to be easy. And it wasn't or just something
• 55:51
  that came out of nowhere and blindsided you.
• 55:56
  I think. I think we knew this was going
• 56:00
  to be hard but it was still super hard when we did it which
• 56:02
  was justifying their requirements and like making
• 56:05
  sure we stick to it. At the end of the day, we were
• 56:09
  trying to create something that we didn't have too many examples
• 56:12
  to follow. Like when we will be would often
• 56:16
  do is we would talk to other ml teams at other organizations.
• 56:20
  Like oh hey what are you guys doing?
• 56:21
  Or what are you doing? And that helped us figure out
• 56:24
  like oh what was manageable tasks or mental thing we could
• 56:27
  take on or sometimes They would talk about their own obstacles
• 56:30
  or nightmares that let us know, like, oh, what's possible or not
• 56:33
  possible, or what should we really try to do?
• 56:36
  But I think I still think of the other end of the day, it was
• 56:39
  really, was scoping like, figuring out.
• 56:41
  Like what, what can we take on? What can we do?
• 56:43
  Is this possible? What do we really need?
• 56:48
  And I think we spent a lot of months doing that and having a
• 56:54
  lot of back and forth on all aspects about that.
• 56:59
  The we still probably in terms of it are literate, orations,
• 57:05
  probably still have that, but I think that was probably the
• 57:08
  biggest obstacle. Another one was definitely just
• 57:10
  expertise. We ended up using a lot of tools
• 57:13
  that were we weren't necessarily familiar with initially.
• 57:17
  So, it was a learning process for us like we hadn't used pie,
• 57:21
  spark heavily on like, say our feature stores and of things
• 57:26
  learning. That was what stuff It's
• 57:29
  definitely a learning curve there.
• 57:32
  I would say that those are probably our two large obstacles
• 57:35
  that were over there. Perfect.
• 57:40
  Thanks so much. Who be and connect with him on
• 57:42
  LinkedIn. Thank you everybody, for joining
• 57:44
  us. This has been an awesome
• 57:46
  conversation. You all asked some amazing
• 57:48
  questions. I'll send up a follow-up email,
• 57:52
  and I look forward to seeing you.
• 57:54
  We're doing this every week. So, please come join us.
• 57:58
  Next week. See y'all later have a great
• 58:01
  night. Stay safe out there.
• 58:03
  Thanks here and feel safe.