Latest / Agentic Conversations (formally mlops.community) / How A Manager Became a Believer in DevOps for Machine Learning // Keith Trnka // MLOps Podcast #152

Transcript

How A Manager Became a Believer in DevOps for Machine Learning // Keith Trnka // MLOps Podcast #152 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm Keith trinka currently on sabbatical, and I like to do my
• 0:04
  coffee and Aeropress. So even though I'm traveling
• 0:07
  right now, I've got my hair pressed.
• 0:08
  I got my hand grinder, and it's just a wonderful way to do it.
• 0:14
  Welcome back to the envelopes Community podcast, everyone, I'm
• 0:17
  your host Demetrios and today is just me.
• 0:20
  It's a very special episode. I'm talking with Keith trinka.
• 0:23
  This man has been in the game for about 11 years.
• 0:28
  He is the death. Finition of what I would call
• 0:31
  leader. I would love to have him as my
• 0:34
  boss at my next job thinks through things, so thoroughly
• 0:38
  and he is able to lead, through example, it's incredible.
• 0:44
  What we were able to talk about whether that was leading data
• 0:47
  teams or machine learning teams becoming a better software
• 0:51
  engineer himself. I mean, you name it.
• 0:54
  We talked about a whole migration, from a monolith to
• 0:57
  microservices, and they had zero Absolutely zero down time during
• 1:02
  that whole process and it was especially important because he
• 1:06
  was working in the healthcare sector.
• 1:09
  So, Let me give you a bit of background about Keith.
• 1:12
  Before we jump into this full conversation.
• 1:15
  Most recently, he was the director of data science at 98.6
• 1:21
  where he made telemedicine visits easier for doctors using
• 1:26
  natural language. Processing machine learning back
• 1:28
  in engineering, a little bit of AWS and some front-end
• 1:32
  engineering sprinkled on top there.
• 1:34
  So basically, they made a killer application with technology for
• 1:39
  doctors to use. Use and it had a little bit of
• 1:41
  machine learning in there and that's where Keith fit into the
• 1:44
  mix prior to that though. Keith improve the language
• 1:48
  models used in mobile phone, keyboards at swipe and nuanced
• 1:54
  and before that he did a PhD thesis in language modeling for
• 1:58
  assistive technology and currently the man is living the
• 2:02
  dream. He is on sabbatical somewhere in
• 2:05
  Spain. I think one of the islands that
• 2:07
  they have I am always jealous whenever I talk to him over.
• 2:10
  Last couple weeks I've been getting the opportunity to chat
• 2:14
  with him a bunch and it's always sunny where he is and it's cold
• 2:17
  and rainy where I am. So let's jump into this
• 2:21
  conversation. We had with Keith so go for it.
• 2:29
  How did you get into machine learning?
• 2:31
  Well, for me, it really started an undergrad.
• 2:35
  I took a an artificial intelligence class at the
• 2:38
  college in New Jersey and it Is a really interesting class and
• 2:41
  then I did a research project with Professor later on and then
• 2:48
  through various twists and turns and went to grad school, I went
• 2:51
  for a PhD at the University of Delaware and you know then I got
• 2:55
  more and more excited. I was I was more on the language
• 2:59
  modeling side for a while and that that was my thesis, work
• 3:03
  was in language modeling for assistive, technology, State and
• 3:06
  language Molly for a while. And then being an industry, I
• 3:09
  started to see that That language modeling was a bit of a
• 3:12
  narrow scope, and I want to expand more generally into other
• 3:16
  kinds of natural language processing and machine learning.
• 3:18
  So, I slowly made that transition after my first role
• 3:21
  at swiping nuance, Nice. So then you started working, you
• 3:27
  went to swipe and you went to nuance.
• 3:31
  And then how did you start to apply?
• 3:35
  What you had learned like from school?
• 3:37
  Or from those places? I would say both that's actually
• 3:41
  a good call and I was thinking about school in the beginning,
• 3:44
  but I imagine you're learning a ton on the job.
• 3:46
  It's, you know, you're really learning all different angles,
• 3:50
  angles of things. And I think, I've been fortunate
• 3:53
  enough to have a wide range of experiences, so so one of the
• 3:57
  things that's that's maybe it's not uncommon but it's not
• 4:00
  totally common, which is having teaching experience from my time
• 4:05
  in grad school, and I learned a lot from that.
• 4:08
  And then, that helps me when I'm entering people or if I'm doing
• 4:12
  a lecture in a company or any kind of presentation there.
• 4:17
  And then in Industry you find probably in your first week that
• 4:21
  it wasn't like School thought. You where's all the clean data?
• 4:25
  Yeah, yeah, yeah. Well yeah, definitely.
• 4:30
  That where's all the cleaned it even on the software engineering
• 4:32
  side, right, my undergrad was was more focused on software
• 4:35
  engineering and you know, you think, oh, I just write code but
• 4:39
  but over the years and Industry, I think for me, I started to
• 4:44
  feel more and more that code was in a lot of ways communicating
• 4:48
  with other Developers. And and so, what do you mean by
• 4:52
  that? Well, you know, how you name
• 4:54
  your variables, how you name, your functions, how you
• 4:55
  structure your code is you have to get your head space into
• 5:00
  their head and you have to get their head space into your head.
• 5:04
  And so writing code effectively. There's a lot to it, of course.
• 5:09
  But a lot of it is communication when it comes down to it.
• 5:12
  And a lot of the mistakes that happen are miscommunication.
• 5:15
  Of course, school doesn't teach you that school.
• 5:18
  Teaches you, you write the code. And your big O and all this and
• 5:21
  such execute. Hello world.
• 5:24
  Yeah and then of course you have to deploy the stuff and get it
• 5:27
  out for users and then you have to monitor it.
• 5:29
  Make sure it doesn't break. So yeah I learned a ton from
• 5:32
  there but over my career I would say in each different role
• 5:36
  they're even probably some internships earlier on where I
• 5:40
  think over time what I saw was that it usually wasn't the
• 5:45
  machine learning part that led to a project failing.
• 5:49
  Usually it was trying to do something that didn't make sense
• 5:54
  that the users didn't need or the business didn't need.
• 5:57
  Sometimes being too excited about technology or not excited
• 6:01
  enough about understanding the needs of who you're building for
• 6:06
  that. That's one really common failure
• 6:08
  mode and other really common failure mode is Not necessarily
• 6:12
  the machine learning part, but everything around it, right?
• 6:14
  Like, how do you get something out to a server?
• 6:16
  How do you when it goes wrong? How do you revert it?
• 6:19
  How do you know when the revert it?
• 6:21
  How do you know quickly, you know, have you make sure you're
• 6:23
  not woken up 20 times in the middle of the night from alarms,
• 6:27
  at least and all those things end up, being a really big
• 6:32
  factor in actually, making a difference for your users are
• 6:35
  actually making a difference for the business.
• 6:38
  And so I've had the opportunity of having a few different Oz and
• 6:42
  being able to see some of those failure modes over time is what
• 6:48
  got me into more, starting to see the bigger picture and is
• 6:52
  starting to focus more on understanding, what our users
• 6:57
  really need and put extra effort into that focus more on how we
• 7:01
  are going to deploy things in a way.
• 7:03
  That's not going to break our backs.
• 7:05
  How we're going to make sure that things are reliable safe.
• 7:10
  That we don't have. Very surprises.
• 7:12
  You can't eliminate all surprises, but you can, you can
• 7:15
  turn them down a little bit, right?
• 7:17
  But yeah, yeah. I would say, you know, a start
• 7:18
  off more on that ml side, man. Once I realized what it took to
• 7:22
  make a difference for the users, a kind of expanded into a lot of
• 7:24
  the other areas, you mentioned something fascinating, which
• 7:29
  going back to the code being a way that developers are able to
• 7:33
  talk with each other and you have to really take time to
• 7:37
  think through how you're naming variables and the Comments, that
• 7:41
  you're leaving and all of that good stuff.
• 7:43
  Do you feel like some of the pain sometimes when working data
• 7:48
  scientist, try and work together is because they don't have that
• 7:52
  as much. And especially I think about
• 7:54
  when you're playing with a jupyter notebook, sometimes it
• 7:56
  can be very clear and it's just like run execute.
• 7:59
  If somebody has set up that notebook, knowing that others
• 8:03
  are going to use it. But for the most part I feel
• 8:07
  like, that's not really how things especially in
• 8:10
  Exploration. Mode are set up.
• 8:12
  And so, I've heard stories about a data scientist taking six
• 8:17
  weeks, when another colleague leaves the company and they're
• 8:21
  just trying to like piece together, what the hell was
• 8:24
  going on? And so I think back to your
• 8:26
  phrase of hey this is how we're talking to each other.
• 8:30
  When you look at it like that and you look at how just
• 8:34
  inherently data, scientists work and explore.
• 8:37
  Do you feel like that is a good assumption?
• 8:41
  Yeah I think there's two different things going on there,
• 8:43
  right? I think there are some unique
• 8:45
  challenges in trying to explain what you mean by a piece of code
• 8:51
  to another person. Some unique challenges when
• 8:53
  you're dealing with data science and also there are challenges
• 8:58
  with dealing with exploration and I think those challenges do
• 9:02
  apply even in the software engineering world as well.
• 9:05
  And so I think over time we'll start to see those Carlos Moore
• 9:09
  but To go back to some of the challenges in data science.
• 9:13
  I think machine learning is a complex field and the way I view
• 9:17
  it is that there's a lot to know, there's a lot to learn,
• 9:20
  there's so much terminology. It's so you know, math driven
• 9:23
  and sort of that old school math, which has a lot of
• 9:26
  terminology and people really have to adapt to that to learn
• 9:31
  and to develop something. And forget and not everyone has
• 9:35
  that background yeah right and so I think data science, pushes
• 9:38
  people. In some ways, pretty hard on
• 9:42
  some of the terminology and then an individual data scientist.
• 9:47
  Might forget that. Not everyone has the same
• 9:49
  background. I think it's natural to assume
• 9:52
  that other people think the way you do or have the background
• 9:54
  you do. Because that's the easiest way
• 9:56
  to do it. It's like about.
• 9:57
  That's why I say it's kind of a miscommunication, it's just,
• 9:59
  it's just a mistake. It's not, you know, something
• 10:02
  worse going on. Usually it's just the worked
• 10:05
  really hard to learn all this terminology and they assume
• 10:07
  everyone else had to sew. I think there is some stuff
• 10:11
  unique there and on the exploration side of things I
• 10:17
  think they're it's so much of a struggle to make something work
• 10:20
  at all or to make something that might work or has the potential
• 10:25
  via useful at all that that's got to be your top priority.
• 10:31
  But what I like to do, actually one thing that I will I'll take
• 10:35
  a quick detour and say, I think one really eye-opening
• 10:39
  experience that probably has happened to you've probably gone
• 10:42
  through this too, is you look at a piece of code, you say, oh,
• 10:45
  this is awful, we wrote this. And then you look at it as like,
• 10:48
  oh, I wrote this and know where you're going.
• 10:51
  Yep, it's just like such a formative experience when, when
• 10:54
  I had some moments like that, I said, oh, okay.
• 10:57
  You know, it's not just about for other people.
• 10:59
  It's also from Myself down the road and that also gives you a
• 11:04
  hint of ways. You can learn that as you can
• 11:06
  look at some of your old code and say, how can I improve this?
• 11:09
  What can I make this last the test of time, but in Exploration
• 11:15
  mode, I think, if you hit a wall in terms of your research and
• 11:18
  your kind of stomped on ideas, go take 10 minutes, clean up
• 11:21
  your code, break something from a giant, jupyter, notebook into
• 11:25
  cells, that are well-explained while commented or chip away at
• 11:28
  it over time. Also really recommend to people
• 11:32
  like when parts of your notebook, become stable, start
• 11:36
  movement of over. It's a python file, so you're
• 11:38
  getting ready for them to become, long-lasting code, that
• 11:41
  sort of thing, import them into multiple notebooks that sort of
• 11:44
  thing. So I think it's something you
• 11:46
  can chip away at you have to develop the Habit.
• 11:50
  It's just kind of hard though. Exploration is just one of those
• 11:52
  things where you you're left with 90% code that You know, it
• 11:57
  was throw away. And then if you're lucky, I'm
• 12:00
  percent of the code survive eyes.
• 12:02
  Yeah. Yeah set.
• 12:05
  So funny. And I want to put a pin in that
• 12:08
  because I want to come back to some of those best practices
• 12:11
  that you found when you do find something that you want to run
• 12:15
  with and you mentioned. Yeah.
• 12:17
  Starting to find that stable notebook piece or these pieces
• 12:22
  that you then want to keep and make sure you start putting them
• 12:27
  into python code or doing other things with it.
• 12:30
  But before we jump into that, I I am fascinated with the idea
• 12:35
  that you were talking about where you started to recognize
• 12:38
  the bigger picture F. And it reminded me of I'm
• 12:41
  reading this book right now, build by and Tony fadell the
• 12:46
  dude who created The Nest thermostat and, and it's a great
• 12:52
  book and he talks about how even if you're an individual
• 12:55
  contributor. What he says, is you still have
• 12:58
  to look up sometimes and you have to recognize look up and
• 13:01
  look around. Basically, and it's talking
• 13:04
  about how you may think that everyone is knows what's going
• 13:10
  on, and they're pointing you in the right direction.
• 13:12
  And then you just have to go and execute on that direction that
• 13:16
  they're pointing you in, but it is very beneficial.
• 13:20
  If you look up and you look around by talking to people by
• 13:24
  checking, with other teams by seeing what is actually going on
• 13:29
  that, you can better understand the product.
• 13:32
  That you're creating. And so it sounds like you, you
• 13:36
  kind of got that overtime. You were her, what is it?
• 13:39
  Baptized by fire if we could say, yes, and that's it.
• 13:43
  So maybe we can go down that road for a little bit.
• 13:48
  Yeah, I mean, I think even, even when I was, you know, I swipe a
• 13:52
  Nuance I was say a couple of years in.
• 13:55
  I really like to have the context behind things.
• 13:57
  I didn't necessarily know why. I just knew that I did a better
• 14:00
  job with it, but now, the way that I usually explain it is
• 14:04
  that say a mission statement or jira ticket or some some way of
• 14:10
  describing the work or requirements.
• 14:12
  It's going to tell you. Three of the things that you
• 14:16
  need to do in your project in in the next month or so, it's not
• 14:22
  going to tell you the hundred small decisions that have impact
• 14:27
  on the user, that will come up in the nature of that
• 14:31
  development. And so, having that context,
• 14:34
  gives you a way to fill in those blanks in a way that's aligned
• 14:38
  with your users. That's aligned with the business
• 14:41
  that's aligned with the other business units.
• 14:42
  Fascinating. And and me Is that at the end of
• 14:46
  the day, you have a really coherent product, as well,
• 14:49
  aligned with the needs. And so, that's how I think about
• 14:54
  it now is that having that context allows you to make more
• 14:58
  effective decisions and development?
• 15:00
  What does that mean? Like the context and you just
• 15:03
  talked to others? You ask for some kind of report
• 15:06
  as to why we're putting this into play.
• 15:09
  What does that actually look like?
• 15:11
  Yeah. So I think the simplest way that
• 15:15
  I like to put into Citrus, try to understand users.
• 15:18
  You know, try to try to empathize with them, understand
• 15:21
  what their frustrations are. So my last roll was was in
• 15:25
  healthcare and I hadn't worked in healthcare before.
• 15:29
  I didn't really understand how doctors think and what their
• 15:33
  needs are. But now I have a much better
• 15:35
  sense of that. And so you know, if I'm going
• 15:38
  into a project in healthcare are now, I know that doctors are
• 15:43
  sensitive to Things like liability were sensitive to
• 15:48
  safety for instance and a lot of ways.
• 15:51
  And so if a project has been handed to me or, you know,
• 15:54
  someone has an idea and we're talking about it, but no one's
• 15:57
  talked about, like, liability or safety.
• 16:00
  I'm going to step back and think, oh, I know that this is
• 16:02
  something that doctors really care about.
• 16:05
  Are there any implications here as the sometimes?
• 16:08
  I can answer that question and sometimes it just means that I
• 16:12
  say, okay, well what about substance?
• 16:14
  And start to work with other people to fill in those blanks.
• 16:17
  So it's not necessarily that you can fill all of the blanks in
• 16:21
  yourself, but at least knowing, you know, when to double check
• 16:25
  what, you know what, you're trying to build as much as
• 16:27
  possible. And another way to look at it is
• 16:30
  that if you're working with a p.m. or someone in ux is
• 16:35
  everybody's got each other's backs.
• 16:36
  They're not going to get 100% of the requirements traffic.
• 16:40
  You know, if they do a phenomenal job maybe they'll get
• 16:43
  70% of the requirements, right? And you can help out by asking
• 16:48
  some of these questions and getting it to the 75% 80% and
• 16:52
  then what you can do to go further, as you can, build down
• 16:55
  all those, you can demonstrate it to your potential.
• 16:59
  Real users you can listen to their feedback, you can ask, why
• 17:03
  questions? You know, try not to lead them
• 17:04
  such you can start to understand how it may or may not work for
• 17:08
  them. You glossed over something there
• 17:10
  that I want to double click on, which is around how you had no
• 17:16
  experience in healthcare. And then you started working in
• 17:19
  healthcare and you recognize that doctors had certain
• 17:24
  Passions or they cared deeply about certain things.
• 17:28
  How did you even get to note? That was that by listening to
• 17:31
  calls with different doctors or did you go out and talk with
• 17:34
  doctors or did you just was that common knowledge around the
• 17:37
  office? Well, that's a great question.
• 17:41
  So, I join that company when we were fairly small.
• 17:43
  So, we were like 15 to 20 people.
• 17:45
  Then we grew over the years and we had one full-time doctor on
• 17:50
  staff. We didn't actually have a
• 17:52
  working Product yet. And and so I really need it like
• 17:57
  chat logs between patients and doctors.
• 17:58
  And so, one of the things that we did is we created artificial
• 18:03
  scenarios and I read about patient acting and I learned as
• 18:07
  much as I could about that, and you would be the doctor and I
• 18:10
  would be the patient and run through.
• 18:12
  So you have a career in acting is that if that's drawing on, I
• 18:17
  get it. Now, it all makes sense.
• 18:18
  Is that what you're doing in Spain?
• 18:20
  Known for? I mean, I would Like right now I
• 18:24
  know. No.
• 18:25
  Now, I mean, you know, we were just trying to solve the problem
• 18:28
  of how do we get some data? And I think that we would go
• 18:32
  through and then, you know, maybe afterwards.
• 18:34
  Oh, it's a oh, why did you know, read through 10 of our chats and
• 18:38
  I would say, oh well, why did you do this or why did you do
• 18:40
  that? And then I'm that doctor would
• 18:42
  explain it and I would say, oh okay, that's what you're doing
• 18:46
  like well but you don't you pretty much know that this is a
• 18:48
  common cold. It's like yeah, but have to make
• 18:50
  sure it's not this or have to make sure it's thought this.
• 18:53
  Because even though it's unlikely, it could be fatal,
• 18:56
  right? And so it's worth that extra
• 18:59
  check to check for these things. And so we'll talk to him about
• 19:03
  those kinds of things. And then, sometimes we would
• 19:07
  make test harder, you know, with it was one where we try to do in
• 19:11
  a situation where I was distracted.
• 19:13
  Then Phi, we learned a lot from that, don't don't try to
• 19:17
  distract doctor as well or practicing.
• 19:19
  But over time it's, you know, we made a man.
• 19:23
  Friends with not just him, but a lot of the other doctors and
• 19:27
  talk to them, listen to their feedback.
• 19:29
  When when things didn't go, well, when they didn't, like,
• 19:33
  any of the features, or like some of the features that we
• 19:35
  built, I would really try to understand why and try to
• 19:38
  understand what it was about that and just ask lots and lots
• 19:43
  of questions and get to know them.
• 19:44
  And, you know, let's build that relationship as well and they'll
• 19:48
  just volunteer information to you is another way to go about
• 19:52
  it. So there are A lot of things
• 19:54
  there of just connecting to people and learning from them.
• 19:57
  That was a very fortunate situation where the doctors were
• 19:59
  part of the company. And so I could do that, that's
• 20:02
  not always the situation, building Healthcare technology.
• 20:04
  And so, it could be much tougher to get user feedback.
• 20:07
  You might have to rely on, you know, a dedicated team to do
• 20:10
  that and read the reports. But that's, that's another way
• 20:13
  to do it. You can also just read through
• 20:15
  data center is another way to go about it or sort of analyze the
• 20:19
  data. So actually you can to get
• 20:21
  multiple different perspectives on it.
• 20:23
  Same thing. Yeah, that's brilliant.
• 20:25
  The idea that you as a Engineer or data scientist would go out
• 20:32
  there and try and figure out what your users.
• 20:35
  The doctors wanted and you're not at all like your role didn't
• 20:40
  say product manager and no title if I'm not mistaken and so it's
• 20:45
  just something that you're doing because you saw over time that
• 20:49
  projects were failing because you didn't have an understanding
• 20:52
  of what you're building. Yeah, I think at that comic, it
• 20:57
  happened. So early on that, there were
• 21:00
  some cases where projects didn't go well because they were there
• 21:04
  were maybe not well aligned with the needs of our users or
• 21:07
  business but I don't think it was soon enough that we had a
• 21:12
  lot of clear signal on many features.
• 21:15
  So it's not necessarily that You know, things were disastrous and
• 21:20
  sort of, like, intervene to do all this stuff.
• 21:25
  Also, it's also one of those things that's easy to do in a
• 21:27
  small company. If you're all working in the
• 21:30
  same room, you get to know each other, you have lunch together,
• 21:32
  you chat, you build relationships, and you start to
• 21:37
  learn about, you know. Okay.
• 21:39
  So each different person has a role to play in the company and
• 21:44
  how is that role is valuable to the mission of the company.
• 21:48
  And sort of how what their concerns are.
• 21:51
  You know, when you're, when you're talking to them, what are
• 21:53
  they, what are they worried about, right?
• 21:55
  So why is it that they worry about certain things?
• 21:58
  And so there's just I feel like there's so much to learn there
• 22:01
  from frankly just making friends around the office and learning
• 22:05
  what they do and why and what their concerns are.
• 22:08
  There was one thing you said though there was about sort of
• 22:11
  asking doctors what they want. Well, I would say if you're
• 22:15
  going to talked about like the mom test, Is that book a while
• 22:18
  back? And yeah and it's it's a little
• 22:22
  bit you can ask that it's a little bit risky though.
• 22:25
  Ask people for the solution that they want.
• 22:27
  A lot of times people will gravitate towards what they're
• 22:30
  familiar with and then it becomes very hard to innovate if
• 22:35
  you're rebuilding the same thing over and over again.
• 22:39
  Now, there are times and places where that's the right thing to
• 22:42
  do. And there's tons of places where
• 22:43
  that's not the right thing to do.
• 22:45
  So it's always good as much as possible.
• 22:47
  You to dive deep into try to understand the motivation behind
• 22:51
  it, which is why I brought up safety as an underlying thing.
• 22:54
  That's not telling you the solution is telling you
• 22:56
  something that they're very concerned about for instance,
• 22:59
  and an understanding that. Then you can start to think
• 23:02
  through, what's your next set of questions for for a doctor or
• 23:06
  what Solutions might be just off the table because they have
• 23:11
  some, some - safety implications, how do you balance
• 23:15
  between the drive? To try and create something new
• 23:19
  or explore and cutting your losses.
• 23:24
  And saying, this new project is never going to see the light of
• 23:29
  day. It's just not working out that
• 23:31
  that's such a hard question. I can tell you what I've done, I
• 23:35
  don't know that I have some sort of deep wisdom there.
• 23:38
  I would say because it's like you say you're kind of cutting
• 23:43
  your losses on a project, the project.
• 23:44
  Maybe you started off your optimistic and You know, three
• 23:48
  to six months in, you don't have any results, you've tried out a
• 23:52
  few different things, nothing's nothing's working and you're
• 23:57
  asking yourself. Well, do we just not have the
• 24:00
  right people on team for it? Is it the problem where we're
• 24:02
  missing skills? Is it a problem where we don't
• 24:07
  really have a clear idea of what we're building, you know,
• 24:10
  sometimes that's that's it. Is it a problem where the tech
• 24:15
  just isn't there yet? Right?
• 24:17
  So like the hotness right now is GPT right?
• 24:21
  And it's not there yet to practice safe medicines right?
• 24:26
  Like You can't but you know so there's there's and you just
• 24:31
  don't know if you've put time into it you just don't know why
• 24:36
  it didn't work out. You don't know if something else
• 24:39
  had been different, could it have worked out?
• 24:42
  And so, the best I can really say, Is that before you get too
• 24:48
  far along like maybe once your couple days or week in you
• 24:53
  should put a real effort into saying, what's what's the best
• 24:57
  possible outcome of this project?
• 25:00
  What are some other realistic outcomes of this project?
• 25:02
  Try to do some kind of you know what percentage benefit to our
• 25:06
  users? Would this have try to put it in
• 25:08
  terms of business metrics as much as possible and say to
• 25:12
  yourself how much time is, is worthwhile to put towards that
• 25:20
  because if it's something that's a very small benefit, you might
• 25:23
  feel a say, Hey, you know, if if we spend more than two months of
• 25:28
  companies harmonic even in the best-case scenario, the company
• 25:30
  loses money, There's a right you can do some things like that.
• 25:35
  Usually though it's very hard because the the best case
• 25:38
  scenario is is very high and the worst case scenario was very
• 25:42
  negative. And so at some point you make a
• 25:46
  call and there are a lot of projects where I look back and I
• 25:50
  say, oh, if we could put another six months into it wouldn't have
• 25:54
  worked. Or if we had trained up our
• 25:57
  skills in a certain area would have worked or if if we had
• 26:01
  pursued it in a different way, Would it work?
• 26:02
  And I just don't know. I'm left wondering a lot of
• 26:06
  times. I think.
• 26:07
  One thing I will add to that though is that there's sort of
• 26:10
  the leadership aspect of it. I think you're getting out of
• 26:13
  like, okay, 11 to cancel a project, I guess, has this sort
• 26:16
  of healthy way, flooding it, but if you do, you really want to
• 26:21
  have people involved in that, you don't want to be a leader
• 26:24
  that's coming down and saying, like look, For six months and
• 26:28
  and it's canceled today. You don't want to write like you
• 26:31
  want to work with the people as much as possible to say.
• 26:36
  Hey like it's been six months. Here's as much as possible.
• 26:42
  Here's the dollars that our company has spent investigating
• 26:45
  this. Here's the best of our knowledge
• 26:50
  potential benefit and talk with people and say you know, hey
• 26:54
  like it's should we set this aside for now and maybe we'll
• 26:58
  come back to it in a couple years.
• 26:59
  So if we have a good idea but there's the how to it like sort
• 27:04
  of how to to do that. Well so that people aren't
• 27:08
  demotivated as a result afterwards, that's so good.
• 27:11
  So now getting back into this nitty-gritty, when you're
• 27:14
  getting ready to go to production and you're coming
• 27:18
  from your jupyter, notebook, you explored.
• 27:20
  And let's say that this is the successful time.
• 27:22
  It is one of those joyous moments that thing worked and
• 27:27
  you weren't to now start making it more of a process.
• 27:32
  What are those best takeaways that you found for actually
• 27:38
  solidifying that process? So a good first part and this
• 27:41
  can happen entirely by the person that wrote The Notebook
• 27:46
  is to simplify like that. That should be step one.
• 27:49
  Simplify probably there's a bunch of like Subtle features in
• 27:54
  your jupyter notebook that ad you know, double the size of
• 27:58
  your code and add 1% quality or something like that.
• 28:02
  There's probably a lot of exploration like that.
• 28:04
  Rip it all out as much as you can.
• 28:07
  And try to try to simplify because all of the later steps
• 28:12
  are kind of multiplying on that size of that code and the
• 28:17
  complexity of that code. So that's that's a big first
• 28:19
  step. Next time after that, I would
• 28:23
  say for us, it really depended where we are at, in our journey.
• 28:27
  So early on we didn't have a lot of the devops.
• 28:32
  I'm a locks skill set. We are very comfortable with
• 28:35
  even Docker, right? And you know, we we could write
• 28:39
  services and flask and Python and such but we weren't really
• 28:43
  comfortable with a whole lot of those parts and so for us, what
• 28:48
  it would turn into is start to convert.
• 28:50
  Vert your jupyter, notebooks into a training and training and
• 28:54
  serving code and put them into a model list.
• 28:57
  We had What was it? I think Jenkins or something
• 29:02
  like that, to, to do model rebuilds and we had some
• 29:07
  processes around model versioning and so by doing kind
• 29:11
  of a mono repo setup when we're going to production, we didn't
• 29:15
  have to reassess things. Like, how do you how do you
• 29:18
  version models? How do you train models?
• 29:19
  How do you deploy models? How do you host models that you
• 29:23
  do? You know, have you do auto
• 29:25
  scaling all these things? It was all sort of figured out
• 29:27
  for us. I'm so that's sort of how that
• 29:30
  was over time. We switch to something more like
• 29:33
  microservices where you know we might have one service for a
• 29:37
  model so that it would be like one repo with training and
• 29:39
  serving in it and we didn't have so many models that we had a
• 29:46
  well-defined template, but we're kind of getting there.
• 29:49
  And so a lot of times we would do that.
• 29:51
  It would usually involve partnership between a research
• 29:56
  scientist and a software engineer.
• 29:59
  It had some 80 BS experience to sort of say Hey you know, you
• 30:02
  want to do this thing. You're some some things that you
• 30:04
  got to think about. Or hey, you've got some unique
• 30:06
  situation with a particular model.
• 30:09
  Here's what we got to do. I would say I'm also leaving at
• 30:12
  one step in there, which was for us, there was there are multiple
• 30:17
  stages of experimentation. So you have a jupyter notebook,
• 30:20
  stage of experimentation. You have some idea that
• 30:24
  something has potential for users, and the next stage after
• 30:29
  that, isn't a full release for everyone.
• 30:32
  Right. There are a lot of projects that
• 30:34
  proceed that way. What I had seen was that you
• 30:38
  would typically take months and months to jump from say jupyter,
• 30:42
  notebook to full prod release. And then there were just be a
• 30:47
  lot of like really low hanging fruit.
• 30:49
  Kind of feedback. Like, oh, it doesn't do this,
• 30:51
  right? It doesn't do this right?
• 30:53
  That we could have known and we should have known Everyone, my
• 30:58
  name is aparna founder of Verizon and the best way to stay
• 31:01
  up to date with Emma lops is by subscribing to this podcast.
• 31:07
  And so, what we move to was doing a limited launch, so we
• 31:12
  had an internal beta test group and we would develop for that
• 31:17
  test group. Now, if you're doing an internal
• 31:19
  beta test, You do still have some of the same software
• 31:23
  engineering standards, you know, around say security and privacy
• 31:28
  but if it goes down for a few hours, that's okay.
• 31:30
  It's not the end of the world, it's just a beta test, right?
• 31:33
  And so you're able to accelerate the development of that and then
• 31:37
  launch it with a limited group. And then from there, a lot of
• 31:41
  the things that helped are developing a culture where your
• 31:46
  users and your developers are talking to each other in a
• 31:49
  productive way. And not in like a sending
• 31:52
  10-page, manifestos way. But in like a, you're just in a
• 31:55
  slack Channel talking like, normal people way.
• 31:59
  And then what you can do is you iterate on that little but a
• 32:02
  little week by week until until you've taken care of, a lot of
• 32:06
  the low hanging fruit and then you can have a go no-go moment
• 32:10
  for launch of production. So that's what we did.
• 32:14
  That was extremely effective. There were times where something
• 32:18
  worked in Notebook and we couldn't find find a way to
• 32:21
  build a feature that was valuable for our users.
• 32:24
  And then we said, you know, well, we're going to set this
• 32:27
  back on the shelf for a while. Maybe we'll come back to it in
• 32:30
  some number years. Or maybe we just think that this
• 32:33
  isn't a valid approach, but then there are other times where was
• 32:36
  highly successful and then those users, those beta testers,
• 32:40
  become your Advocate and they talk to their peers and say,
• 32:43
  hey, we really want this feature and so they're kind of pulling
• 32:46
  your feature into the product, instead of you pushing against
• 32:49
  some kind of roadmap. So it that there's so much so
• 32:53
  many layers to the cultural aspect to it, to get things
• 32:56
  working smoothly, where, you know, it's not so heavy on
• 33:01
  process, but it's more about like, hey, you're trying to do
• 33:04
  good for good for your users and how can you make that smooth?
• 33:09
  So how did your team evolve? I know that we had a great
• 33:12
  conversation week or two ago about how you and you said
• 33:16
  something that has stuck with me which is you weren't able to to
• 33:20
  properly hire like a senior engineer.
• 33:24
  Until you felt like you were a good engineer or at least a
• 33:29
  decent engineer. I think this is the and so,
• 33:34
  because you're very humble, I'm sure you're an incredible
• 33:36
  engineering trying to appreciate how you down.
• 33:38
  It's more because I think you didn't, you didn't want to,
• 33:41
  like, talk yourself up but that to me was fascinating because
• 33:45
  it's like, was that because you couldn't attract the right
• 33:48
  Talent? Was it because you You weren't
• 33:51
  able to properly judge who was good, and who was bad?
• 33:55
  How the great question explain that to me, is so, so early on.
• 34:00
  So this was at 98 my six and early on, I think why I joined I
• 34:05
  was like, oh, you know, I'm here to do the machine learning stuff
• 34:09
  and I knew that involved, creating data sets as well.
• 34:12
  I think some people kind of separate those two, but I knew,
• 34:14
  you know, all right. Going to create its to do,
• 34:17
  annotation all that sort of stuff.
• 34:19
  Maybe a little bit of back and Engineering.
• 34:22
  I didn't go into it thinking, oh, you know, I'm going to get
• 34:25
  into how we design databases or into the ADF PS side of things
• 34:29
  or into security side of things or any of these things or front
• 34:34
  end or any of this. But over time, I started to see
• 34:37
  that as the company was scaling. Our lead time was slowing down,
• 34:42
  so our time from a research, prototype success to getting it
• 34:48
  into production being used, actively that time was taking
• 34:52
  longer. And just the nature of scaling
• 34:56
  company. And that's sort of how it goes
• 34:59
  at least. That's how wet I should say.
• 35:02
  I started to look at the bottlenecks, some of it was just
• 35:05
  coming down to like, cross org stuff, I started to see.
• 35:07
  Okay. Well, we really need to
• 35:10
  understand Docker say or we need to understand, flask better, or
• 35:16
  we need to understand on call better or all these different
• 35:19
  concepts and so, Over time, I started to realize, okay, well,
• 35:24
  we really need expertise in this area.
• 35:29
  We tried some things of pulling in people from other teams, but
• 35:32
  they had their own priorities. And so that was always kind of a
• 35:35
  challenge. We tried hiring, I would say
• 35:39
  this was early on at the company and now that I have a lot more
• 35:44
  experience. I have a very different
• 35:46
  perspective on hiring, but at the time I thought it was just
• 35:48
  like you write a job description, send it to HR, and
• 35:52
  then they'll send you a bunch of great candidates and you're
• 35:54
  done. If it doesn't work that way, you
• 35:57
  know, later on. On, I came to see you that it
• 36:00
  really helps to be personally involved in hiring and to
• 36:03
  iterate on your job description, as much as possible.
• 36:06
  Get out there talking to candidates.
• 36:07
  If, if you're not getting enough candidates through escalate with
• 36:11
  your, you know, with your recruiting team, or do some
• 36:14
  sourcing yourself or Outsource that there's so many options you
• 36:17
  have, but you got to solve that problem at the time.
• 36:20
  I wasn't in that mindset yet. And so you know, we're trying to
• 36:25
  hire someone more senior because we knew we're having we're
• 36:28
  spending. Simon engineering problems, kind
• 36:30
  of almost standard, engineering problems with the slight ml
• 36:32
  twist. I think one one point for me
• 36:36
  that was a real turning point was, we were doing a project a
• 36:40
  proof-of-concept project with sage maker.
• 36:42
  I think it was. I think that's that's what it
• 36:44
  was. And had someone on my team that
• 36:47
  had maybe a couple years experience and it was a real
• 36:50
  struggle. There's so many different
• 36:52
  concepts and especially if you're dealing with keeping your
• 36:55
  data, really private, you have really strict.
• 36:57
  I am Requirements internal company requirements for for how
• 37:04
  lock down your I am is that can be a real struggle.
• 37:08
  Especially for someone that's, you know, say, a few years out
• 37:10
  of school, right? And so I said, look, we're not,
• 37:14
  we're not pregnant progress on the hiring thing.
• 37:17
  Maybe, at least I can look into this a little bit and try to
• 37:19
  understand at least basic. Maybe I can be like the person
• 37:22
  that they talk to even if I don't have all the answers,
• 37:24
  maybe I can help them talk through them.
• 37:26
  That's sort of where I started. And then over time, I started to
• 37:30
  get more and more into that. That was sort of how I learn
• 37:36
  more of the nitty-gritty of AWS. And then I started the co
• 37:38
  actually, like, putting a little bit of effort that, oh, I can
• 37:42
  actually, like, hope be that person that people Elsewhere on
• 37:45
  the company, talk to you about different concepts because turns
• 37:48
  out a lot of people struggle with.
• 37:50
  I am a lot of people struggle with All things AWS.
• 37:55
  It's generally not taught school and so the just having someone
• 38:00
  to talk to helps out a lot. And so that was, that was a big
• 38:03
  part of it. Another big part of it was that
• 38:06
  I was friends with with someone that was that was more connected
• 38:09
  in the devops world. And so I had just been was like,
• 38:13
  oh, what's this state of the of Officer for thing?
• 38:15
  You know, they would share that every year.
• 38:18
  I started reaching out I was like, oh, actually there's some
• 38:20
  interesting stuff there. It's really, you know, About how
• 38:22
  do we have high productivity teams?
• 38:25
  And at the time, I was also reading all these management
• 38:27
  books on like having a productivity and so, you know,
• 38:31
  through a lot of twists and turns, I just kind of kind of
• 38:34
  stumbled into some of it, both on the technical side and then
• 38:37
  more on the cultural side as well.
• 38:40
  Yeah. I mean you threw it out there so
• 38:44
  I can't help but ask, but did you find any High productivity
• 38:48
  team tips that you liked? And is there a caveat when
• 38:54
  dealing with machine learning to keep things really simple?
• 38:58
  I would say do retrospectives periodically and what I mean by
• 39:02
  a retrospective is, you know, get get as many people together.
• 39:08
  As, as you can, that were involved in a project.
• 39:10
  X where you can have a conversation where you can have
• 39:13
  a candid conversation and say, you know what went well what
• 39:17
  didn't go? Well what could we do better?
• 39:19
  What were the areas where we slow down the most or what we
• 39:22
  struggled or what and really build that habit of reflecting
• 39:28
  on how you can improve and leaning on others and working
• 39:32
  together to uncover that the very simple sort of thing?
• 39:37
  Related to that, I would say a couple of different approaches.
• 39:42
  Look at software development. As sort of a factory.
• 39:46
  I, you know, I think that doesn't it feel good to call it
• 39:50
  that. But and but it just is to try to
• 39:54
  look at it as a unit that's producing software and try to
• 40:00
  look for the bottlenecks. How do you for one?
• 40:04
  How do you, how do you measure sort of your input and output
• 40:08
  Matrix or how do you assess whether it's working?
• 40:12
  Well and then you know, where is Well or not working.
• 40:16
  Well how do you pinpoint where you're limiting factors are and
• 40:20
  then intervene on where your limiting factors are?
• 40:22
  So there were times later on, we're hiring was, are limiting
• 40:25
  factor so that I spent my time on hiring there are times where
• 40:28
  a Tobias was their learning Factor.
• 40:29
  So I spend time on it is obvious.
• 40:31
  And in those moments you have a lot of different options as
• 40:34
  leader, you can spend time there personally, you can, you know,
• 40:38
  contracts on one you could bring in some expertise, there are a
• 40:41
  lot of ways to address that bottleneck but the first step is
• 40:44
  you have to act knowledge, what your limiting factors are.
• 40:49
  So let's go into the idea of what I found.
• 40:54
  Fascinating less. Again, one of the times that we
• 40:56
  talked to you was I was trying to ask you about data scientists
• 41:02
  and if you taught them to code or you taught them coding best
• 41:06
  practices and something that you had mentioned was, it's very
• 41:10
  difficult to force, something like that on to anybody.
• 41:14
  It doesn't have to be. Data scientists.
• 41:16
  Although they are notoriously there, they're known for their
• 41:21
  coding practices and I think a lot of it, they've gotten a bad
• 41:26
  rap. I'm not one of those people that
• 41:29
  likes to talk a lot of shit on data scientist coding practices
• 41:33
  or say that they're lazy and they just don't want to.
• 41:35
  I know, there are a lot of people out there that have been
• 41:38
  scarred and the data scientists who refuses to even learn get
• 41:42
  that, but that's not me. I luckily have not had Those
• 41:45
  experiences you however, leading this team, you had people that
• 41:51
  you knew it would be very beneficial for them to learn
• 41:55
  coding best practices, but you couldn't just give them that or
• 41:59
  you couldn't just mandate that from above because that is a
• 42:02
  whole different way of your working style.
• 42:05
  I think. So, how did you inspire people
• 42:08
  to learn things as opposed to dictate from above?
• 42:13
  I will say there. times when you, you have to have a mandate,
• 42:17
  if you're having a lot of outages, or if you have privacy
• 42:22
  breaches or something like that, there are times when you have
• 42:24
  to, but that said the next some of it goes back to my experience
• 42:30
  in teaching, actually, where I found the when you had a student
• 42:36
  that was was motivated and excited about the material, Like
• 42:41
  you were just a facilitator, you know, you were just you're like
• 42:43
  oh read this article reasons are and they would just sort of soak
• 42:46
  it all up and so you're there to support them, you're also have
• 42:52
  some students that you know we're just completely
• 42:54
  unmotivated it was a class requirement and they didn't want
• 42:57
  to be there. And there were times where I
• 43:00
  felt like there was nothing you could do to to teach you know,
• 43:04
  first that was on motivated and you know the reality is probably
• 43:08
  a lot more nuanced than that. But I think that had similar
• 43:12
  experiences in in managing a team where you're going to get
• 43:18
  the best results when someone's ready to learn something.
• 43:20
  When someone is both motivated and ready for it, and if you
• 43:24
  push really hard, that people just naturally resist being
• 43:30
  pushed. This is just a natural kind of
• 43:32
  instinctual thing. You know, sometimes people are
• 43:37
  very subtle about how to add a push, but and sometimes I can
• 43:40
  work but you know, I think people learn To some extent at
• 43:46
  their own pace and you want to accelerate that pace and you
• 43:49
  kind of work with, right? The way to do that with someone
• 43:53
  that, you know, maybe doesn't have a history of clean code or
• 43:57
  something like that. Is to say like, look, all right,
• 44:00
  well, you got to get, get your things shipped, you know.
• 44:03
  Maybe there's some bug and you have retrospective and it turns
• 44:07
  out like, oh, the bug was because of some
• 44:09
  misunderstanding. And you, you ask why a lot and
• 44:12
  you find out? Oh, it's because this part of
• 44:13
  code was really unclear. You say okay, well, we'll how
• 44:16
  can we next time around? How can we make our code more
• 44:19
  clear or something like that or like, hey, you know, I'm happy
• 44:22
  to review some code for you and I'll go a step further.
• 44:25
  And I'll say code review is another really culturally subtle
• 44:31
  type thing I've seen cultures of code review that can be very
• 44:36
  aggressive and people are really kind of taking shots at each
• 44:40
  other almost and in that kind of a culture, people start to
• 44:44
  become too. An serve and then they slow down
• 44:47
  and will never submit something for review until it's perfect.
• 44:50
  Yeah. And that's, you want a culture
• 44:53
  where people are looking out for each other and for code review,
• 44:57
  has a mixture of, you know, protecting your users and also
• 45:02
  kind of leveling each other up and getting the tone of that
• 45:06
  right? Is is kind of tricky, everyone's
• 45:09
  different on a case-by-case basis.
• 45:11
  But if you build a team where everyone is trying to level each
• 45:14
  other up, And you have a mixture of skills over time, it all just
• 45:19
  kind of works out. So the hard work goes into, how
• 45:22
  do you build a culture where you've got people with defects
• 45:25
  for choosing a software engineering and people with deep
• 45:27
  expertise, using machine learning and deep, expertise and
• 45:30
  say 80s. They're all talking to each
• 45:32
  other and leveling each other. If you can develop that culture
• 45:37
  then you'll get there. Maybe you won't get there in one
• 45:40
  month but you'll get there. And so, you know, I think you
• 45:45
  just got to work with the people that you have and build a
• 45:49
  culture where you're going to get there, maybe not on a
• 45:53
  schedules but you're going to get there.
• 45:56
  And when you're seeing a little bit more aggressivity, is that a
• 46:02
  word? I don't know.
• 46:02
  But aggressiveness, then you would like in code reviews or
• 46:08
  GitHub, are you pulling people aside in there in your
• 46:11
  one-on-ones with them and say like, hey, what's going on here?
• 46:14
  Is everything all right. Yeah.
• 46:15
  You know I would say Sometimes. Yeah, if I was seeing that then
• 46:21
  then sometimes I'll do that or, you know, if it, if it's one
• 46:24
  person, then yeah. If it's multiple people, then
• 46:27
  that's a sign that something else is wrong, right?
• 46:30
  You have to decide whether is this a systemic problem or one
• 46:33
  person problem because if it's a one-person problem it's kind of
• 46:37
  it's probably wrong to make a policy about it.
• 46:38
  Probably just talk to that person.
• 46:40
  If it's a systemic problem, you probably have something else
• 46:44
  going on. Maybe policy will help may be
• 46:46
  coaching, you know, large group coaching Jing will help all
• 46:49
  different kinds of things. One thing you can do is lead by
• 46:53
  example, as much if you have time, that's the sariang.
• 46:56
  But that's that's how I like to do.
• 46:58
  It is to go do some code reviews where I try to level people up,
• 47:02
  you know, maybe I can, you know, do one a week or something.
• 47:07
  Another thing is is even a retrospective doesn't have to be
• 47:11
  formal but I like that way of thinking.
• 47:14
  So someone could say oh you know, this code review is just
• 47:17
  it's been Online. You know, it's been it's been a
• 47:20
  couple weeks or something and you start to say well, you know,
• 47:23
  what's going on, you know, and you look at it, it's like, you
• 47:26
  know, 5,000 line change and you're like, well, I mean, who's
• 47:32
  got time to read, five thousand lines of code and detail?
• 47:34
  And you start to say, okay, well, this is the situation,
• 47:37
  this is the reality. We have this 5,000 line change
• 47:39
  right now, and how can we make the BET The Best of that?
• 47:42
  But the pr process on GitHub is not ideal for that, kind of
• 47:47
  review, and So that's an example where we learn from that and we
• 47:50
  said, okay, how can we, how can you take that mindset of we're
• 47:56
  all trying to, to watch each other's backs and trying to get
• 47:59
  things done for users and we're all trying to learn and grow and
• 48:03
  we have the situation with a 5000 line PR?
• 48:06
  Well, you know, we say, okay, let's get in a room and talk
• 48:08
  about, you know, the overall price was, let's do maybe even a
• 48:12
  walkthrough right. Like that, maybe that'll take an
• 48:15
  hour, maybe I'll take two hours. It's going to be a lot faster
• 48:18
  than someone just like sitting in a room trying to figure out
• 48:21
  five thousand lines of code. Pray.
• 48:24
  And then maybe after that they can review right in more depth
• 48:27
  but you know, you want to take a step back and kind of reflect on
• 48:31
  things because our standard processes aren't always going to
• 48:35
  work. I want to ask you about this
• 48:39
  migration that you did and how you what was it.
• 48:44
  It was Zero downtime and it was migrating from.
• 48:48
  Yeah, give us that story. Yeah so so For a while.
• 48:54
  We had kind of a model list for our machine learning Services
• 48:57
  because we didn't understand much about the deployment and we
• 49:01
  just kind of put more stuff in there with a classic increasing
• 49:05
  the ram every every time we added a new model kind of thing,
• 49:10
  it was on app tubule which I guess I'm dating myself now, but
• 49:16
  it's kind of like, Heroku is like, the old Heroku, but for
• 49:18
  health care or so this is pretty simple, but it gives you HIPAA
• 49:22
  compliance in a lot of nice things and the company, more and
• 49:27
  more was was just going with like charade.
• 49:30
  Oh yes. And we were having some trouble
• 49:33
  without full to. So sometimes deployments would
• 49:35
  fail for kind of mysterious reasons at the docker registry
• 49:38
  were using for. It would also sometimes fail and
• 49:41
  mysterious region ways. Sometimes we had one shoe where
• 49:45
  it failed in a way that like corrupted, the one Part of the
• 49:50
  thing so we couldn't update anything off of a certain base
• 49:53
  python image on that one. Took us forever.
• 49:56
  It's about foot. So we had a bunch of issues that
• 49:59
  were we were frustrated about. So we're motivated.
• 50:03
  So we said okay well we gotta move this over 280 PS, the two
• 50:06
  options that we knew about worry Cs and and Lambda and the the
• 50:12
  team did a comparison between the two did load test thing.
• 50:15
  They found the Lambda was better suited for our needs.
• 50:18
  And then for starting to go towards deployment to I said,
• 50:23
  like, look, we're going to have to switch over and I know that
• 50:28
  deployments have some, you know, some percentage.
• 50:31
  Deployments fail and just from unanticipated things and it's
• 50:37
  not. Okay.
• 50:38
  For us to say, okay? Well there's a percentage chance
• 50:41
  that we're going to take us off down for a doctors and patients,
• 50:43
  we might not know what to do about it now.
• 50:45
  That's that's totally unacceptable.
• 50:48
  So I said, okay well it has to be the deployment has to happen.
• 50:52
  Was Zero downtime maybe one or two failed requests, okay?
• 50:56
  But not you know, 10 minutes or something.
• 51:01
  And if it goes wrong, you have to be able to you know, revert
• 51:05
  quickly say within a minute. And that's particularly
• 51:10
  challenging because we're going from after all the AWS.
• 51:15
  And we needed to keep the DNS the same.
• 51:20
  The right. So you could switch the DNS from
• 51:24
  pointing to ones from one server to the other, but updating DNS
• 51:28
  records isn't fast enough. And so I don't actually remember
• 51:33
  all of the nitty-gritty details, I think there was, I think, what
• 51:36
  we ended up doing is switching at the NS to a layer of
• 51:39
  indirection that we could update quickly and then we were able to
• 51:42
  switch that essentially pointer much more quickly.
• 51:47
  So we could do an instantaneous switch over or switch back.
• 51:52
  I think the thing that for me that was I was really proud of
• 51:56
  is that the team tested that they could do a deploy.
• 52:00
  And revert with zero downtime, like Anna lower environment
• 52:05
  system actually tested that. And I said, the team depending
• 52:10
  on your test like I was actually going on vacation when this was
• 52:13
  happening so founding on your test like like a good Bots.
• 52:17
  No no. You know, right at the retina
• 52:19
  critical moment. I said, like, look if you feel
• 52:21
  comfortable with it like you can do this to play to prod.
• 52:25
  I trust you guys. No, I think you've done your
• 52:27
  homework here. Make sure don't do it Friday
• 52:30
  night. Of course we do.
• 52:31
  It kind of midday or something, some low volume time that where
• 52:36
  you going to be around to watch, it should.
• 52:39
  And, and there were able to do that, you know, zero downtime
• 52:43
  migration from One Cloud to another while I was on vacation,
• 52:48
  everything went smoothly, nothing went wrong and wasn't
• 52:51
  there something about choosing between Lambda and eks that
• 52:55
  later you realize so Lambda might not have been the best
• 52:59
  choice. Yeah.
• 53:01
  Yeah. So for us it was Lambda and ECS.
• 53:04
  We didn't have actually kubernetes experience at all
• 53:07
  this thang in the org, but So we did a Bake-Off between a to did
• 53:13
  some load testing and actually it's surprisingly even for the
• 53:17
  machine learning workload without any provision currency
• 53:22
  in our load test, we found that that Lambda had lower latency
• 53:26
  which shocked me that, you know, just was list of rising.
• 53:30
  Some of that probably has to do with the way that ECS
• 53:32
  auto-scaling was set up and and such Rebel forces Lambda when we
• 53:37
  had actually migrated to Lambda. What we found was that We would
• 53:40
  periodically have just, you know, cold start latency spikes
• 53:43
  on Lambda, you know, we set up provision concurrency to
• 53:47
  minimize that, but periodically with slabin, maybe get some
• 53:51
  unusual traffic for for a minute or two.
• 53:54
  And I think on reflecting on that because later I tried out
• 54:00
  the load, testing tool, use called Locust.
• 54:02
  And I think what has happened is that we probably used a load
• 54:07
  testing profile and we're doing sort of scaling load but it was
• 54:12
  it was maybe too predictable. I think and Lambda doesn't
• 54:17
  exactly tell you how Auto scales but probably it was simple
• 54:21
  enough for Lambda to gracefully on.
• 54:24
  Scale very nicely where I was. Our real production, traffic was
• 54:28
  not predictable enough for Lambda to scale gracefully.
• 54:32
  So and then that's one where I wish we had met be done more.
• 54:39
  Realistic load, testing to understand to be able to do that
• 54:43
  comparison better to be able to make a better decision ahead of
• 54:46
  time. But I look back at it.
• 54:48
  That's one of those ones where I don't know, it could have gone
• 54:51
  either way. You know, I think it could have
• 54:53
  been And maybe that was maybe if we'd done both, maybe that was
• 54:57
  the better decision, it wasn't too bad.
• 55:00
  It just had Some surprises. You know, I thought we had
• 55:03
  assessed cold, start problems, sufficiently and we hadn't
• 55:07
  looked into it as much as as much as maybe we could have.
• 55:12
  Yeah, you had a vacation to get to man, no problem there.
• 55:17
  Well I think as you do you schedule a break and then you
• 55:20
  know, something delays, something else and you know,
• 55:24
  something some window that you saw, you're going to be around
• 55:26
  for it does and you're like I don't know sometimes with
• 55:29
  scheduling your just like you can't win with scheduling
• 55:31
  sometimes right? Yeah well dude Keith thank you
• 55:36
  so much for coming on here. I think this is perfect place to
• 55:38
  end it. I would love to chat with you
• 55:41
  for more and more Hours. But as you know, I got to go run
• 55:44
  and grab my daughter from school.
• 55:46
  So this has been fun, man, I appreciate you coming on here
• 55:49
  and chatting with you. This it's been great and thanks
• 55:50
  for having me. And let's shout again, more in
• 55:53
  the future is Skyler. I lead machine learning and
• 56:00
  health rhythms if you want to stay on top of everything
• 56:03
  happening in ml. Ops subscribe to this podcast
• 56:07
  now. Now, now now now, now now Now.