Latest / Agentic Conversations (formally mlops.community) / All Data Scientists Should Learn Software Engineering Principles // Catherine Nelson // #245

Transcript

All Data Scientists Should Learn Software Engineering Principles // Catherine Nelson // #245 · Agentic Conversations (formally mlops.community)

• 0:01
  I'm Catherine Nelson. I'm the author of Software
• 0:04
  Engineering for Data Scientists and freelance Data Science Was
• 0:09
  and I make my coffee in a fancy espresso machine and I like a
• 0:14
  latte. What is our MLA's community?
• 0:17
  We are back with another podcast episode.
• 0:19
  I'm your host Demetrios, and today we're talking with
• 0:22
  Catherine all about software engineering for data scientists.
• 0:25
  This is an episode where we break down what is going on when
• 0:30
  it comes to data scientists levelling up and understanding
• 0:34
  how systems work, not just their little piece of the puzzle.
• 0:39
  So if you are a data scientist and you're looking to expand
• 0:43
  your skills, this is a great one for you.
• 0:45
  And in addition to this, I would highly recommend that you get
• 0:48
  Catherine's book that just came out with the title Software
• 0:52
  Engineering for Data Scientist. And if you're not a data
• 0:55
  scientist and you're listening to this, hopefully it will give
• 0:58
  you a little more empathy for the data scientists out there
• 1:00
  because lots been put on their back.
• 1:03
  One thing that I was thinking about after we got off this call
• 1:06
  and after I talked with Catherine, is just how vague a
• 1:10
  data scientist job description is and their roles in one
• 1:15
  company it could mean one thing and another company it.
• 1:18
  The tasks that you are expected to undertake can be completely
• 1:23
  different. And because of that, because
• 1:25
  there's not that standardised understanding of what a data
• 1:30
  scientist actually does and should know and should be able
• 1:34
  to accomplish. Well, get a bit of everything.
• 1:39
  And so you may have that data scientist that knows how to
• 1:44
  write in YAML or that is able to take ownership of the model and
• 1:50
  the model they create, they put it into production.
• 1:54
  They don't just throw it over the wall.
• 1:55
  Or you might have the complete opposite where there's a data
• 1:57
  scientist who doesn't understand git.
• 1:59
  So this is an episode to go into the cultural aspect of that.
• 2:06
  And if you are data scientist, give you a bit of direction on
• 2:11
  what you should be known and also why you don't know these
• 2:15
  things. Let's get into it with
• 2:17
  Catherine. And as always, if you like, this
• 2:21
  episode would mean the world to me if you can share it with just
• 2:25
  one friend and get Catherine's book.
• 2:28
  If links in the description talk to you later, I have to mention
• 2:39
  to start your first book because I think you wrote it with a
• 2:43
  friend. Honest, right?
• 2:45
  That's correct. Yeah.
• 2:46
  Yeah. That was my entry into the world
• 2:49
  of writing books. And yeah, I was really glad to
• 2:53
  have that opportunity to write with Hannah.
• 2:55
  That was a tonne of fun and yeah, really enjoyed that.
• 3:00
  Yeah. And so the other thing that I
• 3:03
  wanted to mention before we really get into the meat and
• 3:05
  bones of software engineering for data scientists and that
• 3:09
  whole discussion is the idea of meeting in person more often.
• 3:14
  And I know that you just got back from ICON.
• 3:17
  Yeah, that's right. Yeah.
• 3:19
  How was it? It reminded me how good meeting
• 3:23
  in person is. It was because it's not the
• 3:27
  thing. It's not the meetings that you
• 3:29
  expect to happen. That other really about really
• 3:35
  made that special and worthwhile.
• 3:37
  It was the unexpected things like the dinner with an old
• 3:41
  friend where we just discussed anything and everything and all
• 3:46
  our thoughts on the rise of AI and what we were doing in our
• 3:50
  careers and all these kinds of things.
• 3:53
  And you just don't get that kind of thing when you have to pre
• 3:57
  plan meetings. So yeah, honestly, it felt
• 4:03
  intimidating going to a big conference, Thousands of people.
• 4:10
  I work remote. I don't spend time in places
• 4:14
  where there's thousands of people, but it was good to
• 4:15
  remember just how worthwhile it is.
• 4:19
  So I had almost the same situation happened because I
• 4:23
  just got back from the data and AI Summit and it's so funny.
• 4:29
  You cross paths with people and you realise, hey, I didn't know
• 4:34
  you were going to be here. Ohh yeah, You know, And then all
• 4:37
  of a sudden some people that I've only been talking to online
• 4:40
  for the past four years, now I get to see them in person and we
• 4:45
  have coffee or we have dinner or we have lunch, whatever it may
• 4:48
  be. And you can't get that if it's
• 4:51
  like, hey, are you free? Yeah, send over my calendly link
• 4:54
  and then it's blocked off. It's like those serendipitous
• 4:59
  moments. It's very hard to get virtually,
• 5:02
  yes. And so I feel you 100%.
• 5:06
  And that's also, I mean, we're doing the AI quality conference.
• 5:09
  I am thinking about that so much right now is how to make the
• 5:13
  attendee experience just top notch and provide for those
• 5:18
  serendipitous experiences. Because I imagine there's going
• 5:21
  to be a lot of people that are going to know others at the
• 5:25
  event, but they don't realise that the other person is going
• 5:28
  until they get there and they're waiting in line for coffee or to
• 5:31
  register with them. And then it's like, hey, let's
• 5:33
  hang out. And so how do you have all of
• 5:37
  the cool talks, but also how do you have the spaces for people
• 5:40
  to hang out and have those moments?
• 5:43
  Well, I have one suggestion for you which which I heard at a
• 5:46
  conference a while ago, and they actually announced in the sort
• 5:51
  of intro keynote talk, but if you were standing in a circle
• 5:56
  with a bunch of people talking to each other, they encouraged
• 5:59
  you to leave a gap in that circle if someone knew you could
• 6:03
  join in. Like if you don't know a bunch
• 6:06
  of people to get started with and there's a bunch of people
• 6:09
  hanging around talking, it feels intimidating to like step into
• 6:12
  that. But if you leave the gap to
• 6:14
  encourage new people to join, that just makes more space for
• 6:19
  those serendipitous get togethers.
• 6:21
  Yes, it's much more welcoming. I know we didn't come here to
• 6:24
  just talk about in person tech conferences all day.
• 6:29
  I really want to know about your new book which is all around
• 6:36
  software engineering for data scientists, right?
• 6:38
  Yep, that's, that's the title. That's what it's all about,
• 6:41
  yeah. Yep, you you can't get anymore
• 6:44
  clear than that. I think.
• 6:46
  It is a topic that has come up so many times in the last four
• 6:51
  years doing this podcast. And whether it is like, I think
• 6:56
  one of the huge champions for this is Matt Sharp talking about
• 7:03
  clean code for data scientists or we have Lazlo in the
• 7:07
  community who is gung ho about like production code and what
• 7:12
  you need to know if you're data scientists.
• 7:14
  And so there's a lot of threads we can pull on.
• 7:18
  Let's just start like, why did you feel inspired to write this
• 7:22
  book? Yeah, so it's something that
• 7:24
  gets so much bad press. You see all these Reddit threads
• 7:28
  about like, data scientists write terrible code.
• 7:32
  What is going on? I'm ashamed of birth, but then
• 7:37
  my experience when I was early in my data science career, I was
• 7:42
  the only data science on a team of designers and engineers and
• 7:48
  product people. I nothing in my background in my
• 7:53
  the courses I've taken had prepared me for that.
• 7:58
  They'd given me the language to work on a software team.
• 8:02
  So I had questions like what is an API?
• 8:07
  What is the test? How do I write a test?
• 8:10
  And I wanted to learn more about this and I started looking at.
• 8:15
  I asked people for recommendations for books or
• 8:18
  whatever, and the the books and the examples, they were all
• 8:23
  aimed at web developers or the examples were in Java or similar
• 8:28
  things. That it wasn't.
• 8:31
  There's a whole load of extra knowledge that I needed to be
• 8:35
  able to get all the good stuff about what is clean code that
• 8:39
  weren't that wasn't really useful to me as a data
• 8:42
  scientist. It wasn't approachable because
• 8:46
  all of a sudden now you're like, wait, C.
• 8:49
  What? Is that Java?
• 8:51
  Whatever. Yeah, maybe.
• 8:52
  Do I need to like learn this language well enough to
• 8:55
  understand the examples so you learn what it should be?
• 8:59
  Ohh yeah. So that can be very painful.
• 9:03
  And then later on I had a mentees who I saw having exactly
• 9:09
  the same problem. So round about middle of 2021, I
• 9:13
  was just thinking this should be a book.
• 9:16
  I didn't think I was going to write it, but I was like this,
• 9:19
  this really shouldn't exist. There should be a guy that's
• 9:23
  aimed at people who are writing code in Python, who are not
• 9:28
  particularly developing for the web, but they want to write good
• 9:30
  code. Because as data scientists, we
• 9:35
  don't want to write bad code. We're not writing sprawling
• 9:38
  Jupyter notebooks just to be perverse or anything.
• 9:41
  There are good reasons for this, like.
• 9:44
  I thought it was, yeah. I thought they were into S&M and
• 9:47
  that was the reason why. You had it right, And just like
• 9:53
  the exploratory nature of a lot of data science projects does
• 9:58
  lead you to write long scripts and do that quickly, just to
• 10:02
  kind of try out your ideas and go through that process.
• 10:08
  It's useful to explore data in a Jupiter notebook where you can
• 10:11
  see examples of it easily. But then as data science has
• 10:17
  become more mature, then you start to need to put that code
• 10:22
  into production. And that just needs a completely
• 10:25
  different way of thinking about things.
• 10:27
  Yeah. So actually the bar, it becomes
• 10:30
  a lot higher there. They designs because you have to
• 10:32
  know how to explore your ideas quickly, but then also turn it
• 10:37
  into solid, robust production code.
• 10:40
  Yeah. So it's funny, you mentioned the
• 10:43
  idea around wanting to explore as fast as possible.
• 10:46
  And basically there the code is the least of your worries.
• 10:51
  Yeah. So you're just trying to see
• 10:52
  like, hey, is this assumption that I have actually valid?
• 10:55
  Is there something there? And you're figuring out where
• 11:00
  you're going to go with things and what you're going to invest
• 11:02
  time into. And so the last thing that you
• 11:06
  want to be investing time into is making sure that your code
• 11:10
  looks nice, because that's the least of your worries.
• 11:13
  Yeah, you just gotta hack it together as quickly as possible.
• 11:16
  But what I wanted to equip people with in my book was to
• 11:20
  know the difference between the code that you just hacked
• 11:23
  together as quickly as possible and what good code is.
• 11:29
  Yeah. And take a whole chapter.
• 11:31
  I just to like explore the various ways that could.
• 11:36
  The chapter is called What is Good code?
• 11:38
  And I just explore that idea and why you might want to do this
• 11:43
  because it helps you scale out, but it helps you avoid technical
• 11:47
  debt when you get into production systems and so on.
• 11:50
  Yeah, it does feel like the wise data scientists understands that
• 11:56
  difference and say, no, all right, I'm doing this now just
• 12:00
  to move fast. But once I have something and
• 12:05
  once I'm ready, I'm not going to just say this is OK to send over
• 12:12
  to someone else on the team. I need to take the time and
• 12:16
  really format it or just put put in those hours.
• 12:19
  Yes, to make sure that people understand where I'm coming from
• 12:25
  and it's not just spaghetti code.
• 12:28
  And as you get more experienced, you start to develop a sense for
• 12:33
  when pieces of that code might be useful for another project or
• 12:37
  similar projects, and develop a library of things that you can
• 12:41
  use. And that because you're going to
• 12:44
  run it repeatedly, you want that.
• 12:46
  You want to be sure that works well, you want it to be
• 12:49
  efficient, you want it to be tested, and so on.
• 12:52
  Yeah. So there is a few things that
• 12:55
  you mentioned that you feel like there's a disconnect, right?
• 12:59
  And I've heard it a lot on here where people will say, wow, I've
• 13:04
  worked with a data scientist and they didn't even understand Git.
• 13:08
  And so that was like the that's like the meme, like you were
• 13:11
  saying on Reddit, you hear a lot.
• 13:13
  I've been guilty of it. Yeah.
• 13:15
  And it's so easy to throw data scientists under the bus.
• 13:19
  Sorry for all the listening and out there, but I feel like it's
• 13:24
  getting better. And books like yours and just
• 13:27
  like movements, people talking about it, it's understandable
• 13:31
  now that you shouldn't be doing that or it's not the best.
• 13:36
  Where else do you feel like there is a disconnect because
• 13:39
  you mentioned the APIs you you hadn't understood from all of
• 13:45
  your training to become a data scientist things like APIs or
• 13:50
  like code review, imagine was foreign concepts.
• 13:54
  What else is there that you you're like, OK, let's, let's
• 13:59
  build a bridge into more software engineering practises
• 14:03
  that data scientists can follow. Security was definitely another
• 14:07
  one. That's something that isn't
• 14:10
  talked about in data science courses because you're like,
• 14:13
  ohh, let's go download this data set from the web.
• 14:16
  We don't need to. We don't need to take any care
• 14:19
  with that. We don't need to treat that
• 14:22
  securely. Or the Pypi packages.
• 14:25
  It's like, yeah, we'll just download any of these part by
• 14:28
  packages. Right.
• 14:29
  Yeah, well, just save our models and pickle and outplay them
• 14:33
  everywhere and yeah, ohh. Classic.
• 14:35
  But if you're just dealing with open data, open source software,
• 14:40
  like most of the data science training, then why would you
• 14:45
  know any of any of those aspects of things?
• 14:49
  That's just not relevant. But then suddenly you're working
• 14:53
  in a software team, you're working with real people's real
• 14:57
  data. It becomes really important.
• 15:00
  And are there specific ways that you've found basically just to
• 15:04
  level U? Let's take the security one for
• 15:07
  example, like me not knowing anything to me at least having a
• 15:11
  grip on it. Should I just familiarise myself
• 15:14
  with the terms or should I familiarise myself with like all
• 15:17
  the different ways that there are vulnerabilities?
• 15:19
  Or should I just know don't download open data sets off
• 15:22
  hogging face? In, in my book, the tapped
• 15:26
  around security. It's just like, hey, security is
• 15:29
  a thing you should think about. This is what the risk means.
• 15:32
  This is what the threat means. This is what the vulnerability
• 15:34
  is. Here's where you can go and find
• 15:37
  ones that might be relevant to the work that you're doing.
• 15:40
  And some basic stuff aren't just like physical security and so
• 15:46
  and some details about the consequences of when people
• 15:49
  don't. So I put in some details on like
• 15:52
  high level, high profile security breaches and just how
• 15:56
  big a deal that is. Security for machine learning is
• 15:58
  this whole new thing as well an adversarial attacks on machine
• 16:03
  learning models. That is absolutely something
• 16:06
  that they'd find for some machine learning engineers
• 16:08
  should be aware of and thinking about just the possible methods
• 16:14
  of attack and what they can be doing to mitigate those.
• 16:18
  It was funny, I saw a LinkedIn post.
• 16:21
  You know how LinkedIn now will like prompt you to answer?
• 16:25
  It's like you're an expert and you should answer this question.
• 16:28
  Somebody just I'm pretty sure they're trying to like train the
• 16:31
  model off of all those answers. And I saw something that I
• 16:36
  thought was hilarious because I saw somebody's answer pop up on
• 16:40
  my feed. It had nothing to do with the
• 16:43
  question, and it was obvious, like just data poisoning.
• 16:46
  And so I can only imagine that person.
• 16:50
  'S. Writing a whole lot of answers
• 16:53
  or potentially like just prompting chat TBT to be, hey,
• 16:56
  I'm angry grumpy coder who doesn't like this answer these
• 17:01
  10 questions for me and then they go and answer them.
• 17:03
  And now you have to think about data poisoning.
• 17:06
  So hopefully people also are keeping that and they account,
• 17:10
  but that I would imagine is more in the realm of a data
• 17:13
  scientist, like they understand that much more than they
• 17:15
  understand Kubernetes. Death, definitely, and a lot of
• 17:21
  what you can do about that comes down to careful monitoring and
• 17:25
  analysis of your model and how it's performing.
• 17:27
  So that is absolutely in the realm of the data scientist or
• 17:30
  the machine learning engineer. Yeah, yeah.
• 17:34
  We would always ask, I think in the beginning of the community,
• 17:37
  because when basically 2020 when things were very unclear on who
• 17:45
  owns what, what is a data scientist, what is an ML
• 17:48
  engineer? Not that I'm saying like it's
• 17:50
  super clear now, especially with the rise of an AI engineer and
• 17:53
  what's the difference between that and a mill engineer?
• 17:56
  But I remember we would constantly kind of be asking one
• 18:01
  another, who gets the ping at 3:00 AM when the model's gone
• 18:05
  rogue, who's owning it? What is the ownership look like?
• 18:10
  And nine times out of 10, it's not the data scientist because
• 18:13
  they don't they're not able to be like, ohh, rollback.
• 18:17
  And. They.
• 18:18
  Can ones? They're the ones taking the big
• 18:20
  picture of you rather than the like, immediate production
• 18:24
  monitoring, I should think. Exactly.
• 18:27
  And so that kind that that dovetails nicely into the next
• 18:32
  question that I wanted to ask was, yeah, from data scientists
• 18:36
  to machine learning engineer, If we loosely just describe like a
• 18:42
  data scientist being someone who is very much about the modelling
• 18:47
  and playing in there almost like statistics and analysts and
• 18:53
  creating models, but less on the infrastructure side.
• 18:57
  And the mill engineer being someone who maybe is more on the
• 19:01
  platform side supporting the data scientists and less on the
• 19:06
  modelling and definitely less in the Jupiter notebooks realm of
• 19:10
  things. How have you seen people
• 19:14
  successfully transitioned from a data scientist to a machine
• 19:18
  learning engineer? I think the biggest thing is a
• 19:24
  change of mindset because you're moving from someone who's
• 19:28
  focused on exploration and uncertainty and oh, I've got
• 19:34
  this big problem to solve, but I don't know how I'm going to do
• 19:37
  it. And I'm completely happy with
• 19:40
  that and exploring that space to someone who's saying I'm gonna
• 19:45
  make this run repeatedly, I'm gonna make this scale up, I'm
• 19:49
  gonna make sure it's running well.
• 19:52
  So you moving from that very open and exploratory type of a
• 19:58
  job to something where you're very focused on standardisation
• 20:03
  and making sure that everything is efficient and robust and
• 20:10
  tested. So you might still be running
• 20:14
  some experiments, but it's in a, it's in a narrower field.
• 20:19
  So yeah, I think that's the that's the main thing.
• 20:23
  That's what I've seen people do. And a lot of the ways that
• 20:27
  they've gone about that is more getting really interested in the
• 20:31
  code that you're writing and letting how to write that well
• 20:35
  so that you're less focused on the project direction and the
• 20:39
  science and more interested in like, ohh, how am I actually
• 20:42
  doing this task? How do I do that efficiently?
• 20:47
  And thinking about how to scale things up.
• 20:50
  Thinking how to make each piece more efficient and fit together
• 20:54
  better. Yeah, It does feel like it's
• 20:57
  going from the ad hoc nature to the reliability piece and saying
• 21:05
  we've done this once and now we're going to do this N plus
• 21:10
  one. Yes, yeah.
• 21:13
  So you have to figure out how to do it and scalability, all these
• 21:17
  questions that are kind of for lack of creativity.
• 21:25
  It's what the software engineers bread and butter is, right?
• 21:28
  And so it is more I do understand the idea of changing
• 21:32
  the mentality from being that exploratory nature to being more
• 21:37
  software engineering focused. Where you've kind of established
• 21:41
  that The thing is possible and that you know you're gonna do
• 21:44
  it. You have a good idea of what the
• 21:46
  outcome of the project is going to be at the start, and you know
• 21:50
  that you need to build that. Mm-hmm.
• 21:53
  And so have you seen those POC's be hacked together and just
• 22:01
  really try and move as fast as possible to get something out
• 22:07
  and then we can iterate on making it better?
• 22:11
  Is that how you generally would go about the projects or do you
• 22:15
  still want to focus on making sure things are tip top before
• 22:20
  they get pushed out, even if it is going to take longer?
• 22:26
  It's going to be a balance, right?
• 22:27
  And this, this is something that I feel is not particularly,
• 22:32
  there's not a good established process for moving from that POC
• 22:38
  to something in production. There's always, there's been
• 22:43
  talk for a long time of do you put notebooks in production?
• 22:48
  So you try and find some way of just taking that exploratory
• 22:52
  code and justice running with it?
• 22:55
  Or do you sit down and do a big refactor into something that is
• 22:59
  more usable by the rest of the production system?
• 23:03
  And I'm firmly in the second camp.
• 23:05
  I think taking that messy code and then refactoring it is
• 23:10
  absolutely the way to go. But that requires a lot of
• 23:15
  skills that might not be the bad and butter that data scientist.
• 23:20
  And it's something that I don't really, I don't often see as
• 23:23
  baked into like the project lifecycle data science project.
• 23:28
  You kind of need to set that time aside to do that instead.
• 23:33
  How many story points? Right, quite a few actually.
• 23:39
  Yeah, exactly that. That is so funny that you
• 23:44
  mentioned it and it is almost like with your book, data
• 23:50
  scientists don't have the excuse to just throw it over to
• 23:53
  somebody else. Because now hopefully any
• 23:57
  software engineers or data engineers or anybody out there
• 24:00
  working with a data scientist who says, hey, well, I don't
• 24:03
  know how to make this better out, then just let us know and
• 24:08
  we can send them a copy of your book and they can read it and
• 24:14
  have a few story points for that one.
• 24:18
  I mean, there might be situations where it is a good
• 24:20
  idea to hand that code over if you have some like particular
• 24:25
  requirements if the data scientist needs to move on to
• 24:28
  another project. But then that kind of that
• 24:33
  causes, that means there's a gap between the intention of what
• 24:37
  the data scientist is aiming for that model.
• 24:39
  You don't. You lose that insight into why
• 24:43
  that particular model was chosen if that if they're no longer
• 24:49
  responsible for that code. So it makes it a little harder
• 24:54
  to then update the model in the future.
• 24:57
  If updating the model needs the data scientist to go back to
• 25:01
  their exploratory code and then train a new model and then had
• 25:06
  that over and have someone else write the code to actually put
• 25:10
  that new model into production. That doesn't seem like a smooth
• 25:15
  workflow to me. No, yeah, that's not scalable.
• 25:20
  What is a better way? I think if the data scientist is
• 25:22
  able to own that model to a certain extent, it's helpful.
• 25:28
  Um, I don't want the data scientist to end up doing
• 25:32
  everything. Uh-huh.
• 25:34
  But at least being able to work together with the software
• 25:38
  engineer on that production code so that they have an
• 25:42
  understanding of what's gone into it, being able to
• 25:46
  communicate better rather than just throwing it over the wall
• 25:50
  and thinking their jobs down. Well, because the pushback that
• 25:54
  I've heard from quite a few people is that you wanna have
• 26:00
  everyone working in their zone of greatness.
• 26:04
  And so if you do have a data scientist whose zone of
• 26:07
  greatness is more that exploratory and being able to
• 26:11
  spot trends or anomalies or whatever it may be, understand
• 26:15
  the data and have a very intimate relationship with that
• 26:20
  data, that's where their zone of greatness is.
• 26:22
  And then any moment that they're not doing that, you're really
• 26:25
  missing out on someone that you're paying a lot of money to
• 26:29
  do that. That thanks a lot of sense,
• 26:32
  yeah. Have you thought about that
• 26:34
  other side of it and what the, I guess what the pushback would be
• 26:41
  if that was it? Because I do, I intuitively I
• 26:45
  know what you're saying is a better route because you don't
• 26:51
  have these hang ups. But I I also see the side where
• 26:55
  you have that problem when people are saying, well, you
• 26:58
  know, I'm good at this. I'm not good at refactoring
• 27:01
  code, or I'm not good at all this other stuff that comes
• 27:04
  along with it. If I'm if I was a data scientist
• 27:09
  who's specialisation was in that his own of greatness was the
• 27:14
  exploration, I would still want to be able to read and
• 27:18
  understand the production code that is based on that work.
• 27:22
  So I think that's what I'm aiming.
• 27:25
  That's one of the things I'm aiming for with my book, is to
• 27:28
  equip people with the language and the terminology so that they
• 27:35
  can understand more complex code, even if they're not
• 27:40
  writing it themselves. O What are some of these keys to
• 27:44
  being able to work with software engineers or data engineers or
• 27:49
  just data scientists? If you're the lone data
• 27:51
  scientist on a team, how have you seen that be successful?
• 27:56
  I think one of the big things is just understanding the
• 28:00
  incentives of the software engineers, and this is common to
• 28:04
  anyone in a different discipline that you're going to work with.
• 28:06
  Like what is their background? What makes them tick?
• 28:12
  What are they trying to achieve? And hopefully they do the same
• 28:16
  with you so that, you know, like we're all working on this
• 28:19
  towards this one thing in a different ways.
• 28:23
  But yeah, understanding how important things like
• 28:27
  standardisation are to software engineering and why that might
• 28:30
  be important because that's what makes code easier to maintain.
• 28:36
  That's what makes it easier to get started on someone else's
• 28:40
  code. All these things and more
• 28:44
  specifically if I'm thinking about trying to understand more
• 28:47
  complex code then things, having a having some knowledge of
• 28:53
  things like object oriented programming, what that is, what
• 28:57
  that looks like and say to tests look like, What is a unit test?
• 29:02
  What is in integration test so that you just have some better
• 29:06
  expectations around what a bigger code base might look
• 29:10
  like? Bad understanding the CI CD
• 29:14
  pipeline, understanding how to push things to get that makes a
• 29:17
  lot of sense. What even is the ICD?
• 29:21
  Yeah, correct. Yeah.
• 29:23
  Yeah, and you would hope that most people understand it, or at
• 29:28
  least these days are aware of the the different terms.
• 29:34
  But sadly, I guess it's not always the case.
• 29:39
  No. And I think even if you're
• 29:41
  coming from a peer, even if you're coming from like a
• 29:45
  computer science degree, you might not, you might not get
• 29:49
  that exposure. You're into the theoretical side
• 29:51
  of things. And I'm thinking about the the
• 29:55
  Genia data scientist, Genia avalanche there who is like
• 30:00
  fresh out for boot camp, fresh out of a masters degree.
• 30:06
  They're probably just not going to have had exposure to a lot of
• 30:09
  these terms. And it it only makes sense once
• 30:13
  you. Once you have a basic idea of
• 30:16
  what's going on, then you can start to read the documentation.
• 30:19
  But there's this period where your you don't, even though
• 30:24
  enough to learn more from the more complex documentation.
• 30:29
  Yeah, it's funny because I was going to mention even just
• 30:32
  seeing some. Well, no, let me take that back.
• 30:38
  The junior data scientist who has only played on Kaggle and
• 30:43
  only used the Titanic data sets, and now they're thrown into a
• 30:48
  job where, yeah, you've got I to worry about.
• 30:51
  You've also, you've got all the, like you said, like maybe you're
• 30:55
  working with a UX designer, maybe you're working with data
• 30:59
  engineers or software engineers, DeVos folks.
• 31:02
  And they all have to be conscientious about different
• 31:09
  parts of the system. Yeah.
• 31:11
  And so they're going to be coming to you asking you about
• 31:13
  certain parts of the system. And if you have that common
• 31:15
  language and that common understanding of what things
• 31:18
  are, you're going to be able to go much further.
• 31:21
  And ideally, you'd have a great mentor in your team who would be
• 31:25
  able to explain these things to you and change all your
• 31:28
  questions. Yeah.
• 31:29
  Thank you. True.
• 31:32
  The amazing prompt there is, yeah, you're a software
• 31:36
  engineering mentor and I have lots of questions for you.
• 31:40
  Ohh, that's yeah. But even understanding how to
• 31:45
  help you with the code and how to help you think about things
• 31:49
  differently and look at the system design documents, and so
• 31:55
  I understand the desire for different data scientists to
• 32:01
  want to be able to understand the system as a whole as opposed
• 32:06
  to just their little piece. You don't need to be an expert
• 32:10
  on the whole, but just knowing what's there is I think is key.
• 32:16
  Yeah. And knowing the different, I
• 32:18
  find it super helpful too, just to know the different tools that
• 32:24
  are out there and what they do. Like, yes, me learning even
• 32:27
  though I'm not, it's not like it's my bread and butter, but I
• 32:31
  do understand what Kafka does, right?
• 32:34
  Or I do understand, OK, what's the difference between why do I
• 32:38
  always see Kafka and Flink together?
• 32:41
  And now I'm seeing, I'm starting to see lots of design diagrams
• 32:44
  with Apache Arrow. What does that do?
• 32:47
  And the more that you can get into those types of things and
• 32:52
  and that's probably a lot of reading of different engineering
• 32:55
  blogs that will help you see like, OK, what design principles
• 32:58
  are they taking and what are they actually doing?
• 33:01
  What's the design decisions they made and the systems that they
• 33:05
  created? That can help a tonne too.
• 33:09
  Yeah, definitely. Yeah.
• 33:11
  I think just reading around what's going on, what's, what's
• 33:17
  the new trends are, has been something that's been really
• 33:20
  helpful to me by all career. When you think about data
• 33:23
  scientists like interacting with larger code bases.
• 33:28
  How can this be done well and not pushing problems?
• 33:35
  Hopefully there's redundancies in place so that nothing can
• 33:38
  actually escape, but you can imagine that someone who is
• 33:42
  fairly new at this can mess up a few times.
• 33:49
  Yeah, I think for starters, like adding software system, you want
• 33:55
  to be really clear about what the requirements are for that
• 33:58
  code and like how is it going to interface with the rest of the
• 34:03
  system? So designing that well from the
• 34:06
  start and making sure that that's clear to the data science
• 34:09
  team who's working on their bit code, it's going to help a lot.
• 34:14
  And so if the system needs to return an answer in some
• 34:18
  specific amount of time, then make that clear and make that
• 34:22
  something that needs to make that something that's baked into
• 34:26
  the requirements. And then, yeah, if you got
• 34:30
  incentives like this, then that puts the, you know, that that
• 34:37
  makes people want to build that system so that it fits those
• 34:43
  requirements. So then you start needing to
• 34:46
  make each piece of code run well so that the whole system runs
• 34:54
  well, and also setting up tooling that works with data
• 35:01
  scientists rather than against them.
• 35:04
  O when you talk about that, it reminds me of, it's just when I
• 35:10
  was at the Data and AI Summit, there is a company whose whole
• 35:15
  thing is built around making sure you have the right
• 35:19
  permissions in place for all the data.
• 35:22
  And I was like, wow, this makes a lot of sense because I've
• 35:25
  heard horror stories of people who join a company and then
• 35:29
  three to six months later, they still don't have access to the
• 35:32
  right data to get their models running or they're getting,
• 35:36
  they're getting dummy data and it looks nothing like what is
• 35:39
  actually in production. And so any model that they try
• 35:42
  and push is failing horribly and.
• 35:46
  Yes, it was like, ohh, this, yeah.
• 35:49
  Super cool to think about as a as that, but I just wanted to
• 35:53
  interject on what you were saying with the provisions
• 35:56
  provisioning access and the the other piece to that when it
• 36:02
  comes to like the data scientist working with larger code bases,
• 36:11
  I think the biggest question that we had over the span of two
• 36:19
  to three years basically before transcript was released and
• 36:23
  everybody's attention shifted the constant discussions that
• 36:28
  would come up probably once every other month in the MO
• 36:31
  community, we're putting Jupiter notebooks into production.
• 36:34
  And then the other side was should a data scientist learn
• 36:37
  Kubernetes? And so I'm wondering from your
• 36:42
  side of things, how far is like too far?
• 36:47
  Well, First off, maybe, and I've seen it a lot, there's a lot of
• 36:52
  people that have come through the community and there's one
• 36:55
  that is absolutely hilarious. Somebody introduced themselves
• 36:58
  in the MLS community, Slack, and they said, I used to be a happy
• 37:03
  data scientist and then I had to push my code to production.
• 37:06
  And now I'm like having to learn Kubernetes and I'm having to
• 37:10
  learn all this stuff because they didn't have the support
• 37:12
  from a team and the software team was like not able to give
• 37:18
  them what they needed. So you do hear those stories of
• 37:23
  when a data scientist will level up and learn Kubernetes.
• 37:26
  I think for 90% of the data scientists, it's way too
• 37:30
  cumbersome to ask that of them. And so I guess the the question
• 37:37
  I have is when is it taking it too far?
• 37:41
  Yeah, Yeah. That's a great point because I
• 37:44
  can imagine that if you're responsible for everything to do
• 37:50
  with the model and that includes the Kubernetes side of things,
• 37:54
  even just the Kubernetes piece could be a full time job.
• 37:59
  It makes me wonder why there isn't a team supporting that
• 38:03
  model if you're taking it that far.
• 38:08
  But then if there isn't the need to be exploring new models so
• 38:14
  much, then that might be the way that you end up going.
• 38:20
  It's a it's a real problem with just data science being this
• 38:23
  very vague job description that could mean pretty much anything.
• 38:30
  Yes, yes, the data. I remember I was talking to
• 38:34
  someone in Amsterdam where we had a meet up and they were
• 38:37
  saying how they were hiring a data scientist and I was like,
• 38:41
  oh, So what does a data scientist mean to you?
• 38:43
  And they said, you know, like they can do anything and
• 38:47
  everything from data pipeline to CID to Kubernetes to monitoring
• 38:53
  it and working with data dog. And I'm like, ohh, so you're
• 38:57
  looking for a Unicorn. You're not looking for a data
• 38:59
  scientist. They're like, yeah, but unicorns
• 39:01
  exist. I know, I I've done it.
• 39:03
  And I'm like, man, that is just really hard to expect of
• 39:08
  someone. And I guess it's more, it's more
• 39:14
  common these days and you've probably undoubtedly heard about
• 39:19
  like the T shape where you can be understanding the whole life
• 39:24
  cycle, but you go deep on one aspect.
• 39:26
  Of it, yeah, absolutely. It is unfair to expect someone
• 39:30
  to learn all these things. But I can also see how it
• 39:32
  happens because data scientists. Data science does attract people
• 39:37
  who like to learn new things. So I can see that once people
• 39:44
  get into the data science job, they're like, oh, I need to
• 39:47
  learn this thing to make that happen.
• 39:49
  I need to learn this, and they are able to learn all these new
• 39:54
  things when they're interested in them, and then suddenly that
• 39:57
  sets a precedent for being able to do everything.
• 40:01
  Totally. It's like, OK, you're analytics
• 40:04
  engineer, your data engineer, your DevOps, your data
• 40:08
  scientists, machine learning engineers all wrapped up into
• 40:11
  one. So yes, hopefully you're getting
• 40:15
  paid like two people, not one, because with all that knowledge
• 40:21
  that is incredible, but. And if you're having a good time
• 40:24
  learning all these new things and you're excited about it,
• 40:27
  that that's fine. Yeah, yeah.
• 40:30
  And then one of our one of the original Co hosts of this
• 40:34
  podcast, David Aponte, he started as a data scientist and
• 40:37
  then he moved into machine learning engineering and he
• 40:40
  started learning Kubernetes. And then he really enjoyed that
• 40:43
  for a while he was doing more like DevOps as RE stuff.
• 40:46
  And then he started going a little bit further down the
• 40:49
  stack. And now he's just optimising
• 40:52
  GPUs. And he's very much like he's
• 40:56
  doing the research side of machine learning, but to make
• 40:59
  sure that the these LM's or just foundational models can work
• 41:05
  fast on whatever GPU or CPU is coming out-of-the-box with the
• 41:11
  computer. And so it's, it's like what you
• 41:15
  just said, since he enjoyed learning stuff, he just kind of
• 41:20
  kept going and kept learning. And next thing you know, he's
• 41:23
  got a PhD. And he is, yeah, way beyond
• 41:29
  whatever the ever expected, right?
• 41:31
  But that's how it works. Alright, real quick, let's talk
• 41:34
  for a minute about our sponsors of this episode Making It All
• 41:38
  happen, Lattice Flow AI. Are you grappling with stagnant
• 41:42
  model performance? Gartner reveals a staggering
• 41:45
  statistic that 85% of models never make it into production.
• 41:48
  Why? Well, reasons can include poor
• 41:51
  data quality, labelling issues, overfitting, underfitting and
• 41:55
  more. But the real challenge eyes and
• 41:57
  uncovering blind spots that lurk around until models hit
• 42:01
  production. Even with an impressive
• 42:03
  aggregate performance of 90%, models can plateau.
• 42:07
  Sadly, many companies optimise for prioritising model
• 42:12
  performance for perfect scenarios while leaving safety
• 42:16
  as an afterthought. Introducing Lattice Flow AI, the
• 42:20
  pioneer in delivering robust and reliable AI models at scale.
• 42:25
  They are here to help you mitigate these risks head on
• 42:28
  during the AI development stage, preventing any unwanted
• 42:31
  surprises in the real world. Their platform empowers your
• 42:35
  data scientists and ML engineers to systematically pinpoint and
• 42:38
  rectify data and model errors, enhancing predictive performance
• 42:43
  at scale. With Lattice Flow AI, you can
• 42:46
  accelerate time to production with a reliable and trustworthy
• 42:49
  models at scale. Don't let your model stall.
• 42:54
  Visit Lattice Flow Dot AI and book a call with the folks over
• 42:58
  there right now. Let them know you heard about it
• 43:01
  from the MO OPS Community podcast.
• 43:02
  Let's get back into the show. So on that note, I think there's
• 43:10
  probably an interesting topic to cover, which is the future of
• 43:16
  data science and how you feel things are moving, Especially
• 43:21
  because no, a lot of people are just able to grab a model off
• 43:27
  the shelf, get it running, and it's better than spending six
• 43:31
  months trying to get their their model going, right?
• 43:38
  Yeah. And I was asked this question at
• 43:42
  a conference I did a talk at a few weeks ago and my answer is
• 43:47
  completely changed from then to now.
• 43:51
  I'll, I'll explain this. So three weeks ago I said that I
• 43:55
  was pretty certain that data science was kind of diverging
• 44:00
  into more analytic side and more machine learning engineering
• 44:04
  side. So you either got interested in
• 44:08
  the statistics side and you went deep in the analytics side or
• 44:13
  you are training models and experimenting with those and
• 44:17
  then maybe moving to deploying the models into production.
• 44:20
  But then I recently started consulting for Jedi I startup
• 44:26
  and started playing around with LMS. And for me this this brings
• 44:32
  an extra dimension because evaluating LM's is hard.
• 44:38
  It's really uncertain how you do this at the moment, but I think
• 44:44
  that working out whether lines are doing the task that you want
• 44:49
  them to be doing is absolutely in the data science set of
• 44:53
  skills, but it needs both statistics knowledge and
• 44:58
  knowledge of the internal workings of the machine learning
• 45:01
  model. So I think it brings both those
• 45:04
  divergent skill sets back together again.
• 45:07
  Ohh fascinating. So the people that are primed to
• 45:11
  take on these kind of hairy problems of evaluating the
• 45:17
  output of the models, our data scientists.
• 45:20
  Yes. That makes 100% sense to me, and
• 45:27
  especially when you start looking at the output as a
• 45:31
  corpus of data as opposed to like 1 to one like is this this
• 45:36
  one answer what we thought was going to come out?
• 45:39
  It's like no. Are these 1500 answers?
• 45:43
  How do we look at those as a data set?
• 45:46
  Yeah, yeah. I was listening to a podcast
• 45:50
  with which interviewed Christopher Manning at Stanford,
• 45:55
  and he made the point really well that machine learning has
• 45:58
  changed from you collect the data set that's aimed at your
• 46:03
  problem and then you train a model and you have this model
• 46:07
  that's suited to that problem. And then it's relatively easy to
• 46:12
  evaluate because one mobile, one problem.
• 46:16
  But now with aliens, you're expecting them to do all these
• 46:20
  different tasks. So that means that if you change
• 46:24
  something about the model, there's tradeoffs all over the
• 46:28
  place. It might get better one place,
• 46:31
  it might be worse on another, just because you've changed the
• 46:34
  prompt a little bit, so evaluating whether the change
• 46:39
  you've made is a good thing or not suddenly becomes super hard.
• 46:44
  Yeah. You know what, another piece
• 46:47
  that I, this is a little bit not related, but it sparked this
• 46:51
  idea, yeah, that I've been thinking about quite a bit is
• 46:55
  how a lot of these tools, especially when it comes to
• 47:00
  prompting and prompt tracking, they're not built for the people
• 47:05
  that are actually doing a tonne of the work.
• 47:07
  Like the person that is there iterating on lots and lots of
• 47:12
  prompts can be anybody. It can be a data scientist, it
• 47:16
  can be a marketer, it can be a sales person.
• 47:19
  And a lot of the tools that are out there right now are like the
• 47:22
  weights and biases that are specifically geared towards data
• 47:27
  scientists. And so a marketer or a
• 47:30
  salesperson isn't going to be like, all right, now let me just
• 47:32
  go open up weights and biases to track my prongs, right?
• 47:36
  So it's a little bit tangential, but I was thinking about that a
• 47:40
  tonne and thinking how I sit I and generally against advising
• 47:49
  people to start something in this space because it's just so
• 47:53
  crowded and there's so many different tools out there.
• 47:56
  But that feels like something that might need to happen,
• 48:01
  especially for the for the random marketers who are getting
• 48:05
  a tonne of value by running a bunch of their marketing copy
• 48:09
  through chatbot. Yeah, I, I feel like the
• 48:13
  community hasn't even settled on a process for how you do these
• 48:18
  evaluations, let alone turn that into any kind of tool.
• 48:22
  There's there's no consensus yet, so it's going to be
• 48:27
  fascinating to see that evolved to see that.
• 48:31
  Yeah. And evaluations, the tools that
• 48:33
  are out there on evaluation, I have a document that eventually
• 48:40
  I imagine I'll make public, but it's just like a Notion database
• 48:43
  and it's called same, same, but different.
• 48:46
  And it's all the tools that are out there that I've seen or come
• 48:51
  across that are in the mill OPS or like the LMAI infrastructure
• 48:56
  space. And one thing is for sure,
• 49:00
  there's a lot of evaluation tools out there.
• 49:03
  Yeah, that's right. Yeah.
• 49:05
  And it, I guess it makes sense because like you're saying,
• 49:07
  it's, it's a hard problem that has not been cracked yet.
• 49:10
  So everybody's trying to take a swing and saying, hey, well, we
• 49:14
  did this at XY, our last company.
• 49:17
  We kind of got it figured out. So here's how you can do it.
• 49:22
  But yeah, we we probably will be standardising around a how in a
• 49:31
  year or two hopefully, but judging by how fast.
• 49:34
  Things move, Yeah, because it feels like it's.
• 49:37
  It's very easy to make something that works well in a few cases
• 49:43
  and kind of eyeball it right some write some tests.
• 49:48
  But how do you scale that up that that's hard?
• 49:51
  Yeah, exactly. And yeah, predictably, being
• 49:55
  able to just trust the output, I mean, honestly, that's kind of
• 50:00
  the whole reason that we are doing the AI quality conference
• 50:03
  because it's like, well, we're all a little bit confused at how
• 50:07
  to go about this. Let's talk to each other and see
• 50:10
  if anybody if we can surface some of these practises where
• 50:15
  people have figured out different chips and tips on what
• 50:19
  to do and how to make it more reliable.
• 50:21
  Because like you said, getting 80% of the way there is great.
• 50:25
  And then, yeah, it could take you double, triple the amount of
• 50:28
  time to get 90% of the way there.
• 50:30
  Absolutely, yes. Yeah, it sounds like it's going
• 50:34
  to be a great event. Yeah, yeah, I'm looking forward
• 50:37
  to it. And you're able to make it
• 50:38
  right. Yes, yeah, that's.
• 50:40
  Right. Yes.
• 50:41
  Ohh. I'm so excited.
• 50:42
  Yeah yeah, me too. Well this has been really really
• 50:48
  thought provoking talking to you.
• 50:49
  I am 100% in the camp of data scientists need to learn how to
• 50:56
  at least understand what the system does.
• 51:00
  And it will be immensely valuable for any data scientists
• 51:03
  out there to learn the jargon at the very least and understand
• 51:08
  what APIs do, understand how to work with git, understand what
• 51:13
  security limitations there are or what's good, practise what's
• 51:17
  bad practise when it comes to security and all that fun stuff.
• 51:22
  So it's really cool to see that your book is out there.
• 51:25
  For anybody that wants to get a copy of your book, we're going
• 51:30
  to leave a link in the description, but it is called
• 51:34
  done, done, done, done. As we mentioned before, software
• 51:37
  engineering for data scientists. That's great, yeah.
• 51:41
  And thanks for writing it. I know that there's a tonne of
• 51:44
  people out there that are going to be better engineers because
• 51:49
  of it. That's fantastic, and I had a
• 51:51
  lot of fun writing it as well, so I'm excited to see it get out
• 51:54
  into the world and start hearing what people think of it.
• 51:57
  So. What was one thing that you
• 52:00
  learned while writing it that stays with you still?
• 52:05
  How much time I can save by automating things?
• 52:08
  Because there's there's an initial setup cost to setting up
• 52:12
  tools and to automate processes that you do repeatedly.
• 52:19
  But once you've done that and you can just press a button and
• 52:24
  run your test and lynch your code and run the formatting,
• 52:27
  it's great and save you so much time.
• 52:30
  Yeah, it's it's heavy on the front end, but then you will
• 52:37
  think your future self will be thanking you.
• 52:39
  Yes, absolutely.