Latest / Agentic Conversations (formally mlops.community) / Multilingual Programming and a Project Structure to Enable It // Rodolfo Núñez // MLOps Podcast #153

Transcript

Multilingual Programming and a Project Structure to Enable It // Rodolfo Núñez // MLOps Podcast #153 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Rolf on, yes, everybody calls me roll up and I
• 0:04
  work entailed. A telecom company as a senior,
• 0:08
  Emma loves engineer. And I do not drink coffee, but I
• 0:12
  really like to drink water and hot chocolate.
• 0:16
  Oh the project structure I think.
• 0:17
  I really like them which is the idea of automating things and
• 0:21
  stuff doing them over and over again iy talking to that project
• 0:24
  directly actually I wasn't living creepy as like.
• 0:28
  Give it to me. I'll see you.
• 0:30
  A few things. Yeah, maybe he can open source
• 0:33
  that, I would be really nice. What's going on everybody?
• 0:36
  This is your host Demetrios. And today, I am joined by none
• 0:39
  other than Abby. And today, we are also talking
• 0:43
  with Rollo who is what I would like to call a recovering data
• 0:46
  scientist. He has moved from being a senior
• 0:49
  data scientist, into being a, senior machine learning engineer
• 0:53
  at Intel, which is not the Intel that you probably think of, that
• 0:57
  makes the chips, its Intel, a Company out of Chile and they
• 1:01
  are doing telecommunications. So we talked all about different
• 1:07
  learnings that he has had going from being a data scientist into
• 1:11
  machine learning. It was B, should almost be
• 1:14
  titled what you need to know in ml Ops if you are a data
• 1:18
  scientist and he had a few hot takes, what do you think about
• 1:21
  this Abby? I felt like I did a creep at
• 1:26
  some of us takes know with all of them but I think something's
• 1:31
  worth reading twisting the one part about automating the
• 1:34
  workforce instead of doing the same thing over and over again.
• 1:37
  Even if it doesn't take a lot of time, but it is a blocker and
• 1:42
  the second thing which was about reproducibility and the kind of
• 1:46
  almost all CDs devil organization that they have with
• 1:49
  other projects. They're interesting.
• 1:52
  Onboarding process. They have Uber the bringing new
• 1:55
  hires so that they can teach the culture side of things.
• 1:58
  And the culture isn't hey, we need to have a copy every Friday
• 2:02
  and you need to say to nice things about your team members,
• 2:05
  the cultures, which is also nice culture, but that's not the key
• 2:09
  culture that we're talking about in this respect.
• 2:15
  You don't. Like if you don't like people,
• 2:18
  then that might not be the culture for you.
• 2:20
  Exactly. If what we're saying.
• 2:22
  But it is true that this type of culture that we're talking about
• 2:25
  with you need to do these things like this here is your cookie
• 2:29
  cutter template that you're going to use and you should know
• 2:35
  a few of the essentials as we called it.
• 2:37
  And we really jump off this conversation with the essentials
• 2:42
  which are data scientists and what they need to know to be
• 2:46
  dangerous Beyond just data science.
• 2:48
  I mean, obvious one, there is get the other one.
• 2:52
  One is just some kind of version control and it goes back to that
• 2:56
  reproducibility you were talking about.
• 2:58
  So I loved that. And then he talked about the
• 3:01
  clean code methods and how you need to be thinking about clean
• 3:05
  code, and he went into the two different camps of should you be
• 3:08
  commenting your code, or should you just should your functions
• 3:12
  be self-explanatory, or should your variables be
• 3:15
  self-explanatory? So you don't need to comment if
• 3:17
  I really meant to PT 3, we can comment the code automatically.
• 3:22
  There we go. So it's even easier.
• 3:24
  And now there's no excuse to not be commenting your code and
• 3:28
  having this clean code. And even I mean, what?
• 3:31
  G PT 3, I've seen some examples of people saying stream you
• 3:36
  know, update my code or make it better, right?
• 3:39
  What is it? Re-fracture my code and GPT 3
• 3:42
  will give that to you. So you it as a data scientist,
• 3:45
  if you are not feeling strong about your code just throw it
• 3:49
  into G PT 3. Hopefully, it's still free by
• 3:51
  the This episode comes out and you can refract your code and do
• 3:56
  not trust everything that it comes out with because I've seen
• 3:59
  too many outputs from GPT three that are absolute bullshit.
• 4:04
  That is a whole nother story for a whole nother podcast though
• 4:08
  March is sort of the time of release of the next Upp version.
• 4:11
  I'm not too sure if we this episode will be released by
• 4:15
  marshburn is going to be extremely.
• 4:18
  Oh, we'll see. We'll see how that goes.
• 4:21
  And we'll see you. If it has a little bit better,
• 4:23
  accuracy. So, as far as this conversation
• 4:27
  goes, Rollo was awesome, I really appreciated his Outlook
• 4:31
  and his views on things. Hope you do too.
• 4:34
  Before we jump into it. I would like to say that we have
• 4:38
  so much going on in the ml Ops community and so many people
• 4:41
  tell me that they do not realize all the cool stuff we have going
• 4:44
  on. So I'm just going to list a few
• 4:46
  things that we've got happening. You might want to check out the
• 4:50
  virtual meetups that go on on. Wednesday's every other week,
• 4:54
  all kinds of good stuff. And if you're a visual person
• 4:57
  and you like to see system diagrams, or if you like to do a
• 5:00
  little Hands-On workshops, we're doing them at the virtual,
• 5:03
  meetups Wednesdays at 6:00 CT. That is 9 a.m. in California,
• 5:08
  and I think it's 12 p.m. in New York and then we've also got
• 5:14
  three different emails that are happening.
• 5:16
  And we, I just started posting a ton on Tick Tock.
• 5:19
  So if you do the tick tock, think, um, And follow me over
• 5:22
  there. And last but not least, we've
• 5:25
  got a Blog and there's so many good things if you like to read
• 5:29
  and you're not into the boring blogs that had are talking about
• 5:35
  what is ml Ops and you want to go a little bit deeper.
• 5:38
  That is what our blog is for. Without further Ado, let's get
• 5:41
  into this conversation with robo.
• 5:50
  Let's just ease into it with this, all right?
• 5:52
  This is where we'll start. Let's ease into it with your,
• 5:56
  take on why data scientist should know how to code and code
• 6:01
  properly. But at the end of the day and
• 6:05
  data scientists are building software, very specific kind of
• 6:11
  software. It's a software that takes data
• 6:16
  transform data and output model or a Big turn or whatever it is,
• 6:22
  it's not always with material learning, sometimes it with
• 6:25
  other tools, but at the end of the day, you want to get a
• 6:29
  product out and it has to be useful for people, right?
• 6:34
  So if, if data scientists, do not know how to properly
• 6:40
  structure, and I mean the basics.
• 6:43
  I'm not, I'm not asking data scientist to be like, Pearls at
• 6:48
  coding. No, no.
• 6:50
  I mean, the basic, the very Basics.
• 6:52
  What are the basics? Get, ya give absolutely great.
• 6:55
  If data scientist is working without get I immediately, tell
• 7:01
  them, wait, not stop, stop right there in red flower using it, or
• 7:06
  maybe another tool, maybe, I don't know, SVN or Mercurial,
• 7:10
  although I think it is better. But what I mean, is, do you need
• 7:14
  to you, you need to be using Version Control of a good
• 7:19
  version control. Not at the an Title 1 and title
• 7:23
  to entice the tree. No.
• 7:26
  So that's the Step Zero. You need to be using gate or
• 7:30
  something like that. Another very basic point for me,
• 7:35
  is good naming of variables and functions and stuff like that
• 7:41
  because this is an. And now I'm going to talk about
• 7:46
  kind of two different schools. There are people Who usually say
• 7:51
  that you should comment everything on your code.
• 7:55
  So all are not not everything but a lot.
• 7:58
  So it is clear to other people who will then read your code to
• 8:02
  what you wanted to do, y etcetera.
• 8:06
  And there is another school that says you should not be
• 8:10
  commenting code because it should be self document.
• 8:14
  I'm more on the second school. I like to write code in Clean
• 8:20
  manner with a good with good names for variables, good names
• 8:26
  for functions. For example, I use for
• 8:30
  bullion's. I always use the prefix East or
• 8:36
  has or something like that. So, it is clear that it's a
• 8:40
  volume because you are basically asking a yes, or no question
• 8:44
  four variables, I tend to use kind of Not not too long
• 8:51
  variables, not to do name too but I don't use just the F, for
• 8:56
  example, for a data frame. I think people should name their
• 9:01
  objects, their classes, or whatever.
• 9:03
  They want. In a way that it makes it easier
• 9:08
  to understand when people are reading it.
• 9:11
  And no matter what, where you are reading.
• 9:14
  Because sometimes they do defined, let's say you define a
• 9:17
  data frame at the beginning of your code.
• 9:19
  And Then you use it in con, I don't know, 10 to 100 times
• 9:27
  during their whole code and you're using another data frame
• 9:32
  that you use in making half of that are Health those points.
• 9:38
  It's if you name, it data are DF and the other one data, it's
• 9:44
  very easy to mix up. So in general, I want to give
• 9:48
  data frames like Like last names or something like that.
• 9:52
  Like, dataframe, dataframe underscore role to to know that
• 9:57
  this is the first data frame that I read, I read or dataframe
• 10:01
  underscore missing values clean. So I don't have to explain to
• 10:07
  you who I did that because the name tells you, what is that?
• 10:12
  That way you can more easily, share your code to other people
• 10:18
  to other data. These two other ml UPS Engineers
• 10:23
  or to Forever. It's going to take that code and
• 10:27
  you can use it in any way. Yeah.
• 10:32
  Speaking by sharing, I have to give a shout out to will who won
• 10:38
  the greatest award and the ml Ops Community for 2022.
• 10:43
  It was the most cringe thread. And what he mentioned in this
• 10:48
  thread was that I just watched a data scientist copy and paste
• 10:53
  code from a notebook into vs code, save the file to disk and
• 10:59
  email it to us so we can put it into production while he's on
• 11:04
  vacation in Hawaii. That is a team player right
• 11:07
  there. That is a data scientist.
• 11:10
  That is the exact opposite of what you are asking for.
• 11:13
  And what I'm wondering is, how did we get to that?
• 11:15
  Like how does a person like that survive in the Workplace.
• 11:21
  Yeah, well, I mind is blown. You're speechless right now.
• 11:25
  You can't even think about it. Yeah, I have a, I have a
• 11:28
  strategy where I work. We usually hire a lot of Junior
• 11:36
  of data scientists. So most of them already come
• 11:41
  from. I don't know, school, maybe, or
• 11:43
  boot camps or whatever, and they are used to work on jupyter
• 11:48
  notebooks, or they don't know how How to use git or they just
• 11:52
  now get exist and that's all the debris basic.
• 11:56
  So, what we do at least and it's useful in, my opinion, is to
• 12:01
  have a very structured, our onboarding process everything,
• 12:06
  well documented, and at the very end of that onboarding process,
• 12:10
  we give them kind of an assignment.
• 12:14
  I need some standard brake. It's a depraved that every data
• 12:18
  scientist, that enters our team. Has to do it there.
• 12:22
  We kind of force them to use, good tools, we force them to
• 12:27
  use, get because they need to get the, the break with that.
• 12:30
  They need to create their own repo and push everything.
• 12:35
  At the end of the day, we want them to know the basics of get.
• 12:40
  Again, we don't want them to be processed.
• 12:43
  No, it's not necessary. We don't even talk about pull
• 12:46
  request or stuff like that. It is necessary.
• 12:50
  At the end, but we don't start with that.
• 12:54
  We just want to then to know the basics.
• 12:57
  And after that we give them a template for for projects, we
• 13:02
  developed in-house project template for data science
• 13:08
  projects, quite quite open. I mean quite General and I think
• 13:15
  every team should be doing something like that, everything
• 13:19
  should have a template. Already set up and it will
• 13:23
  depend on on the team. Some things will need more of
• 13:27
  some stuff and other teams will need other stuff, but you need
• 13:31
  to have a structure way to work together.
• 13:36
  So, and the people who join in the team, get easily kind of the
• 13:42
  culture of the team and kind, and they can adapt, and then
• 13:49
  they can easily share that. With more senior members, for
• 13:52
  example, under the into senior members can review that code and
• 13:57
  give feedback in a more streamline Runner interesting
• 14:02
  because this is something. Well, it's always talked about
• 14:07
  as one of the best practices, which is let's work with
• 14:09
  templates. And there's one very interesting
• 14:12
  template that is out there for their science projects, at least
• 14:15
  the one that I've been using, which is basically something
• 14:17
  called cookie cutter project. I don't know if you check that.
• 14:21
  But there are plenty of those, but one of the big differences
• 14:25
  is different companies have different stages of d&i, science
• 14:28
  projects and different team sizes as well.
• 14:31
  How do you make that distinction?
• 14:33
  As to, you know, not every data scientist is supposed to write
• 14:36
  production level code or not. Every code is supposed to be
• 14:40
  pushed into production. What's the distinction factor
• 14:43
  for you? Very you decide, this is where
• 14:45
  we spend a lot of effort on writing unit tests writing
• 14:48
  integration tests, and sort of Working with more of like an
• 14:52
  object-oriented design working more so with abstractions
• 14:56
  instead of encapsulations and all that stuff and where do you
• 15:00
  think about software design? And where is it a deer science
• 15:03
  project? Well at the end of the day, the
• 15:05
  final objective is to create something that is useful for the
• 15:10
  Enterprise for the market, so the answer should depend on
• 15:14
  that. Sometimes when we do things kind
• 15:18
  of too perfect we end up not Doing it because we take too
• 15:22
  long and doing and find it. And at the end, we don't finish
• 15:26
  anything. I think it's useful, it's very
• 15:29
  important to find the the balance there.
• 15:34
  Some projects can be much more easier to go than to put into
• 15:40
  production that they don't need to follow to strict standards.
• 15:44
  I still think we should follow kind of the basic standards
• 15:48
  which I already talked about. But in I think it's it in
• 15:54
  general it's not a problem to not work with glasses or stuff
• 15:59
  like that in most of the most of the projects that we developed
• 16:02
  on on my team. At least basically a series of
• 16:08
  scripts that at the end of the day produces an output, the
• 16:14
  objective is to take that output and put it into production, the
• 16:19
  way you do that. Not that important.
• 16:22
  As long as you manage to get that product out.
• 16:26
  So we don't always use classes. We don't always use oriented
• 16:30
  each object, programming, Etc. It will depend on what we need
• 16:36
  to do. If we need to maybe create a
• 16:39
  tool for others to use, then it could be more important because
• 16:44
  it has to be pretty General. It has to be useful for many
• 16:49
  other. People many other data
• 16:52
  scientists. For example, if we are creating
• 16:54
  attitude for data scientists to monitor their their models, it
• 17:01
  has to be well structure and maybe with a classes and
• 17:06
  everything like that but if you are just getting a score for out
• 17:11
  of a model and you want to score clients with that, it's not
• 17:15
  necessary really? You can just run a series of
• 17:18
  scripts in a certain order. And if they are well set
• 17:23
  documented and the Emma loves engineer who takes that can
• 17:29
  easily understand all of the steps and put that into
• 17:33
  production and get value out of that project so it will depend
• 17:38
  on on that. What's your final objective and
• 17:42
  do you really need to use more General programming?
• 17:49
  I see and also so one of the one of the things that you do talk
• 17:53
  about a lot especially I've seen in your LinkedIn post and some
• 17:56
  of your other comments has even three insert an ambassador for
• 18:00
  our markdown and quarter. So do you want to talk?
• 18:05
  You should we be using those over using?
• 18:07
  Let's say to protect folks that I started programming on vitamin
• 18:12
  A big at the beginning at the very beginning like kind of at
• 18:15
  school and then I switch to our when I started.
• 18:21
  Working then I was like working with both languages sometimes.
• 18:27
  So after working a lot with our analog, with our story, because
• 18:31
  if you are using our and you're not using our studio, you are,
• 18:35
  you're probably not doing it right because our study is kind
• 18:41
  of the best idea for our results.
• 18:45
  To the goal is pretty good, for general purpose also.
• 18:49
  So I tend to mix. Both of them.
• 18:51
  And then when I started working again, more in Python, I started
• 18:59
  noticing a lot of deficiencies on the basic jupyter notebook
• 19:04
  structure because I was constantly comparing it with our
• 19:09
  markdown or cuarto cuarto, see more new.
• 19:12
  So I was not comparing it to court at the beginning and but
• 19:17
  some of the pros that has Armor down and quarter which are very
• 19:23
  similar are that if you are not writing your output inside the
• 19:29
  same document that you are counting in for something to be
• 19:33
  the notebook, do right, do right?
• 19:36
  A cell you run those that sell you saved in the file and the
• 19:40
  output of that cell is automatically save inside that
• 19:45
  interactive, no, dude pack python file.
• 19:48
  So if you open that file with That text editor, you will see
• 19:52
  that at the end of the day, it's just, it's just a Json, Json
• 19:57
  file with a bunch of nonsense. Basically, when you save and
• 20:02
  upload an image, you hit had a bunch of nonsense because you
• 20:07
  are saving an image on a Json file.
• 20:09
  So you need to write a lot of random characters, kind of
• 20:13
  random to save that image. So the, the computer can
• 20:18
  interpret that But on Armor down, you don't do that.
• 20:22
  You get your output in your code, editor, your file, it's
• 20:28
  just code. Your file is just the Martin
• 20:33
  code that you rode with the python code or our code or C++
• 20:38
  code that you wrote etcetera and you can mix all of those in the
• 20:43
  same document. You you can very easily create a
• 20:47
  cell for Python and then The next cell is for our and then
• 20:51
  the next cell for batch etc. Etc, etc.
• 20:54
  Coding jupyter, notebook. In contrast, you need to choose
• 21:00
  a colonel and you can run everything with that Colonel.
• 21:04
  Maybe you can mix. I'm not sure.
• 21:06
  I haven't done it. If you have its tell me, but at
• 21:11
  least as far as I know you need to choose a colonel and run
• 21:15
  everything with that kernel, you can run our, you can run but you
• 21:19
  Also on jupyter Notebook, if you change the colonel but you need
• 21:24
  to choose one on our studio armor down and korto.
• 21:29
  You can choose whatever you want.
• 21:31
  So it's much more flexible going back to the the problem.
• 21:37
  We saving your your output on the file.
• 21:41
  That is really a pain in the ass to because if you try to commit
• 21:47
  that into a git Repository, You will be very quickly start to
• 21:53
  use a lot a lot of memory because when you change and
• 21:57
  upload every time you change the plot and you can meet that, you
• 22:01
  are committing the whole blood and the whole blood is, I don't
• 22:05
  know, maybe 100 lines, 200 lines of code that change and maybe
• 22:12
  you. You only change one point.
• 22:14
  Maybe it was a scatter plot with just one difference or maybe you
• 22:17
  change the limits of your lat and that will save a whole new
• 22:22
  plot and that will happen for every one of your output.
• 22:26
  If you plot the data, the data frame or ahead of that of data
• 22:31
  frame, you will also save that data.
• 22:34
  And you, if you commit that, and you send it to your repo, if you
• 22:40
  are working with on a open repo, for example, you could be
• 22:44
  filtering out sensitive information that you shouldn't
• 22:48
  because So one thing is to share your code, maybe your code is
• 22:54
  open source or maybe your colleagues to private, but
• 22:58
  another whole different things to share data.
• 23:01
  Sometimes you are working on an open project.
• 23:04
  But with private data, with jupyter notebook, you can you do
• 23:09
  good sleep. So then dated you could be
• 23:14
  following some security issues. For me, you spoke about these
• 23:20
  different, basically, the templates that you have for the
• 23:24
  data scientists and you also are talking about are now and I know
• 23:29
  that you mentioned, you're very proud of a setup that you have
• 23:34
  put together with a few other of your colleagues.
• 23:37
  Can you walk us through what that setup is and what it
• 23:40
  entails and why it made you so proud to have put that into
• 23:45
  Practice at your current job. Yeah.
• 23:48
  When I started working as a data scientist about 60 years ago,
• 23:55
  with think my first assignment as a junior there was to put in
• 24:01
  put in place many of the tools that we were going to use
• 24:06
  forward because I was the first data scientist on my job.
• 24:10
  I started digging like best practices and stuff like that.
• 24:15
  And I found a pretty good video. I don't remember who was talking
• 24:20
  but it was a video about good practices about on data science
• 24:24
  from 2017. Maybe quite no no to see 2015.
• 24:30
  Quite all, they show up a project template that it made
• 24:35
  sense for me, but I felt it wasn't complete.
• 24:38
  So I really based on that started adding progress.
• 24:44
  The, a lot of things with a lot of iterations with talking with
• 24:49
  other co-workers and we ended up with a project template that
• 24:55
  it's in my opinion, pretty General, you are not forced,
• 25:00
  you're not to force to do stuff, one way, or not, or than, or
• 25:04
  another, but it gives order to the project.
• 25:09
  So the, the basic idea is that when you are working with data,
• 25:16
  you are working kind of with a data flow.
• 25:20
  So you, let's let's speak about data in the general sense.
• 25:26
  You can have data as a data frames or I mean csvs or
• 25:33
  whatever storage you. You have your data on door,
• 25:36
  maybe images, or maybe just config files, everything there
• 25:42
  is data at the end of the day. So what we try to do is to have
• 25:48
  a pipeline of of many scripts that take an input data gives
• 25:56
  you an output date. So every streak, maybe with some
• 26:01
  a couple exceptions will take input data and will give you
• 26:05
  output data for that that script.
• 26:07
  And then the next script will take maybe input data from the
• 26:10
  last script and maybe from other sources and output.
• 26:14
  Something else and so on. So on, so, on that is very
• 26:19
  useful in my opinion, because when you are running a whole
• 26:24
  pipeline, if you work, you if your pipeline breaks, at some
• 26:28
  point, you will know where you where it broke and you will
• 26:33
  know, okay? At least I have everything at up
• 26:35
  until their correct. I need to debug this section
• 26:40
  this script and then I can run from there too.
• 26:44
  And it's sold something that I hear many people who use jupyter
• 26:51
  notebook, talk about, which is that with jupyter notebook, you
• 26:55
  don't always have to run everything again because it sits
• 26:59
  there and the outputs. And what I always say to them is
• 27:03
  why don't you save your out into well folder, a specific folder.
• 27:09
  So there we are solving that, that problem and Each file has a
• 27:16
  certain output. So if you have a lot of outputs
• 27:20
  it could be kind of messy at the end of the day because you could
• 27:25
  have, I don't know, ten output files for one project because
• 27:30
  you have 10 scripts so it could be messy to know where each
• 27:34
  output comes from. So what we do is that each time
• 27:39
  a script saves a file, it saves it with a with the prefix and
• 27:44
  the Graphics, its kind of the ID of that script.
• 27:48
  We're right pipelines with specific prefixes.
• 27:52
  For example, 800 a01, a02 Etc. So whenever you see an output
• 28:00
  with the prefix AC, or to, you know, that the ACO to file
• 28:05
  produced that that output. So if that output is incorrect,
• 28:10
  you already know where to go. Look for where you need.
• 28:14
  To go debug or study another very important point is to
• 28:20
  modularize your code. In our team, we try to break our
• 28:24
  whole process into many different scripts.
• 28:26
  So, we have this checkpoints, for example, we can have the
• 28:31
  pre-processing broken down to 10 different steps.
• 28:35
  First, we read the data and then we read another data and we mix
• 28:40
  it. We merge, the set, and then we
• 28:43
  take that in another screen. And clean the missing values or
• 28:47
  duplicates Etc. And so on and so on each script
• 28:51
  should be doing one job. Kind of like functions but not
• 28:56
  that much. We use a lot of functions inside
• 28:59
  once created. And so it's not that specific,
• 29:04
  but it's very useful because you know, that one strip is going to
• 29:09
  be doing one job, and the job that is doing is also in the
• 29:13
  name of the The script, the name could be a 00 underscore
• 29:20
  reading. First Data set.
• 29:23
  Weather dot by the are or whatever.
• 29:26
  So it's very tidy way to organize projects in my opinion.
• 29:32
  And also it allows you to mix many languages together because
• 29:38
  every script is independent cops.
• 29:41
  You have to immerse yourself in the amylose content.
• 29:44
  And the best way to do it is to subscribe to the email Ops
• 29:49
  Community podcast soon. Good luck and keep learning So
• 30:00
  if you want to mix python script with a with an R script, it's
• 30:05
  very easy because you just write your python script which will
• 30:09
  have an input. Will do something will have an
• 30:12
  output and then that output you can read it with an R script and
• 30:16
  do something with our have an output.
• 30:18
  Then you read that with another script with whatever language
• 30:22
  you want and go and so on. So one of the questions I have
• 30:27
  on this one because this does seem like a very good way of
• 30:31
  thinking about problems. Instead of just doing things
• 30:34
  randomly, one of the main reasons of one of the things
• 30:37
  that I've seen is, sometimes one of the Cordia scientist that is
• 30:42
  working on a project leaves Midway, and it's a very easy to
• 30:46
  bring on. If you already have such an
• 30:48
  existing structure, the it's very easy to bring on a new
• 30:51
  person or for everybody else on the team to sort of pick up the
• 30:54
  work. From a theft.
• 30:57
  But one of the things I wanted to ask you on along these lines
• 31:01
  with more. So how much of all of this is
• 31:04
  documented how much of the script?
• 31:06
  How much effort is put into the script documentation and
• 31:09
  everything, so that everyone knows what is the real
• 31:12
  significance of this. Also, another thing, which is
• 31:16
  basically the do we really need the script?
• 31:19
  How do you sort of identify that you're not putting too much
• 31:22
  effort and designing structures? That could probably be just like
• 31:27
  a one-time use on this break template.
• 31:30
  We have, we have a lot of folders, different different
• 31:33
  folders. For example, we have the file
• 31:35
  folder, the files folder where we have everything, basically
• 31:39
  that we don't want to commit the data sets or documentations or
• 31:46
  references to papers logs figures at the models that they
• 31:51
  the object of the model. That is saved everything.
• 31:55
  Sabin files on different subfolders which are data sets
• 32:00
  documentation and model output and on model output, we have the
• 32:06
  object of the models on one folder logs and another folder
• 32:09
  for figures and another folder reports on another folder Etc.
• 32:14
  So it's very easy to go and find whatever you you are looking for
• 32:18
  the data sets, we separate them on input data, intermediate data
• 32:23
  and output data. We could add more more layers if
• 32:27
  we want and there is an out of files.
• 32:30
  We have many other different folders.
• 32:33
  For example, we have the pre-processing folder.
• 32:36
  Where is all of the processing for whatever you are doing the
• 32:40
  models, the model folders were you have the scripts for the
• 32:44
  models that you are working on the reports folder, where you
• 32:49
  have the scripts for the report? Not the reports itself because
• 32:54
  those are safe. On the files folder.
• 32:56
  But the the script of the reports and we have also more
• 33:01
  folders that I'm not going to get into right now, but we have
• 33:04
  a Sandbox folder. So whenever you are starting a
• 33:09
  script and you don't quite know where it's going to land, or if
• 33:14
  it needs to be split into or whatever, we usually develop
• 33:18
  that on the sandbox folder and everything on Sandbox.
• 33:21
  It's not officially in when you receive a Eject from us,
• 33:26
  everything on Sandbox, you can delete it and there's no
• 33:29
  problem. But you can start working on on
• 33:32
  Sandbox. And when you already know what
• 33:36
  you will need, what we are doing, how much splitting you
• 33:41
  need to do, then you split that in, I don't know, two scripts
• 33:45
  for example and you place them wherever they need to go.
• 33:50
  This also makes me think of a very interesting direction that
• 33:53
  be going towards, which is The more popular books in the field,
• 33:56
  which is basically from monoliths to microservices.
• 33:59
  I think that's sort of a be taking off, which is 12.
• 34:02
  Ping particular API is very different things because you're
• 34:05
  managing everything so well, you can easily continue to rise it
• 34:09
  and debug individual Parts as well.
• 34:13
  That's another important point. We put everything there on a
• 34:19
  container, so it is easy for for it to run on.
• 34:25
  On what whatever computer you are you are using and with the
• 34:30
  virtual environments for every language because you can have a
• 34:33
  virtual environment for python another virtual environment for
• 34:36
  our and it's it's very easy to set up both of them and that way
• 34:42
  you can have no troubles when you put that into production in
• 34:46
  the intermediate. So if you want to go into
• 34:48
  communities, you can have her questions as well.
• 34:52
  No, I'm actually I really like the The fact that you spoke
• 34:56
  about reproducibility earlier and now this seems like you're
• 35:00
  organizing everything so that reproducibility is that main
• 35:04
  player and it's in the Forefront of what you're doing.
• 35:08
  Did you get burned enough times to where you realize that needs
• 35:12
  to be very much a part of what we do or was it just something
• 35:17
  that came from software engineering that you had in your
• 35:20
  blood? How did you realize that
• 35:23
  reproducibility was so important and then then why have you
• 35:27
  started implementing it or what other things have you done to
• 35:32
  implement it besides the scripts that you're talking about?
• 35:35
  I'm not quite sure where it started but I like to think
• 35:41
  problems in a broad kind of stunts in a general manner,
• 35:44
  maybe it was a little bit influence because I study math.
• 35:48
  So when you study math you have to abstract yourself a quite a
• 35:53
  lot and think about Random bizarre spaces.
• 35:59
  So I did not study computer science, but I really like it.
• 36:03
  I saw that influence it a little bit too, but I did have some
• 36:11
  kind, some troubles on some project because I need to run
• 36:17
  something manually for a specific project.
• 36:21
  So every day or not, every month, I need to run.
• 36:25
  Something. I need to take me like, two days
• 36:27
  to run that every month, so it wasn't too much, but it was
• 36:33
  annoying. I was just losing my time, so I
• 36:37
  started thinking how I can how to make this and do it in a more
• 36:43
  General way. So I think that all of these
• 36:46
  factors contributed but I'm not sure if there is kind of a
• 36:53
  particular one. That was More important.
• 36:56
  Another question is, let's say, I don't know if you do, you
• 37:01
  might have some scripts to be able to manage the events and
• 37:04
  stuff which is the entire event processing side of things.
• 37:09
  How do you, or how often do you cater to the schema changes on
• 37:13
  that one? What they mean with schema
• 37:16
  changes. So anytime you're working with
• 37:19
  any sort of events, there are schema changes over a period of
• 37:23
  time for the events, which is wherever you're putting your
• 37:25
  events from how often you use it as a team and go like, okay,
• 37:29
  let's go into the code work. On the news changes that have
• 37:33
  been made in there in the source where we're pulling our
• 37:36
  diaphragm and now let's let's make sure everything's sort of
• 37:39
  adapted to that. Hmm, with the event, you mean
• 37:44
  class? Like kind of get have actions, I
• 37:47
  or stuff like that. Or what kind of events are you?
• 37:51
  Are you speaking? Are you to be able about?
• 37:54
  So it could be something like e.t. elevates.
• 37:58
  So, not not not essentially on the cic Deep.
• 38:01
  I'm also around the DI acquisition side of things
• 38:04
  because that's where a lot of changes have happened at least
• 38:08
  for me, which is any time we're working with a lot of integrated
• 38:11
  Services. Later.
• 38:13
  We are using 4 different. Services that are 14 or a
• 38:17
  payment Gateway and maybe the small so happens who waited
• 38:20
  because I've worked with e-commerce company specifically.
• 38:23
  I don't know how much of it happens at your work which is
• 38:26
  you're working with different services and they have changes
• 38:29
  in the code word changes the schema for you and that would
• 38:32
  make sure that you now have to sort of adapt on a few scripts
• 38:35
  accordingly. Okay.
• 38:38
  So you mean if somebody is as and if something changes in kind
• 38:42
  of the data or the structure of how things should work or moving
• 38:47
  to the Apes, all plays out of that stuff, okay, we don't work
• 38:52
  that much with apis. We work with internal data.
• 38:56
  I work for a telecom company, so we had all of the information in
• 39:01
  house. What you should lie changes
• 39:05
  although it's so The materials. But not usually sometimes,
• 39:09
  sometimes CDLs change. We need to add that.
• 39:13
  Do we need to correct this these models and what we are trying to
• 39:18
  do. But we are not yet, doing we are
• 39:21
  starting is to implement. I like a library called Great
• 39:27
  Expectations, which is very useful to assure that your your
• 39:34
  data quality is Maintain, and if it's not, it will tell you, it
• 39:41
  will give you an alert. And I think it's very, very
• 39:47
  useful. And we are starting to to use
• 39:52
  Great Expectations. And the idea at least is to use
• 39:57
  a couple checkpoints in the in the whole process.
• 39:59
  For example, at the beginning of the, of the model of the
• 40:02
  beginning of the kind of the pipeline of the model and we We,
• 40:07
  we want to check if the raw data set is has a still the same
• 40:12
  structure, the same columns, the same data types Etc.
• 40:19
  At the end, we went to check that.
• 40:22
  The model is outputting the kind of the same structure of data,
• 40:27
  Maybe with a given distribution or something, similar to that
• 40:34
  distribution. And in between we can place a
• 40:39
  couple more checkpoints to for example, to check that the the
• 40:43
  model. When you give the data to the
• 40:45
  model, we are giving it the right data to process it in the
• 40:49
  right manner. So at least those, three steps
• 40:53
  are very important in my opinion, and it's very useful to
• 40:58
  control those changes in kind of the ecosystem.
• 41:03
  You said, you have a lot of In-House data that is entirely
• 41:07
  owned by you. How do you or do you have like a
• 41:11
  deer cataloging solution is a built-in house for your team or
• 41:15
  do you use an external one? Also what are the words?
• 41:20
  Some things like you've learned along the way, which is how to
• 41:22
  build the right kind of data cataloging solution.
• 41:24
  Because I think one of the things that a lot of people
• 41:28
  don't talk about as often. It's the quality of your code is
• 41:31
  not directly dependent on the number of hyperparameters or the
• 41:36
  newer. Your mother but actually on the
• 41:37
  quality of the tea itself and that's where we sort of need to
• 41:40
  put in a little bit more effort. We did have a data catalog but
• 41:46
  it was internal like building house, but now we are moving to
• 41:52
  a new data catalog for the whole company because they want that
• 41:57
  we were using was only for the analytics team not for the whole
• 42:02
  company. But now we are working with
• 42:05
  another Another company to build data catalog company
• 42:11
  company-wide another working on that directly.
• 42:14
  So I don't have the details and even if I had, I don't know if I
• 42:20
  could share them, but I know that there are people working on
• 42:26
  that. So I want to go back to
• 42:29
  something. You said earlier about the data
• 42:31
  flows and you mentioned let's look at data as if it was this
• 42:37
  idea of it flowing and going from one place to another with
• 42:41
  the inputs and the outputs and then you don't have to think
• 42:44
  about. Am I using are on this piece of
• 42:47
  data in my using python. It's all just going into
• 42:51
  something flowing into something and then flowing out of
• 42:54
  something. Yeah.
• 42:55
  How else have you been? Thinking about that because or
• 42:59
  house. If you been generalizing that
• 43:01
  idea and using it, when you're designing the system that you're
• 43:06
  working with, there's a another folder that I didn't talk about,
• 43:10
  which is called pipelines. And there, we have the most
• 43:14
  important files because they're, you build the real Pipeline.
• 43:22
  And I mean, the Bible and for development, not for production
• 43:27
  and Necessarily. But there are the data
• 43:31
  scientists scanned called each script.
• 43:34
  In a certain order, if we are working with parallel processing
• 43:39
  or with serial processing, it doesn't matter much in terms of
• 43:44
  writing your script, it will always be serialize.
• 43:48
  You always write things in a Serial manner.
• 43:53
  Even if it runs parallel you have to write something before
• 43:57
  dealing one. Fake.
• 43:58
  So in this pipe lines you can call every script.
• 44:02
  Do you want run them in parallel, maybe or in in series
• 44:09
  in the end of that, that script of Pipeline and you can output
• 44:16
  all of the kind of the accumulative log of every every
• 44:23
  file, and you can print how much time did it too?
• 44:27
  For every, every step for every, every script Etc.
• 44:33
  And I think that's, that is very useful because you can then
• 44:39
  write in your read me and your Martin rid, me of the project
• 44:43
  and you can write a section on how to run this project and it's
• 44:49
  basically just run this file it. Maybe you have a lot of
• 44:53
  pipelines and each pipeline is for a specific.
• 44:57
  Acidic task. So you can say, well, if you
• 45:00
  want to give a score for every, for every client with these
• 45:08
  parameters, just run this script and you name it.
• 45:12
  And that way that pipeline will do, we know that you and you
• 45:16
  have to run that tied line, and that pipeline will do all the
• 45:19
  other job. They will call every, every ski
• 45:25
  every street every script will take They're out if their inputs
• 45:29
  and give you their outputs respectively.
• 45:32
  And when you do her eyes that, you just have to run one line of
• 45:37
  code, do care run and you get the the path to that pipeline
• 45:44
  that you are interested in. So it's pretty easy to put into
• 45:50
  production afterwards. The, the envelopes engineer
• 45:53
  doesn't need to understand everything that is going on.
• 45:57
  On in there as long as the data, scientists, understand State and
• 46:02
  makes sure that it's well structure.
• 46:06
  The envelopes engineer can just take it and run that line of
• 46:10
  code. And that's that's all the Prague
• 46:13
  will work, ask the data scientist intended that list.
• 46:18
  So, As far as. This goes, man, there's so much
• 46:24
  for me to chew on here that we've talked about and I really
• 46:28
  appreciate all these different insights.
• 46:31
  Now that we're getting to the end, I know that you had a ton
• 46:35
  of stuff that you mentioned. You could talk about, we hit
• 46:39
  almost everything, but is there anything that you wanted to talk
• 46:42
  about that? We have not hit yet.
• 46:44
  While you're thinking you mentioned to me, these are the
• 46:47
  things that I end up talking with people about, and I end up
• 46:50
  having strong opinions on And so, I think we could talk about
• 46:54
  a Minute Podcast and you do not need to say anymore to me if you
• 46:59
  want to do that, then I'm all-in.
• 47:03
  Yeah and maybe there are two topics which happened we come
• 47:07
  and talk about. The one is the the fact that I
• 47:11
  really think the elbow method, this is more for data scientist.
• 47:15
  Not for envelopes Engineers. But the elbow method, it's just
• 47:18
  a lie. It's an illusion and the other
• 47:21
  one is the agile methodology. Many people don't like them and
• 47:24
  I don't like the agile methodology has that kind of
• 47:27
  scram. But I think that there are there
• 47:30
  are a lot of very good things that you can get out of that and
• 47:36
  use for your team. These are the two topics I think
• 47:40
  we haven't talked a so what is? Yeah.
• 47:43
  Can you break down the elbow methodology?
• 47:46
  And let us know why you have to sit.
• 47:50
  It's easier if you. You beat realize it.
• 47:54
  So if there are people listening to this on Spotify, it won't
• 48:00
  help to share screen. So I just recommend you to
• 48:07
  create our Vector for, I don't know, from one to ten, ten
• 48:12
  thousand and then get the exponential of negative X.
• 48:21
  Where X is the Ha and blood that you will get a shape pretty
• 48:28
  well-known shape, which is very similar to what we usually get,
• 48:35
  when we are studying, clustering, and we plot how how
• 48:42
  much more information we are getting out of one more, one
• 48:47
  more, one more cluster. So just take that tape and then
• 48:51
  ask yourself. And where is the optimum cut?
• 48:55
  Where you, you gain enough information.
• 48:59
  So it is worth it to split. One more your, your groups and
• 49:05
  you will get a value, but everybody with this, it doesn't
• 49:09
  matter. And then after yourself, well
• 49:12
  maybe you tried five groups, one group two group, three groups,
• 49:18
  four and five. So we you have five points now.
• 49:22
  Do the same. But try with 20 groups.
• 49:25
  So we do we'll have 20 points and the optimal point will
• 49:29
  change because your perspective will change, you will see the
• 49:33
  same points, but visually, it will be different and the same
• 49:38
  with 100 points. And it will be different.
• 49:41
  Do you will find a new optimal? So it is not a good method to
• 49:46
  find the optimal amount of groups to split what I would
• 49:52
  rather. Either do is to take the points
• 49:54
  that I suspect are kind of good run, I don't know.
• 49:59
  Run whatever. I want to run with that
• 50:01
  afterwards. A model or a prediction or
• 50:03
  whatever run that and measure how much how much better is one
• 50:10
  versus the other with the methods that you really value
• 50:16
  with. I don't know we the error or I
• 50:20
  or the procedural requirements. Or what whatever you're
• 50:24
  measuring, but get the better amount of groups based on that,
• 50:31
  not the elbow method because the elbow method, it will depend on
• 50:34
  how many points. You are testing, right?
• 50:37
  So that's the sort of core idea behind cross sampling as well,
• 50:40
  right? I'm not sure.
• 50:43
  I'm not sure what crossed something is what do you mean?
• 50:46
  Exactly. Um so basically when you have a
• 50:49
  sample of data that you've working but instead of Taking
• 50:53
  disparate samples for examples in two different parts, which is
• 50:56
  collected, different things so that you have sort of enough
• 50:59
  diversifying the samples and then covert ever works.
• 51:02
  Best instead of taking on a group in sing, let's go work
• 51:06
  with this group of data down, its kind of this, similar idea
• 51:10
  one other piece before we go. You're leading three many teams
• 51:15
  inside of the company and they're focused on security.
• 51:19
  I want to just touch on this for a moment because I think
• 51:22
  security is one piece of machine learning and ml Ops in general,
• 51:28
  that does not get talked about enough.
• 51:30
  What can you tell us about? Your learnings from leading
• 51:34
  these teams? How are you looking at security?
• 51:37
  Now, how is your view of security and how you do things
• 51:40
  changed over the time since you started leading this?
• 51:43
  Teams the most amount of work that we have done on security
• 51:47
  issues. Inside the team has been related
• 51:52
  to The Geek or git repositories. It is very I already spoke about
• 51:58
  this but you can very easily share sensitive data by mistake,
• 52:03
  right? Maybe you, you are writing a
• 52:05
  code, you commit something by mistake.
• 52:09
  It's it will stick their you. You can maybe you can erase it
• 52:15
  later but it will stay there in your history of to repository.
• 52:20
  And that that is one of the virtues of get near you.
• 52:25
  You can Retreat, whatever you you were working on.
• 52:28
  If you wrote a password, for example, a password for for a
• 52:34
  database which is, well, it's a lot of people has done it.
• 52:39
  I've done it. I've written my authorization
• 52:42
  code in them. After use, like, get Guardian to
• 52:45
  Sir cash stay alert and be like, oh, shit, I'm exposed
• 52:49
  information story goes toward and then lamech variables.
• 52:55
  Yeah, that that's very, very common.
• 52:57
  I think I have done it also and I have, but I think that the
• 53:02
  most important thing is to know how to clean that, because you
• 53:06
  can there you at the end of the day, you have a hidden file on
• 53:11
  your repository. That MoDOT get file inside
• 53:16
  there, you have all of the history of all the changes, you
• 53:20
  have done. So one, one quick way to solve
• 53:24
  that is just delete that file but that it's not the ideal
• 53:29
  solution because you will lose all of your projects history to
• 53:34
  you will see a you will still have your, your working
• 53:40
  directory. So you can salvage some some
• 53:42
  things But it's not the the ideal solution.
• 53:47
  So what we did was to use a library called BFG which stands
• 53:54
  for big. Big tracking No Big B XG is big
• 54:01
  dragon from Doom. I don't know if you have played
• 54:04
  them, but there is a pretty huge gun, which will deliberate
• 54:10
  everything inside. So this Library.
• 54:14
  Does that with get and you can, you can select which things
• 54:18
  expects. Specifically is specifically, is
• 54:22
  it's going to erase so you can give it a limit on MB, maybe you
• 54:27
  can you want to erase everything that has, more than that way.
• 54:32
  It's more than 10 megabytes because maybe you accidentally
• 54:37
  committed a CSV file with data and you want to reach that.
• 54:41
  So you can, you can erase Everything, which is too big
• 54:45
  because if it's too big, it probably shouldn't be in your
• 54:49
  git. Repository git, repositories are
• 54:51
  made to store code to check differences between lines lines
• 54:56
  of code, not between whole files.
• 54:59
  That's one very, very useful thing about that Library.
• 55:02
  It's a Java library and you can download it and then just call
• 55:07
  it with jar and the other teeth. A very useful thing that you can
• 55:11
  do is to Delete specific commits.
• 55:16
  Maybe you. You already know that in a
• 55:18
  specific commit, you come into the password or you committed a
• 55:22
  data set or you can meet it. Whatever.
• 55:26
  That is very sensitive. So you can go ahead and tell
• 55:29
  that library to get erased that commit, what happened to us in
• 55:35
  our team is that many people had done this in many projects.
• 55:39
  At the same time we need to Erase like 200 Repose or we need
• 55:49
  to tweak 200 trips. So he, it wasn't viable to do it
• 55:55
  one by one because we didn't even know where we need to
• 56:00
  delete. So what we did was to create a
• 56:04
  mixture of everything and they and here, I'm going to go back
• 56:08
  to the multi-language topic. We created a Python file to
• 56:15
  recognize regular Expressions on our repository.
• 56:18
  So, we download the, we downloaded the repository, we
• 56:22
  dyed into each repository individually with a simple form,
• 56:26
  then we, we search for certain keywords, like past PWD.
• 56:34
  Password, etc. Etc, etc.
• 56:37
  So we we found all of the possible.
• 56:43
  Security breaches. And I say possible because
• 56:46
  sometimes we had like password equal password so that was not a
• 56:52
  security breach but we managed to get all of the possible
• 56:56
  security breaches and then in with that.
• 56:59
  But when somebody scripting we automated that, and then when
• 57:04
  everything was ready and everything was clean, we the
• 57:08
  armor down. We did, we did some reports on
• 57:12
  that. So, So at the end of the day,
• 57:14
  when when I run the, the project this project goes and takes all
• 57:20
  of the data in our workspace and on bitbucket, and cleans it, and
• 57:25
  then push it back to, which forced the clean repositories
• 57:32
  and then outputs a report. We how many security breaches
• 57:37
  has every every repo? How many there are left?
• 57:41
  How many there are left that we need Still take a look into
• 57:45
  because when you, when you use this, you cannot change your the
• 57:51
  main branch or Master Branch you do, you cannot change your
• 57:54
  production Branch because you will break everything.
• 57:57
  So if the problems are still there, you need to go ahead and
• 58:02
  fix it. Commits something new and then
• 58:05
  clean the game. So everything here, we did it
• 58:08
  with multi-language develop development and we just use the
• 58:14
  right tool to do the the best tool to do the job that we
• 58:19
  wanted to do in that specific section, incredible.
• 58:23
  Dude, this has been more than I bargained.
• 58:27
  For I want to thank you so much for coming on here.
• 58:30
  Head teaching me so much that I did not realize beforehand.
• 58:36
  I mean, this little foray into security was beautiful.
• 58:40
  Thank you so much. We're going to end it here.
• 58:42
  And in case anyone wants to get a hold of you.
• 58:46
  I know you do stream. If somebody wants to watch you
• 58:49
  streaming, where can they find them that?
• 58:52
  Yeah. Yeah, we stream with two friends
• 58:57
  from two co-workers and we usually talk about data science
• 59:01
  related topics. Last time we talked about Great
• 59:04
  Expectations and tomorrow we are going to talk about the DPT 3
• 59:10
  and so on. So on we talked about a lot of
• 59:13
  things related to that and the our twitch channel is called and
• 59:18
  coders with a underscore and underscore coders.
• 59:22
  We also About YouTube channel with just one video we are
• 59:25
  working on that. We're very very busy certain
• 59:29
  gotta start someone yeah. At least there's one.
• 59:34
  Excellent. Well thanks man.
• 59:35
  This has been awesome. Yeah, I'll also link.
• 59:38
  Meaning, if somebody wants to connect the week, we will leave
• 59:42
  all these links in the description in case anyone wants
• 59:44
  to check it out. Yeah, thanks a lot for the
• 59:46
  invite. Thank you so much laughing.
• 59:49
  See.