Latest / Agentic Conversations (formally mlops.community) / MLOps.community meetup #9 with Charles Martin - 10 Years Deploying Machine Learning in the Enterprise: The Inside Scoop!

Transcript

MLOps.community meetup #9 with Charles Martin - 10 Years Deploying Machine Learning in the Enterprise: The Inside Scoop! · Agentic Conversations (formally mlops.community)

• 0:02
  Hey, speaking of the slack. So everybody that's jumping in
• 0:06
  right now. If you didn't hear me earlier,
• 0:09
  we have a slack Channel 2. Ml Ops slack Channel.
• 0:13
  And this is a place where we can talk about anything and
• 0:17
  everything. Ml Ops.
• 0:18
  You can also DM me directly like, some people are doing
• 0:22
  right now telling me to cut my beard or take my glasses off.
• 0:25
  They looks like they look like they're not doing anything
• 0:29
  special, so So yeah, get into slack.
• 0:31
  If you're not in there already. I think Chris will put it up in
• 0:34
  the chat and just let people in now and then, I'll, I'll put it
• 0:38
  up in the chat momentarily. Cool.
• 0:40
  Perfect. Yeah, I'm getting some funny
• 0:43
  slack messages. Every time I come on and we
• 0:46
  start this, it seems like I get some funny slack messages from
• 0:50
  people. So your beard just keeps getting
• 0:52
  longer. So They said it was the
• 1:00
  highlight of their life. So yeah, nobody ever, really
• 1:05
  knew that Jesus was into ml. Huh.
• 1:07
  I'll stop sharing my screen. So you guys get a picture of all
• 1:11
  of us. You got the slack Channel up in
• 1:14
  the webinar chat. We look like, we got a critical
• 1:17
  mass here. Let's go ahead and get started
• 1:19
  today. I'm super excited for what we've
• 1:22
  got over here. Joined, by Doctor Charles
• 1:26
  Martin. He is one of those rare humans
• 1:29
  that That is super smart in. What is it?
• 1:33
  Organic? Chemistry knows a little bit
• 1:35
  about software engineering. He knows a little bit about
• 1:37
  machine learning. He's got all of this to be able
• 1:40
  to wrap into one. He's given classes and lectures
• 1:44
  at Cal Berkeley and at Stanford, so we are very lucky and you
• 1:50
  know, I was talking to my wife about this the other day about
• 1:54
  what's the value that somebody will get by coming to a meet-up
• 1:58
  and listening to it in real time.
• 2:00
  I'm and the thing that we agreed upon is that the difference
• 2:04
  here, instead of you watching. This on YouTube is that you guys
• 2:08
  get the opportunity to ask questions.
• 2:10
  So really I encourage you to take advantage of that.
• 2:14
  I've got a bunch of questions, but you're here with us you have
• 2:17
  this amazing human in front of you, talking about all this
• 2:21
  great stuff and we're going to be talking about ml Ops and
• 2:25
  really getting into his last 10 years.
• 2:28
  But before we do that, At go ahead and grab whatever Your
• 2:32
  Favorite Beverages and sit back because we got a little story
• 2:35
  time. I out of respect for you,
• 2:37
  Charles. I got a tea because I know it's
• 2:40
  early where you are. So really nice II.
• 2:43
  We drag the, you know, the super organic sodas here so we do it.
• 2:48
  Yeah. Because you are in San
• 2:50
  Francisco, California. Cisco.
• 2:52
  That's right. Yeah.
• 2:53
  So the I can't speak for the rest of the crew and how they
• 2:57
  are drinking or what they're drinking on.
• 3:00
  But let's just get started with a little background information
• 3:04
  on how you got into Tech, how you went from being in organic
• 3:09
  chemistry. And then you went into Tech and
• 3:12
  then sharks mentally got to em out.
• 3:15
  Sure. So, my, my background is really
• 3:18
  I have an undergrad in math chemistry and essentially
• 3:21
  physics and I did my PhD work in theoretical chemistry.
• 3:27
  So theoretical and the reason I did that is because when I I was
• 3:30
  going to go in the software. But when I got to Ohio State,
• 3:34
  the Ed computer science department of Acts machine and
• 3:37
  the chemistry Department had a great super computer.
• 3:40
  I'm like, okay, this is and I and I had done a lot of
• 3:42
  chemistry. And physics already, when I
• 3:44
  started college. I had already done all my
• 3:46
  organic chemistry. I started, I was 15, my
• 3:49
  grandfather used to own a pharmaceutical company.
• 3:50
  So I was really interested in chemistry.
• 3:52
  So I did all that stuff. So I got the college.
• 3:54
  Okay. I want to work in.
• 3:56
  I was not a very good lab chemist.
• 3:59
  But I was really good at math and programming.
• 4:01
  So I went into programming the supercomputer and I did that for
• 4:05
  a long time and I got a PhD and did a postdoc and my post, my
• 4:09
  PhD was in basically quantum chemistry.
• 4:13
  And my postdoc was in statistical physics and neural
• 4:15
  network biophysics bio physics. Theoretical biophysics.
• 4:18
  So neural networks and Neuroscience.
• 4:22
  So, you know, at that time, that was when the, you know, when I
• 4:27
  was a grad student is when the internet really hit Mosaic came
• 4:30
  out and I realized, okay. Look, this is the thing to do,
• 4:36
  because basically academics with dying, you know, is the end of
• 4:39
  the Cold War. So I came out to Silicon Valley
• 4:41
  and work with my New advisor, son-in-law, who had a search
• 4:44
  engine company. Unfortunately, it was not Google
• 4:48
  but, you know, I sort of just had to jump in and I realized
• 4:52
  that all the stuff I learned that like I've been doing a
• 4:55
  probably 10 years of scientific Computing and quantum,
• 4:58
  mechanics, and statistical physics, this kind of stuff, but
• 5:02
  an industry, you know, all the technology is totally different.
• 5:05
  So I basically had to start from the ground up and I just sort of
• 5:08
  took the opinion. I should have the same level of
• 5:11
  knowledge as any Peter science student at Berkeley or Stanford.
• 5:14
  So, I started reading the books trying to study, it was kind of
• 5:18
  a weird thing because, like, people would ask you, do you
• 5:20
  know, algorithm. Of course, I do algorithm like
• 5:22
  in solve eigenvalue problems. I can do systems of linear
• 5:24
  equations. I can do nonlinear.
• 5:26
  Maybe Monte Carlo, but what about sorting?
• 5:28
  I go what's a sort algorithm. Why would I want to do that?
• 5:31
  That's totally stuff like, whoever writes their own sort.
• 5:33
  Algorithm, right? I've never done anything.
• 5:35
  So, you know it. So you go a little crazy at you.
• 5:38
  Get these interviews, and I know I don't do web engineering.
• 5:41
  So I talk to you guys about. Why do you always ask questions
• 5:44
  about recursive algorithms? I have never written a recursive
• 5:47
  algorithm for anything I've ever done in 20 years ago.
• 5:50
  We write them every day. I go.
• 5:52
  Well, you're not doing machine learning.
• 5:53
  That's for sure. You know, your stuff is just not
• 5:55
  used. So it's a very different
• 5:58
  background coming in from computational physics and
• 6:01
  chemistry. And I think what's happened is
• 6:04
  that That was something you did 20, 30 years ago.
• 6:08
  I'm a tall guy. 30 years ago, know, what I am now, it's
• 6:14
  mainstream. So what happened was the stuff I
• 6:16
  worked on which was basically you needed a, you know, twenty
• 6:21
  ten million dollars supercomputer to do.
• 6:24
  Now you can do on your iPhone, you do an edge computer.
• 6:27
  So the stuff we did is like there's this great Foundation of
• 6:32
  all this stuff and now I'm finally able to leverage it.
• 6:35
  It and so, but, but while I was waiting for that to happen, I
• 6:39
  had to learn all this other stuff out.
• 6:41
  I do Java Swing programming, how to work.
• 6:44
  How do you Chef? You know, what is puppet, you
• 6:48
  know, this kind of stuff you. How do you know?
• 6:51
  How do you get? You know, what?
• 6:52
  If you know, all these different systems?
• 6:54
  And you basically, I was in the guts of it for many years
• 6:58
  working as a contractor gives, you know, you propose an idea to
• 7:03
  someone but it's got to go in there.
• 7:05
  Action, how do I get this thing in the production?
• 7:07
  And then, you know, over a period of time?
• 7:09
  You have to you learn all this stuff, you realize, you know,
• 7:10
  okay, maybe, you know, now I've sort of come full swing.
• 7:16
  So now I'm doing consulting work.
• 7:18
  I'm doing, I'm now a scientific consultant for the page family.
• 7:21
  So they have this anthropocene institute is a global climate
• 7:24
  change Institute, where engine ocean water air, quality energy.
• 7:28
  So primarily think like nuclear power, Quantum, Computing.
• 7:32
  And now, of course the Yeah, you helped out like I wanted to, so,
• 7:38
  one of the things go ahead. No, sorry.
• 7:47
  So one thing we were doing is time to look at some of the
• 7:50
  people doing clinical trials, at Stanford and say, look there
• 7:54
  these AI search engines for covid-19.
• 7:56
  Why don't we see if we can get you using one of these?
• 8:00
  And so like they're a couple people who built them.
• 8:02
  There is an open source version of this.
• 8:04
  Now, Google yesterday just released one.
• 8:06
  And so we're going to go talk to some of the people like the head
• 8:09
  of translational medicine and talk to him about, you know, is
• 8:12
  this something to be useful for you?
• 8:14
  And, you know, I understand enough about molecular
• 8:17
  biochemistry and molecular biology to have a reasonable
• 8:20
  conversation with them. Yeah.
• 8:22
  I couldn't run a clinical trial, but I can read the literature
• 8:25
  and I understand it was like, I did this stuff as a postdoc.
• 8:27
  So, Other things were looking at in you are there so that sort of
• 8:32
  like step one. Can we just get them using
• 8:33
  technology that they don't really use here?
• 8:36
  They're not used to using? Yeah, they don't they don't
• 8:40
  they're not aware of like advanced search engine.
• 8:42
  Then, you know, they're not really doing computational
• 8:45
  biology. So these are things were trying
• 8:47
  to. See are there Technologies?
• 8:49
  We can just give you and help you just use and research them.
• 8:53
  So that's a lot of what I'm doing.
• 8:54
  The rest of it is more. Also because it's a family
• 8:58
  office then they deploy Capital. So I helped that projects.
• 9:02
  So, you know, people who didn't catch that, he's working with
• 9:06
  the page family. The first name of that page guy
• 9:09
  that you're working. Well, I work with Larry's
• 9:11
  Brother, people don't know, their cage that you've probably.
• 9:15
  So Larry is actually his mentor and he started a company to and
• 9:21
  it was like I do Yahoo groups. So these guys so he runs
• 9:25
  basically this part of the and we have an it's bit more.
• 9:29
  If it's a, the main interest is just like some of the stuff
• 9:35
  they're just making PPE, for example, as make it BB and give
• 9:38
  it a people like can we find people or one project we've
• 9:42
  talked to is looking at some of the hospital data.
• 9:46
  Can we? Help me emergency rooms, figure
• 9:50
  out why certain people are being admitted and others.
• 9:53
  Aren't To, you know, to the ICU units and this is, you know,
• 9:57
  like these are just where can we make some sort of impact?
• 10:02
  And and that's, you know, and so try to use AI but you know
• 10:06
  chemistry as well or just yell, you know, laser printer, you
• 10:10
  know, 3D printers, you know, to. So, they're really trying to
• 10:13
  make a they're really trying and prior to that.
• 10:15
  I worked with him in the past, really looking at, you know,
• 10:17
  nuclear energy policy, you know, really understanding that like,
• 10:21
  if you want to fix climate change, you need nuclear it
• 10:24
  away. To do and and quantum, chemistry
• 10:28
  and nuclear physics are very, very similar.
• 10:32
  So I'm not an expert in nuclear physics.
• 10:33
  By understand enough of what's going on.
• 10:36
  I it's a very like we basically have it's the kind of stuff I
• 10:40
  was doing, is basically the same field and at a base level.
• 10:45
  That's what you know, but now I you know, I make Spam on the
• 10:48
  internet. So I have a client for example,
• 10:52
  they're great guys out in Slovenia and they have a product
• 10:55
  that generates fake text. And it's great.
• 10:58
  Like they can make like weight loss articles, you know, and
• 11:01
  stuff like this and they saw him quiet, you know, I consulted
• 11:04
  with them. And I know they they basically
• 11:06
  wanted to learn AI. They taught themselves II and I
• 11:09
  just sort of like, was there, as kind of like their Mentor,
• 11:11
  helping them, and they've got this product working and I just,
• 11:13
  you know, tried to help them really just, you know, they need
• 11:17
  to do it themselves and I just kind of help them drive the car.
• 11:20
  Like I keep him from driving off the edge of the road.
• 11:22
  Yeah. Yeah.
• 11:24
  And that I really want to get into this, these Slovenian guys
• 11:29
  in the, their fake news or whatever.
• 11:32
  I'll send you a link to the product.
• 11:33
  We have a working product last night.
• 11:34
  Talking, they 22 paying customers, Evie.
• 11:37
  It's a great example of me. Look, here's what they decided
• 11:41
  and they look, I can't one of things I can talk about.
• 11:43
  I think with ML Ops or any AI project is, what's the
• 11:47
  difference between a successful project in a failed project?
• 11:51
  Exactly. And the reality is you have to
• 11:54
  want to learn the client has to want to learn and everyone
• 11:58
  working at the client has to want to learn.
• 12:01
  So, I've had climbed these guys. They basically taught
• 12:04
  themselves. They are, they have the fast AI
• 12:06
  class, which is that USF. We're just around the corner,
• 12:09
  here out my window, and they talked themselves in their own
• 12:12
  networks, and they asked me to come in, and just help them
• 12:16
  understand at a higher level, what they were doing.
• 12:19
  And it's almost like, I mean, you know, II could help them but
• 12:25
  they did it and I just sort of guided them to look.
• 12:27
  This is what I think about this. This is why I think about that.
• 12:30
  There are no really writing answered quite like one of the
• 12:32
  things I did is I helped them build a tool, which would
• 12:35
  analyze the coherency of the text.
• 12:39
  And I say, look, you can use machine learning techniques to
• 12:41
  analyze text, coherency, and that will help you understand
• 12:45
  whether what you're generating is realistic text or non
• 12:47
  realistic text. So there are other metrics other
• 12:49
  than like perplexity Mac because you can't Have like how do you
• 12:52
  test it? Right?
• 12:54
  How do you test something that requires a human to look at
• 12:56
  every single example, that's not so easy.
• 12:59
  Yeah. And so so I had to give them
• 13:02
  like twice in the airway and so we work with their staff and
• 13:06
  they were very successful, the projects that are not
• 13:09
  successful, are the ones were like, what those are Is How We
• 13:12
  Do deployment and you have to fit your system into our
• 13:15
  deployment system. So spend the next six months
• 13:17
  figuring that out. Well, you told me don't want to
• 13:20
  do it, you know. I want to keep doing what
• 13:22
  they're doing in that doesn't work.
• 13:25
  You know, you can't want you happen.
• 13:28
  It's not like I know everything but you have to want to engage,
• 13:32
  and if you don't engage, you know, you're not, you know,
• 13:36
  you're not going to do it because something you told me
• 13:39
  yesterday that I thought was pretty mind-blowing.
• 13:42
  Was that ml? Ops is not a technology problem.
• 13:46
  It's an organization problem. Oh, absolutely.
• 13:49
  Absolutely. It's because what?
• 13:51
  His I mean look, why do we have mlr?
• 13:55
  Why do we have devops? If you think of go back to web
• 13:58
  Engineering in web engineering is quite painful, you know, you
• 14:02
  would you create a wise old I am you create a war file and
• 14:05
  jumping into the EPA ejb server, you know, and you know, there
• 14:09
  are a lot of things that developers would do over and
• 14:11
  over and over again. And there is, you know, if we
• 14:14
  just automate this, our lives would be a lot easier, you know,
• 14:18
  we don't have to keep copying this file to that far.
• 14:20
  We going to do a recap. I meant, every time you make a
• 14:22
  change, the database you want to read the for the code run that
• 14:25
  you run the test Suite, read the play, the code, push it to the
• 14:29
  web, you know, these kinds of things.
• 14:32
  And you know, it's the this is when you go into a larger
• 14:39
  organization. They usually have some sort of
• 14:45
  deployment system built already. And one thing I know that most
• 14:50
  of them are homegrown like you don't have, you know, a lot of
• 14:55
  you there is no other aren't really ml products that people
• 14:58
  might get chef but Chef is a specialist tool.
• 15:02
  It's not a GUI tool. It's not like serverless on the
• 15:04
  Amazon. So you have to and what happens
• 15:07
  is The APA systems are designed to deploy the things they've
• 15:14
  been used to building. And machine learning is very
• 15:17
  different. The stack is completely
• 15:19
  different. The stock working.
• 15:20
  The hardware requirements are different.
• 15:22
  The memory requirements are different.
• 15:24
  The swap requirements are different.
• 15:26
  The you has different artifacts that you don't have in web
• 15:31
  engineering. The deployment systems are, I
• 15:35
  mean fairly mean it's not that hard to deploy a flask API.
• 15:41
  It's not a cop. Most of these things are not
• 15:43
  complicated, web Engineering Systems that were not like a
• 15:48
  giant Ruby on Rails app. It's just the emotions were just
• 15:50
  microservices, but getting that thing to work in the
• 15:56
  environment, you're in the environments are not structured
• 16:01
  to Handle this new workflow and most of the work you end up
• 16:07
  doing is just trying to figure out how to make something hit
• 16:10
  within the workflow and the more stubborn people are about it.
• 16:15
  The harder it is, which was your next question.
• 16:19
  They're extremely stubborn. Because look, most people are
• 16:22
  not in a company boost and I'm talking about Enterprise has not
• 16:26
  not so much sorrow startups have the same problem startups come
• 16:28
  in and the engineers will go. This is how we do software
• 16:31
  engineers. Like, why are you doing that?
• 16:34
  This is how we do. So, you know, they know what to
• 16:36
  do, but there's no critical judgment as to is this the right
• 16:42
  thing to do. So you end up doing goofy
• 16:44
  things. Let's do some details.
• 16:48
  Like for example, Devops is about provisioning.
• 16:52
  Right? You want to provision something.
• 16:54
  So you, you have to go and ask someone the permits get
• 16:56
  permission to do the provisioning.
• 16:57
  Okay. Well, we go into a client.
• 16:59
  One of the clients had the largest Hadoop store in the
• 17:01
  world. I just had to system in the
• 17:04
  world. Can you give us a machine to run
• 17:07
  our the deploy our app? Yes, no memory.
• 17:11
  Why would it need memory? Hadoop doesn't use memory.
• 17:13
  It has no swap. He gave us a machine with no
• 17:17
  swap. Okay, what am I supposed to do
• 17:20
  with this? So, you know, I have to hire
• 17:23
  someone who's especially the guy when I would work with me.
• 17:26
  He actually worked with Ken Thompson invented the Unix
• 17:28
  operating system. And he's gotta go, you gotta go
• 17:30
  and figure out what's going on here.
• 17:32
  Another example you provision us a machine, especially in large
• 17:36
  organizations provisioning as part of the budget.
• 17:38
  So you can't provision your own machines.
• 17:40
  So they provision a machine. Okay, in the old days when you
• 17:44
  installed python, you had to turn off.
• 17:49
  V. The Traveler on the CPU in order
• 17:56
  to optimize the blast libraries because we install Python and
• 17:59
  numpy. There are Fortran libraries in
• 18:01
  order to optimize the Fortran Library.
• 18:03
  So you get correct performance. The installer would turn on,
• 18:06
  would ask you to turn off the throttle.
• 18:08
  We need root privilege to do that.
• 18:11
  So that doesn't exist in the web engineering world.
• 18:13
  When you do deployments, you don't need root pledge to turn
• 18:16
  off the throttle. So that's a huge problem and it
• 18:19
  creates performance bottlenecks. And these are things I'm like
• 18:22
  now you or today. Same thing, like today.
• 18:26
  I worked with a client. They had built their own
• 18:29
  internal Heroku system. I'm working Roku like tenure,
• 18:33
  and I was doin Ruby on Rails, 10 years ago, Road, you know, but
• 18:36
  yeah, they had built as a copy basically Heroku.
• 18:39
  And and of course, there's no documentation.
• 18:41
  It doesn't work correctly and the others, no support like it
• 18:44
  hasn't been maintained. So you're like, okay.
• 18:46
  I want to deploy condom. I wanted to play conda.
• 18:50
  Oh and Heroku, how do you do that?
• 18:53
  It turns out that that's really painful conda.
• 18:56
  We'll just use pip install our go, that defeats the whole
• 18:59
  point. You don't understand.
• 19:01
  You just don't understand the stack.
• 19:04
  So you have a stack of things work on days, what you use for,
• 19:08
  dependency management and they say to you.
• 19:11
  Well, your we built you this devops system, but forget about
• 19:14
  all the dependency management, you're doing, just redo it all
• 19:17
  from scratch because it doesn't fit to our environment.
• 19:20
  What was the point of having Dev Ops in the first place?
• 19:23
  You didn't make my life easier. You made it harder.
• 19:27
  You just give me an Ubuntu machine and I'll just write a
• 19:29
  bash script that deploy it. You know, that would take me a
• 19:33
  day, not six, not six weeks. This is typical.
• 19:38
  This is typical. This is what we see internally
• 19:41
  and the problem is that look there's a vein I do, you guys
• 19:45
  know vapnik Optic is a famous machine learning guy from like
• 19:49
  10 years ago. He was a Russian team over here.
• 19:52
  Wrote a sort of common. His Manifesto on machine
• 19:54
  learning that big thick book on back next book and then he
• 19:57
  invented the svm. So he invented logistic.
• 20:00
  We reinvented logistic regression so, you know here but
• 20:03
  it was a guest VM was really popular back then like thought
• 20:05
  to be the next big thing. In fact that you used to say,
• 20:10
  One should not solve a problem by posing an intermediate
• 20:15
  problem, which is harder to solve.
• 20:21
  System. You see in the Enterprise, make
• 20:25
  it unbearably hard to get 10 lines of code into production.
• 20:31
  And that's why it's an organizational issue because it
• 20:34
  gives what's happened is this this scaffolding is built and
• 20:37
  there are people. And again, it's not all.
• 20:39
  It's not like you went and bought a product and usually
• 20:41
  these are you know, the scaffolding is there.
• 20:43
  You have to get access to the database.
• 20:48
  Well, when you do machine learning you're going to pull
• 20:50
  all the data out of the database.
• 20:53
  Really. Yeah, I'm gonna do a query that
• 20:55
  this I have 12 joints and I'm going to pull them from your
• 20:58
  production database, you know every day.
• 21:02
  Well, you're going to see if you're working with
• 21:03
  International Company. You shut, you get a block of but
• 21:05
  obviously, you can't do that. So this they've never seen
• 21:08
  things like this. Like, why would you do such a
• 21:09
  larger? Because we need all the data.
• 21:12
  Oh, well, then the data needs to be in a replicated system, but
• 21:15
  the replicated system isn't involved many companies.
• 21:18
  It's not a real replica. It's missing stuff.
• 21:21
  How many times we got in the clients in?
• 21:22
  The replica isn't actually a replica.
• 21:26
  Give us a backup of the data. We work with clients.
• 21:28
  We try to do a backup. Oh, they, you know, they
• 21:31
  somebody suggests using your try to use the Oracle backup.
• 21:35
  It's a 700 page book of how to do a backup.
• 21:39
  So you find that there's some guy running, you know, Java Java
• 21:43
  program on his laptop at night every day, backing up pulling
• 21:48
  all the database onto his home computer to back it up.
• 21:51
  And that's the guy you got to find to give you the data.
• 21:53
  So it just doesn't work. Like the ml says they're not
• 21:56
  designed heal, orb provisioning, an AWS buckets.
• 22:02
  I would like innate I've worked when I started this 10 years
• 22:05
  ago. I couldn't even get.
• 22:06
  How do we provision a bucket? But you have to, you have to say
• 22:09
  Of Amazon on your own personal credit card and the company will
• 22:13
  reimburse you what? I can't get a company credit
• 22:17
  card. I can't spend $500.
• 22:20
  No. No, we can't do it.
• 22:22
  I mean, I literally had arguments with the cfo's of
• 22:25
  companies in the in the city. Like what are you doing?
• 22:27
  Like, you know, wild and you have like one guy in the company
• 22:32
  who has the AWS card, right? And he you gotta find him to get
• 22:36
  the keys. He had to use his card because
• 22:39
  you know, the way AWS works you if you School of different, you
• 22:42
  know, different keys, on different, credit cards, you
• 22:46
  can't remain on it's very hard to cross communicate.
• 22:49
  So these are things that are like, that's what I mean by its
• 22:51
  organizational. There's usually some person
• 22:55
  whose job it is to manage. Each of the little pieces in the
• 23:02
  deployment process and since and you have to go and find them and
• 23:06
  then you have to figure out what goofy thing is going on.
• 23:09
  What are they doing that? You want to do something
• 23:11
  different and you it just makes it very very hard and it's
• 23:16
  easier just to do everything from scratch.
• 23:20
  And so talk to us a little bit. I know you've got some stories
• 23:24
  of when shit hit the fan. Can you give us some of those?
• 23:29
  Because I know you got some good ones.
• 23:31
  Who look the most famous story is demand media.
• 23:33
  I have a case study out at the, UC Berkeley at the hospice and
• 23:36
  school. So we had deployed an algorithm
• 23:40
  in production. That what we were doing is it we
• 23:44
  basically I legally, I can't describe it.
• 23:48
  We had the ability to predict Demand on Google.
• 23:52
  So I developed an algorithm is called the was called the answer
• 23:56
  Factory. And there was a very famous
• 23:57
  wired article. I think 2009 about this where we
• 24:00
  had the ability to predict what people were searching for on
• 24:03
  Google and we can predict how many clicks you would get on a
• 24:06
  webpage. So, we built an algorithm that
• 24:10
  allowed us to say Say if you mother but what do you think the
• 24:13
  most popular searches on Google like long tail search.
• 24:17
  You don't want to know what I think it is rated G rated G.
• 24:24
  How to get rid of flabby arms? Right,
• 24:39
  weight loss, cooking recipes, things like this, you know,
• 24:43
  basic stuff. So we could predict how much
• 24:48
  traffic. The web page would get before we
• 24:50
  wrote it. So I basically
• 24:53
  reverse-engineered SEO and this was prior to the Panda and
• 24:56
  Penguin Updates. This is back when Schmidt was
• 24:59
  still the CEO of Google. So, of course we demand media.
• 25:02
  We put this thing into production.
• 25:06
  And yeah, we're at one point. We were generating a hundred
• 25:10
  thousand pieces of content the month, hundred thousand piece of
• 25:13
  content a month and then Google changed CEOs.
• 25:16
  And they just, you know, Larry came in and then they redid All
• 25:19
  the algorithms, you know, they introduced the panda update, the
• 25:22
  penguin updates and all our content then got down, you know,
• 25:25
  all the content got delisted. How do you undo, what you've
• 25:32
  done? How do you get that?
• 25:33
  You take content down, right? I mean, which content do we take
• 25:37
  to? How do we know what has been
• 25:38
  done? So, you know, you have this
• 25:40
  problem where you're generating content, you have this Factory
• 25:43
  generating content and all of a sudden the primary algorithm for
• 25:48
  what the sides what to generate changed.
• 25:52
  And everything. How do you undo?
• 25:54
  What has been done? Do you take content down?
• 25:57
  You know, that contents been a mercerized.
• 25:59
  So it's a huge. It was a huge problem.
• 26:01
  Like and how do you figure out what happened?
• 26:04
  Like, how and one of the things our Google, not there, things
• 26:07
  people don't realize about Penguin and Panda.
• 26:12
  If you guys do SEO, you know, you know what these things are.
• 26:14
  When they changed all the way they do SEO, not only did they
• 26:17
  change the SEO algorithms, but they, they stop giving you the
• 26:22
  inbound query because it used to be that when you did a search on
• 26:27
  Google on your web page. Google would tell you what the
• 26:28
  search was like what search was given to you.
• 26:31
  They stop telling me because if you know what the search is, you
• 26:35
  can front-run what people, you can figure out what people are
• 26:37
  searching for because people basically search for the same
• 26:39
  thing over and over. So, you know, from a deployment
• 26:43
  perspective, how do you undo that?
• 26:46
  So that was a nightmare of. So, I have a whole case study of
• 26:48
  this, we talk about the hot school.
• 26:50
  It from an ml Ops perspective. If you want more of that kind of
• 26:54
  War Story. I mean, yeah, look when you look
• 26:58
  at what is a machine Learning System where you basically do
• 27:02
  the machine ordering systems have three components.
• 27:06
  You pull data from a database, you train a model, you deploy
• 27:11
  microservice. That's basically it.
• 27:14
  That's how most of these systems work.
• 27:16
  Okay. So, you know you deal with the
• 27:18
  CTO of guys, I work so well, why don't you just run the model
• 27:21
  inside SQL Server itself? Okay, that's a crazy on.
• 27:28
  It's a complete misunderstanding of what's going on.
• 27:30
  Right? Like, am I supposed to like,
• 27:32
  like, you understand? Like how would you this?
• 27:34
  Is the thinking, right? Just run, run the model inside
• 27:37
  SQL. Server and train it.
• 27:39
  How would you get the data? Am I going to do a massive join
• 27:42
  every hour on the hour, right? Am I supposed to design?
• 27:46
  Then? You say well I have to nail
• 27:48
  design. So is it get the design
• 27:51
  incremental data joints like right to do it?
• 27:53
  Incrementally, we do incremental update count, heart is the
• 27:55
  design joins that do that incremental updates and make
• 27:58
  sure you get everything right. So, you know you end up.
• 28:02
  So this is typical like, like, okay, how do I get to date?
• 28:06
  Then? Of course, whenever you look at
• 28:07
  someone data, it's a mess. There's no data Ops what people
• 28:12
  need is data Ops, like making sure that the data is actually
• 28:15
  what you say it is. Why are there, you know, you try
• 28:20
  to run something very basic like a, you know, like a like a
• 28:27
  Stringer placement on a reg ex and somebody inserted star star
• 28:33
  into their name. And the star star breaks, the
• 28:36
  string, breaks, the regular breaks the RedHawks, right?
• 28:40
  What happened here? You have to go and sort through
• 28:42
  all the data and figure out what data point broke.
• 28:46
  What happened? That doesn't happen in normal
• 28:48
  systems. You don't have data brick,
• 28:52
  making your application. You know, we like Ruby on Rails
• 28:55
  has validators, right? That's what you'd be of and
• 28:58
  you're only that the data, breaks the application.
• 29:00
  It's because the user, it happens at the user entry point,
• 29:03
  so that can be blocked at you. The JavaScript level or at the
• 29:06
  rails. It was blocked at the validator
• 29:08
  level, but not in these systems and these systems you're looking
• 29:12
  at data that has never been validated.
• 29:14
  So that's a huge problem that breaks and ever.
• 29:18
  And so, how do you automate something?
• 29:21
  This is the problem with Ops from the wisest, an Ops issue,
• 29:23
  right? Why is it options you?
• 29:26
  How do you automate something where you have no control over
• 29:30
  the input data stream? How you automated?
• 29:33
  You're asking me to build an Automated system and have it run
• 29:36
  into end automatically into end. But you have no control
• 29:41
  whatsoever on the data flowing through it.
• 29:43
  And you expect that to work and and there's just no concept of
• 29:49
  like this is the argument. You have with people, you try to
• 29:52
  work within an organization and, you know, the guy who's running
• 29:55
  the database or they're like the guys who do the deployments
• 29:58
  argue. Everything has to run into end
• 30:00
  as why. That's crazy.
• 30:03
  There's no way to guarantee. That means I have to build in
• 30:07
  order for that to work. You're saying, what?
• 30:10
  And what's the point? Right?
• 30:11
  What is the point of doing this? Well, this is how we do it.
• 30:13
  This is how you build systems. Okay.
• 30:16
  We're trying to run an experiment.
• 30:18
  I want to train a model. I want to put in your production
• 30:21
  system, and I want to run an experiment, and I want to see if
• 30:25
  the model works. If it doesn't work.
• 30:28
  I may need to do something completely different, right?
• 30:30
  So, a large part of the model is the data that you're collecting.
• 30:34
  Because the A model, depends on the data set, right?
• 30:37
  So what I'm really doing is saying, does the data in your
• 30:42
  system have value. What we're trying to do, can I
• 30:46
  generate revenue from this data stream?
• 30:49
  And I work with a company. All they are all the old.
• 30:51
  We do experiments here. This, we have a whole system for
• 30:54
  doing experiments. Okay, great.
• 30:55
  Great, you know, you're doing organic chemistry experiment
• 30:58
  that I want to do a physics experiment.
• 31:00
  You know, II don't want to go into the chemistry lab, the new
• 31:03
  physics. That's the problem.
• 31:04
  You know, you're not doing the same experiment, you know, you
• 31:08
  know, I can't do that. So you, you realize that All I
• 31:16
  wanted to pull, I want to just build a model in the point is
• 31:18
  last my hand. I just want to push the data by
• 31:21
  hand and check. It should take like a week,
• 31:24
  right. Six months later.
• 31:28
  Staff guys are involved. Everyone's a how do we automate
• 31:30
  all day? Everything months.
• 31:32
  Tommy, literally month. This is month.
• 31:35
  This isn't like a week. This is months of arguing to try
• 31:38
  to do something and you have to understand that if you can't
• 31:43
  control your data stream coming in.
• 31:47
  How do you automate? How do you automate it?
• 31:51
  How do you say it? How do you say it the boys?
• 31:53
  Especially if you're saying to me now and this is typical,
• 31:56
  you're going to test your data. On the replica.
• 32:02
  But it has to work in to end on the production data that you've
• 32:08
  never seen. So now you're asking me to build
• 32:12
  a system and deploy it on his on inside a live environment where
• 32:18
  there's no quality control on what's happening.
• 32:21
  This is the problem with with ML Ops is that it's not designed.
• 32:26
  And again the the web engineering guys don't have this
• 32:28
  problem because they trap all the errors at the user
• 32:32
  interface. Bottle or they trap it in the
• 32:35
  way in which the database, the query of the database table is
• 32:38
  designed to not allow those errors but a machine-learning
• 32:43
  you're not doing that. You're looking at somebody
• 32:44
  else's data. So you have all the time is
• 32:47
  spent. And again, this is some or
• 32:50
  here's another. Good example, another good
• 32:52
  example. I want to build a model for a
• 32:55
  recommender. I want to recommend people.
• 33:00
  So what do I do? I get your resumes and I run
• 33:04
  them through a clustering algorithm, right?
• 33:09
  Takes three seconds to think that up, you know, it's not a
• 33:11
  not enough does not theoretical physics.
• 33:13
  You know, I'm not, you know, simple idea, okay.
• 33:18
  In order for the clustering algorithm to work properly.
• 33:22
  I have to remove because it's an unsupervised algorithm.
• 33:28
  I have to remove common names because I don't want all the
• 33:32
  Charles's and to Cluster together because it doesn't
• 33:36
  matter to me that you have the same name as me.
• 33:37
  And what happens typically in these algorithms is that Lots of
• 33:42
  common words will appear that cause this thing to Cluster.
• 33:47
  where, You don't want that the cluster that way and examples.
• 33:51
  I was in finance. We built a model to this for
• 33:53
  predicting the stock market. And we were looking at basically
• 33:56
  clustering documents describing company.
• 33:58
  They have Polaris Industries. And Polaris motorcycles.
• 34:04
  And what happens is in the document, the word Polaris
• 34:06
  appears multiple times. So you have to remove it.
• 34:08
  Otherwise, it screws things up, you know, things stick together,
• 34:11
  that shouldn't stick together. Okay.
• 34:14
  Now, I can look at a data set. And I can kind of go through it
• 34:19
  by hand and I could remove all the common names and get the
• 34:21
  model to work. No, no.
• 34:23
  No. This has to be automated.
• 34:30
  I have to train an engineer on how to build.
• 34:38
  A system that can detect common names but not remove the names
• 34:43
  because you need to have some names, right?
• 34:44
  You don't want to remove company names because if people work at
• 34:46
  the same company that's important.
• 34:48
  So I have to train a machine learning algorithm to detect the
• 34:52
  names. I want to remove because the
• 34:54
  Clusters are not clustering properly, all because it has to
• 34:58
  run within the MLM system and this again.
• 35:00
  And and I have tried to take an engineer who is basically a
• 35:04
  great day to in like by working with Clay, a great data
• 35:06
  engineer. Yo, knows far more about
• 35:09
  kubernetes and stream sets and sequel that I do.
• 35:12
  And I have to teach him how to do this, because he can't
• 35:15
  automate the deployments. Whereas I could have done this
• 35:17
  by hand in 20 minutes. So do you feel the automation is
• 35:21
  something that's coming from the software engineering side.
• 35:24
  It's like, we need to automate everything because we had Phil
• 35:27
  winter on here, a few weeks back.
• 35:29
  And he was talking about how automation is the Holy Grail and
• 35:33
  we're always striving for automation.
• 35:35
  Why? And that was why are you
• 35:37
  striving for automation? Why?
• 35:40
  Why are you doing that? What is the goal of any project
• 35:45
  to get paid? What?
• 35:47
  What do companies do? They bill their customers.
• 35:54
  That's what I do. So, what is the goal of
• 35:58
  automation? Right.
• 36:00
  I work at a company. I come in as a consultant CTO
• 36:02
  hired me. The head of search who they
• 36:04
  brought in from. Google said to me we want to
• 36:06
  know everything that you're doing so that we can fire you
• 36:09
  and still be able to run the product.
• 36:12
  I'm like I'm not a prince. I'm not an employee, you know,
• 36:15
  you can sign a maintenance contract.
• 36:17
  I've been working with this company for 10 years, you know,
• 36:21
  it's just the way we want you out.
• 36:22
  We don't want you here. We want to extract every piece
• 36:24
  of knowledge that we, you have. And automated.
• 36:28
  Then we want to kick you out. That's the I'm like, how about
• 36:32
  if I tell you nothing and you just bill you right?
• 36:35
  I mean, it's like you haven't this adversarial relationship
• 36:38
  with people who think like this. That's like that's crazy.
• 36:41
  Like know what you want to know. Why are you automating, you know
• 36:46
  your goal as a machine-learning scientist, as a data scientist
• 36:51
  is to run experiments. You want to run experiments to
• 36:55
  figure out what's going to work because you I know what's going
• 36:58
  to work. So you have especially when
• 37:01
  you're dealing in a live environment with people.
• 37:04
  Search engines are a great example.
• 37:05
  People are searching for things that you want to figure out how
• 37:08
  to give them relevant information.
• 37:09
  It's actually a really hard problem and most people in the
• 37:12
  industry screwed up, totally except Google.
• 37:15
  Of course, this is what Google is based on, you know, they
• 37:18
  figured out search and that's why, you know, they run the
• 37:19
  world now, so you want to run experiments.
• 37:22
  So your goal if you're trying to automate something is to
• 37:26
  automate the experiment. A process.
• 37:28
  Now, once you get something that makes, and this is how you do it
• 37:31
  in trading environment. You work like I work at Black
• 37:33
  Rock. Okay, Block rock.
• 37:36
  I mean, if you guys knew black rock is Black, Rock is the
• 37:38
  800-pound gorilla on Wall Street.
• 37:40
  They manage they have trillions of dollars in asset management
• 37:44
  and I worked in the SAE Group which is the group that invented
• 37:47
  the the index fund. How did we do it?
• 37:49
  So I've worked in the largest financial institution in the
• 37:53
  world. It's not automated.
• 37:55
  That's not how they do it. They do.
• 37:57
  That way what they do is they they invent their algorithms.
• 38:01
  They put them into production quickly.
• 38:03
  They figure out what works and then they automated after.
• 38:07
  You know, it's going to make you money and if it stops making you
• 38:10
  money, and in the trading world, if it stopped, what?
• 38:13
  You don't want to maintain things at that, stop making you
• 38:15
  money because they make you lose money, because now you're
• 38:18
  trading on the wrong thing. So you've got to be able to get
• 38:21
  things into production and out of production quickly.
• 38:25
  That's your goal. But software Engineers that
• 38:29
  that's and you've got to think what part you need to automate
• 38:34
  and what part needs to be done by hand.
• 38:38
  And if your attitude is like, we want every engineer to be a
• 38:40
  commodity. You know that that's not how you
• 38:45
  know the group I worked in at BlackRock and 200 scientists
• 38:48
  200, phds, every single guy in that group.
• 38:51
  Caltech MIT Stanford, super neck, you know Moscow State, you
• 38:56
  know, every single person there had a PhD from the top place in
• 39:00
  the world and they were super smart.
• 39:03
  These are not people. You want to leave and take the
• 39:06
  algorithm with them because they will leave.
• 39:09
  They'll take the album go somewhere to make the money
• 39:10
  themselves. Nothing, I do.
• 39:13
  So it's a completely different mentality.
• 39:14
  Your goal is to keep them there and and you know at some point,
• 39:18
  you know, people always think is automation.
• 39:22
  Why are you doing it? Are you trying to make your life
• 39:24
  easier? Are you trying to commoditize
• 39:27
  the engineers? You can just kick them out
• 39:28
  whenever you want. Is it just sort of this mindless
• 39:31
  thought process or you really doing it to try to do quality
• 39:34
  control. Are you trying to make the
• 39:37
  system more stable? And I think people just don't
• 39:40
  ask those questions. They're just like this is how we
• 39:43
  do it. And you know, you don't know
• 39:45
  what you're doing. So you don't do it that way.
• 39:48
  How do you how do you automate green this thing about Ai and
• 39:53
  which is so important. This is all Green Field right
• 39:57
  things that we're doing. Now in natural language
• 39:59
  processing, for example, two years ago, didn't exist.
• 40:05
  Transformer models for natural occurring, Bert Elmo, this idea
• 40:08
  be able to generate fake text. That was five years ago.
• 40:13
  So as you're working in environments where things are
• 40:18
  changing very rapidly. Look at python.
• 40:21
  You work, you still working. I mean, imagine all the
• 40:23
  companies that automated python 2 7 Right Bank of America, you
• 40:30
  know, I've heard some horror stories about this.
• 40:32
  We automated fight now python 2 7.
• 40:35
  It's gone. Now you have When I was in the
• 40:40
  GI bride, a black rocks, a collision, they automated
• 40:43
  everything. They had two versions of pearl
• 40:46
  running. Remember Pearl Fork.
• 40:49
  He hasn't you guys may be too young to remember that?
• 40:51
  You're all young guys. Pearl used to be the thing to do
• 40:55
  and pearl Fork. And new Pearl is completely
• 40:59
  incompatible with back Purl, all Purl.
• 41:02
  So they had to me and everything is locked in.
• 41:05
  So, they had to redo everything. They had to then maintain two
• 41:08
  versions of pearl. So automation has problems.
• 41:13
  If or what about, you know, automation is problematic.
• 41:16
  Is it locks you in and you're totally brittle?
• 41:20
  And in an environment where you're doing machine learning
• 41:22
  and AI. You're constantly things are
• 41:25
  constantly being updated me. Just just recently other
• 41:28
  example, Ubuntu. Okay.
• 41:30
  So what's the most common thing in the world em up.
• 41:32
  So I've worked in environments. Well you we are you debt?
• 41:35
  We're doing. Gavin Ubuntu 16.
• 41:38
  But deployments are in Ubuntu 12.
• 41:42
  Because it's automated. And if you wouldn't change it,
• 41:44
  he can't change because he got to redo everything.
• 41:47
  We want to 12. Okay?
• 41:48
  Well, because they're not using Docker yet, right?
• 41:51
  Because they automated it, but they don't have Docker because
• 41:53
  no doctor would kind of solve this problem, hopefully, so, So
• 41:58
  you want me to do a Dev in move on through 16 and deploy to Bone
• 42:02
  through 12? Okay, and that that you want me
• 42:06
  to do that. Using, you know, what am I going
• 42:07
  to use Python? 1/3.
• 42:09
  I mean, what do you want to do here?
• 42:10
  You know, it doesn't make any so this automation obsession.
• 42:15
  He's really problematic. If you don't get the point of
• 42:21
  automated that you have a product and you want to scale
• 42:25
  the product. But if you're doing science, you
• 42:29
  don't have the product yet. You don't know what you're
• 42:32
  doing. So the last thing in the world,
• 42:34
  you want to do is automate something before.
• 42:36
  You know, what it is. That's a great.
• 42:37
  What if I have to move from tensorflow?
• 42:39
  2 pi torch? What if I have to go from?
• 42:42
  What if I could change this kind of stuff?
• 42:44
  It's it's the wrong mentality for trying to do Innovation.
• 42:50
  What you need to do is create a carve-out, you create a
• 42:54
  carve-out from the organization with a week, couple.
• 42:57
  And for the rest of the organization and usually what
• 43:00
  happens is, the automated systems are strongly coupled and
• 43:04
  there are these dense dependencies that you can Tundra
• 43:07
  unravel. So you automated things but you
• 43:10
  did it in a way where everything is codependent.
• 43:13
  You forgot about the whole idea of you, we need to have weakly
• 43:16
  dependent distributed systems. So yeah, because you mentioned
• 43:21
  yesterday that this is how companies can get into trouble,
• 43:25
  right? When you're stuck in you say,
• 43:26
  okay, we're going to build this golden cage, around a certain
• 43:30
  version of a certain program or language that you're using.
• 43:34
  And say it's basically putting all your chips on, you know,
• 43:38
  you're betting the farm on this. That's going to work.
• 43:40
  And then later as machine learning is moving so fast, you
• 43:45
  end up finding yourself, you get caught with your pants down, so,
• 43:49
  It makes the whole point of data science is you're trying to do
• 43:52
  new things. So, you know, you're not gonna
• 43:56
  you know, you're using Greenfield.
• 43:58
  Yeah, so God, I don't know what you asked.
• 44:00
  Well I was just I wanted to move on to something else that you
• 44:04
  mentioned yesterday with it goes back to this organizational
• 44:07
  issue and how you can find certain companies that are and
• 44:13
  what you've seen when it comes to bigger Enterprises and
• 44:17
  they're raising more. Capital and Galia Hester's the
• 44:22
  investor start to ask. Well, what's your AI plan?
• 44:25
  Yeah right. Now look you're going you know,
• 44:26
  a lot of companies are right now, we're trying to you know,
• 44:28
  you want to expand right? What's the point of automating?
• 44:31
  Your expanding? Why would you automate something
• 44:34
  that, you know, do you know you're not expanding right?
• 44:37
  See, you know, you're not and not, everyone is McDonald's, you
• 44:39
  know, sometimes you're just a home restaurant, you know, you
• 44:42
  don't need to automate, you know, the cooking, you know,
• 44:45
  what makes the restaurant unique is the fact that you can make
• 44:47
  new dishes all the time. So if you're trying to make a
• 44:51
  McDonald's expand, you want to raise Capital.
• 44:54
  So the drived automation is usually behind that is is a rate
• 44:58
  as a capital rate. So you go to raise capital and
• 45:02
  your investors said you always what you're a, I plan.
• 45:04
  Huh? You're a I plan.
• 45:06
  We're not going to give you money.
• 45:07
  They don't think you're not going to be a business in ten
• 45:09
  years. And, you know, the goal of the
• 45:11
  CEO is probably I want to retire.
• 45:14
  I'm going to get some Capital expand this thing out, you know,
• 45:17
  Jim, the stock price up that I want to get out.
• 45:18
  I'm retired. And I don't give a damn about my
• 45:21
  investors. And you know, what?
• 45:23
  A lot of money, let the pension fund Odette, you know, sell it
• 45:26
  to the Chinese. You know, what are we going to
• 45:27
  do? So, and I'm dead serious.
• 45:31
  About this is how people think especially used to do, you know,
• 45:35
  anyone east of the Rocky Mountains, here in California,
• 45:37
  you know, they're really just trying to retire.
• 45:40
  And so when they hire you, Are you really doing AI or are you
• 45:46
  just setting up a sunshine lab? You know, you just want things
• 45:50
  to look good for your investors so that they give you a little
• 45:53
  money. So you can just keep going and,
• 45:56
  you know, you don't want to do a.
• 45:57
  I would you want to do is you want to meet your earnings and
• 45:59
  you want to meet guidance? That's all you care about is you
• 46:01
  meet are using. Because if you miss guidance and
• 46:04
  you miss earnings, two times in a row stock, price is going to
• 46:06
  tank and your your're. Now it's going to be gone.
• 46:08
  That's all they care about. So the idea that you're actually
• 46:10
  doing aii, A or you're actually doing data science, you know,
• 46:15
  you know if they're to do that, you're there to look like you're
• 46:18
  doing it, you know, you're there to just kind of show up and you
• 46:22
  know, do some stuff and you know, you puke see Peaks up
• 46:26
  around it kind of looks you know, you're not really doing it
• 46:29
  and I think that that I think people really discount that like
• 46:34
  what's really happening here, you know, what are the actual
• 46:39
  motivations? And it's kind of like to make
• 46:41
  the joke. It's Like someone who goes to a
• 46:43
  gym and the trainer says, you got to go on a diet.
• 46:47
  And they just they're not going to go on a diet, you know,
• 46:50
  they're not going to do it and then they just keep doing stuff
• 46:53
  and they get you're not losing any weight because I do like
• 46:55
  Jim. And I did like gymnastics and
• 46:56
  Crossfit. Wait, I have not just got to be
• 46:58
  Honestly, though. I like to wait later.
• 46:59
  I'm not going on a diet every time I'm, yeah, I'm just like,
• 47:02
  every time I go to Donna, get hurt, you know, I end up getting
• 47:04
  pulled a muscle. So just not going to do it.
• 47:08
  I like my work of the weightlifting coach.
• 47:09
  He says, you know fat is where it's at, you know, the Olympic
• 47:12
  weightlifters. All these guys are giants, you
• 47:14
  know, but he's also an older guys out.
• 47:17
  You're too old for this. Don't don't try to pull up.
• 47:20
  People go in and I can go to the gym, but they're not going to go
• 47:23
  to bed. They'd never make progress.
• 47:25
  And, you know, are they really being honest with themselves?
• 47:27
  Look, you know, you're not going to go on a diet.
• 47:28
  So just just stop whining about you.
• 47:30
  You know, I can pop talking about, just do what you're going
• 47:32
  to do. A lot of these companies are
• 47:34
  like this, you know, your you say you're going to do it but
• 47:37
  you don't really want to do it. You don't have any grit and so
• 47:41
  you get the motivation. Maybe they just don't care.
• 47:44
  Like, yeah, well really core if they care about it.
• 47:46
  Not really. You and, you know, you have
• 47:49
  these discussions. I've had these Frank discussions
• 47:51
  with people and it's just like you can't be too blunt with
• 47:54
  people. You're working with either
• 47:55
  paying, you can't really get in your face and go, you know.
• 48:00
  Why are you doing this? Well, we're doing just fine.
• 48:03
  We're really way ahead of everybody else.
• 48:05
  Nobody's, we don't have any competitor isn't like.
• 48:08
  All right, fine. You know, when when Google can
• 48:11
  we used to when Google comes in and decides they want to do your
• 48:14
  business or Amazon when Amazon comes in and decides that they
• 48:18
  want to do what you're doing. Yeah, you'll be going ten years.
• 48:21
  We have worked for a Shriner consultant on the project.
• 48:24
  I was with this guy wrote a short, long fun.
• 48:28
  And the whole, the whole strategy was that you short
• 48:30
  companies that we think Amazon are going to enter into your
• 48:34
  Market because Amazon is going to put you out of business
• 48:38
  because Amazon has been doing AI for 25 years.
• 48:41
  So we just you were just waiting and we short, you know, short
• 48:45
  sellers. Yeah, they are.
• 48:46
  Yeah. The like like, you know, it
• 48:49
  doesn't we didn't save you money and, you know, it's just like we
• 48:52
  used to joke. If somebody buys IBM Watson, you
• 48:55
  should short the Is that real product?
• 48:59
  It's not going to get. It doesn't work.
• 49:01
  So so that's the point. Like, if you're really do, if
• 49:06
  people really want to do something, they won't hire
• 49:10
  someone. Like I worked in client, if they
• 49:12
  set up a whole AI team and the guy they put in charge has no
• 49:17
  experience in AI whatsoever, knows nothing.
• 49:21
  And you have this conversation. Why would you put someone in
• 49:24
  charge? Who doesn't has no experience?
• 49:26
  Well, people who have no experience in charge of things
• 49:28
  all the time. Look at where the world is now.
• 49:31
  Thanks to these guys, you know, great job, you know and who is
• 49:35
  running the Wuhan lab, you know, they experience what's going on.
• 49:39
  You can't put people don't know what's going on in charge of
• 49:41
  things and expect to get results and then whine and complain and
• 49:46
  II things aren't working. Yeah.
• 49:48
  Yeah, maybe maybe because it takes, you know, sick.
• 49:51
  Months to explain something. That should be a conversation.
• 49:54
  You know, this is a real problem with a lot of what we see.
• 49:57
  And I think the clients that I've seen that have been
• 50:00
  successful. Are the ones where the senior
• 50:05
  executives are the really the founders, it's really the
• 50:07
  founders come in and say we want to do this and when the project
• 50:13
  managers pushback or the guys, who do you do the provisioning
• 50:20
  push back? The founder comes and says no
• 50:24
  no, no, we're doing this. Stop pushing back and eventually
• 50:29
  they get to a point and that's where things work and when
• 50:34
  people just are not and we see that in Silicon Valley, you
• 50:37
  know, outside of the valley too, but you know, if they're in
• 50:39
  there and they're just like well, we're just going to keep
• 50:41
  doing what we're doing and these guys push back at you.
• 50:44
  You just have to figure out how to convince them.
• 50:46
  Like, all right. Well, you know, that's, you
• 50:49
  know, I You know, there are some peo it's very hard.
• 50:54
  When you go into an organization, you know, if
• 50:56
  you're not really committed, you have to somehow throw the
• 50:58
  minions in only, all these old dogs.
• 51:00
  You got from a bone. Right, they just be old dogs
• 51:04
  bark at you and you got to throw them a bone to do something and
• 51:07
  you know, if they're not committed to, it is very, very
• 51:10
  hard. And I think a lot of the op
• 51:12
  stuff you see is not designed to enable you.
• 51:18
  It's designed to block you. It's they're basically covering
• 51:24
  themselves. They don't want to take any risk
• 51:26
  of any kind are not there to innovate.
• 51:28
  They're not there to take risks. They want to make sure that what
• 51:31
  They are doing, it's as they want to have zero risk and when
• 51:35
  you take zero risk, give zero, you have zero reward.
• 51:40
  So that's why can't he talk to me about that?
• 51:43
  Saying that you guys got into Valley about the three kinds, of
• 51:46
  course. Yeah.
• 51:48
  This is not necessarily a mile off.
• 51:49
  But yeah, we have a saying when you come to Silicon Valley, you
• 51:51
  know, I, there are three kinds of employees in Silicon Valley.
• 51:55
  It's not a bite there. Plus one people.
• 51:58
  These are people who do work. They do actual work.
• 52:01
  They move things forward. There are the zeros, they do no
• 52:05
  work, but they also don't cause any problems.
• 52:08
  They basically go out and get you a sandwich.
• 52:11
  And then there's a minus ones. These are the guys who don't do
• 52:14
  any work of the road and they screw your work up.
• 52:20
  And amazingly, they actually form little cliques.
• 52:24
  So, there are like little groups of like managers with their
• 52:27
  people and they'll come to a company, then they hire all
• 52:29
  their guys. And they just sort of come in,
• 52:31
  sort of parasitically and sort of, you know, her suck up the
• 52:35
  resources, you know, and they're good at doing that.
• 52:38
  You see a lot of that, of course, are also people who are
• 52:40
  very, very good and they come in and drive things forward, but
• 52:43
  this is a lot of, you know, when you're doing Ela, mobs.
• 52:46
  Uml Ops ask yourself. Are you really enabling people
• 52:50
  to do? They need to do, or are you just
• 52:53
  automating for the sake of Automation?
• 52:55
  And what parts? There was a great paper that
• 52:59
  came out a few days, couple days ago about what, you know,
• 53:02
  someone had done a study of what are the problems that arise?
• 53:05
  When training act when building AI software and they went
• 53:09
  through all GitHub and they went through stock and flow.
• 53:11
  And then look at me, look at thousands and thousands of
• 53:13
  questions and then they interviewed people and ask them.
• 53:17
  What are the problems that come up when you're trying to build
• 53:19
  soft? Because because the point of
• 53:20
  automation is to solve problems. It's the get it through.
• 53:24
  And one of the kind of problems that come up there.
• 53:26
  People don't have Deploying microservices, that's not
• 53:29
  enough. There are, you know, obviously
• 53:31
  in any large Enterprises are orchestration problems, right?
• 53:35
  And I think a lot of the things we see in ml Ops are really
• 53:38
  orchestration. Not so much deployment,
• 53:41
  completely obvious, but most of the problems are things like The
• 53:47
  date is wrong. The labels are wrong.
• 53:51
  The hyperparameters need tuning. I set the, I set the learning
• 53:56
  rate wrong on the soul, over these, the kind of things that
• 54:00
  block it really are blocking you.
• 54:02
  I need to run complex experiments and I can't.
• 54:05
  The things that block you are not, you know, I need to I know
• 54:09
  I need to read the ploys a model, you know, that you know,
• 54:12
  that these problems have been solved in the web engineering
• 54:15
  world, but you know, there are these kinds of things.
• 54:19
  Is that are, you know, there are their performance issues?
• 54:22
  I can't the things too slow. I can try need more GPU time.
• 54:27
  These are things that are blind. I think automation needs to
• 54:29
  focus on what are the things that are blocking you and how
• 54:33
  can you under the things that you do over and over and over
• 54:37
  and how can you automate them? And that's that would be very
• 54:41
  helpful. And I think thinking more.
• 54:45
  And so that's, you know, there are systems that do this, you
• 54:49
  know, there are data science platforms, but it's still not
• 54:51
  enough people really using them that much maybe internally, they
• 54:55
  build something. A lot of companies will build
• 54:56
  this internally and force people to do it, but that's, you know,
• 55:00
  God just just before we, it's getting close to the hour and
• 55:04
  we've got a question in the chat from Gonzalo.
• 55:06
  And he's asking, okay, how can we educate companies about the
• 55:10
  ml lifecycle? And that automate things at
• 55:15
  start is not only the way it should be done.
• 55:18
  So basically, how can you teach a company that it doesn't need
• 55:22
  to be automated from the beginning?
• 55:24
  They have to want to learn. There you go.
• 55:28
  Yeah, that's you, can't teach a student who doesn't want to
• 55:30
  learn, you know, you have to look when you're dealing with
• 55:35
  people, you have to develop personal relationships and you
• 55:38
  have to figure out it, you know, I think that the people are
• 55:43
  seeing this. I mean, certainly organizations
• 55:45
  that really want to do this, figure it out.
• 55:47
  But a lot of them just don't want to change.
• 55:50
  So, you know, they are, you know, you got to fit, you know,
• 55:54
  the clients that I want to change her.
• 55:55
  Sometimes the ones I don't think it's more like I don't like it
• 55:59
  so much. Educating clients as it is,
• 56:01
  finding the ones who want to be educated.
• 56:04
  You've got to find people who have such luck.
• 56:06
  I've had clients like I work at eat years ago.
• 56:09
  I worked at eBay and eBay. I mean, they've everything is
• 56:13
  very process-oriented, mean takes two years to get something
• 56:16
  at the point of eBay. EBay has like a small European
• 56:18
  country. It's bigger.
• 56:20
  It's bigger than some of the state's.
• 56:21
  Yeah. It's big, you know, their
• 56:22
  economy, but they had done the project multiple.
• 56:27
  Times and it just kept failing and they finally came in.
• 56:30
  Brought me instead of kid. Look.
• 56:31
  We're going to hire you. We want to find out why this
• 56:33
  thing is they are. And I think probably the biggest
• 56:36
  thing that companies are concerned about is that they do
• 56:41
  a project and it fails. And then they can't do it again
• 56:45
  because they won't get the budget.
• 56:49
  And if you're trying to educate people on anything, you've got
• 56:51
  to focus on, you know, the cash, where's the money coming from?
• 56:55
  You need to convince them? Look, if you do it this way.
• 56:57
  It's going to fall down. And you know, you've got to be
• 57:01
  more Nimble. Otherwise, you know, you're just
• 57:03
  not going to the project in the die and you're not going to be a
• 57:07
  winner. You need to convince people that
• 57:09
  you're there to be that you're going to make them look like a
• 57:12
  hero. That's where it starts.
• 57:15
  You're going to make them look like a hero, the Project's not
• 57:17
  going to fail. It's going to be successful.
• 57:19
  They'll be able to grow, they'll be able to expand their group.
• 57:23
  And you got to Target, those kinds of people.
• 57:26
  Exactly. So, wow, so many bombs of
• 57:32
  knowledge. You would just dropped on us, so
• 57:35
  maybe a different opinion because I come at it from You
• 57:40
  know, I come at it, from a perspective of I'm trying to
• 57:43
  generate him, try to generate Revenue quickly and I coming out
• 57:46
  from the from really, from the Wall Street perspective, you
• 57:48
  know, having worked as a Quant, and how we did things there,
• 57:51
  which I think is very different than the traditional software
• 57:55
  engineering perspective. Well, yeah, I love this
• 57:57
  perspective of looking at it and saying there's a lot more that's
• 58:01
  happening with machine learning and we've all seen this, you
• 58:05
  know, the statistic that only 15% of your models, get put into
• 58:09
  production. And, and it's a lot more than
• 58:11
  just the tech. That is not allowing you to put
• 58:15
  it in production or a data scientist who doesn't know how
• 58:18
  to run kubernetes. It's a lot more.
• 58:20
  There's a lot deeper levels to that statistic and this is a
• 58:25
  great view, your shining a light on something that maybe most
• 58:28
  people don't think about when they're seeing that statistic.
• 58:33
  Well, I think that they think the models just don't work and
• 58:37
  that that's true too, but There's a lot.
• 58:40
  I mean, we worked at it, you know, we had this company that
• 58:41
  had the largest antique store in the world and at the end of
• 58:45
  working with them for several months.
• 58:46
  They were able to give us one column of data one column and
• 58:51
  then they asked then they complained all the models not
• 58:54
  very good and even my guy who's usually very these friend.
• 58:58
  She's very polite, you know, he's very he's like even he was
• 59:02
  like because I'm very animated. He's not at all.
• 59:03
  He goes he gave us one called the data.
• 59:06
  What would you expect it to be even?
• 59:07
  He was like, this is ridiculous. You know, I mean, and you're in
• 59:11
  your car and you're saying that, you know, we built a model that
• 59:13
  doesn't work. I'm like, one column you have
• 59:17
  millions of columns in this is all you could give us, they
• 59:20
  couldn't get the data out. You know, these are things that
• 59:24
  are, then they, then they coin know they're there.
• 59:29
  The problem is that things are so automated there.
• 59:31
  So brittle that. You can't do anything new and
• 59:37
  this stuff is fundamentally new. And what I always recommend is
• 59:41
  that you need to have a carve-out, you need to carve out
• 59:45
  some peace in the organization where you can get data and you
• 59:49
  can sort of and then deploy your app, and you're not involved in
• 59:54
  all of the Machinery of your organization.
• 59:57
  You don't want to be deep inside having to deal with that.
• 59:59
  Because as soon as you get involved, soon as people tell
• 1:00:01
  you, everything has to be automated as to run through our
• 1:00:04
  system. It's just not going to happen,
• 1:00:08
  and they have to be Able to deploy quickly and they and and
• 1:00:13
  so many problems have come out of people saying everything has
• 1:00:15
  to be automated like it just that's just not, you know, you
• 1:00:21
  didn't build it and you're trying to tell me you you it's
• 1:00:24
  just not it's a not of thoughtful.
• 1:00:31
  Decision. Yeah, it's a, it's a knee-jerk
• 1:00:34
  decision. It's a, it's a decision where
• 1:00:39
  you're trying to protect yourself or you're trying to
• 1:00:41
  force people to do things the way you want it done.
• 1:00:44
  Instead of really looking at it and ask, what really needs to be
• 1:00:47
  done. I love this perspective.
• 1:00:51
  I love talking to you. Charles.
• 1:00:53
  I really appreciate it. Everybody.
• 1:00:54
  That's stayed with us. I thank you all for joining us.
• 1:00:59
  I thank you. Charles.
• 1:01:00
  This has been incredible. I really think go ahead.
• 1:01:06
  I just really think you opened our eyes to some things.
• 1:01:09
  It's been really just, yeah, for me.
• 1:01:13
  I've loved it. And before we go, I'll let you
• 1:01:16
  say a few things, but I want to mention it.
• 1:01:18
  Everyone, we have another meet up tomorrow at 10 a.m.
• 1:01:23
  UK time. So if you're in the Europe
• 1:01:26
  region or the UK, you can join us.
• 1:01:29
  We're going to put it on YouTube.
• 1:01:30
  So if you're in the u.s. you can tune in later that YouTube
• 1:01:35
  channel. I'll send a follow-up email with
• 1:01:37
  all of the recordings of this, in case you want to go back and
• 1:01:40
  look at it and and show somebody else.
• 1:01:43
  Maybe that's in your company about what Charles has seen.
• 1:01:48
  He sees ml ups and hopefully you can use it as Firepower to
• 1:01:53
  convince some people to not be so rigid in their thinking.
• 1:01:57
  I hope I don't see anybody fired.
• 1:02:01
  You know it. Thanks a lot.
• 1:02:03
  I really appreciate having a chance.
• 1:02:04
  I'm going to cut a joint in some more and hear what you guys have
• 1:02:08
  to say. I think it's a great fall.
• 1:02:10
  Yeah, maybe we can convince you to join our slack Channel and if
• 1:02:14
  you want to follow up the conversation with Charles,
• 1:02:16
  what's the best way to reach you?
• 1:02:19
  You know, you reach me on LinkedIn.
• 1:02:22
  I'm on LinkedIn or you can, you can reach me on, you.
• 1:02:25
  Send me an email, Charles at calculation Consulting.
• 1:02:28
  They'll work also perfect. Perfect so far, right?
• 1:02:31
  If you want to keep the conversation going, there you
• 1:02:33
  have it and until tomorrow, or next week.
• 1:02:36
  Thanks to you all for joining in.
• 1:02:38
  And thanks, Charles. This is great.
• 1:02:40
  I appreciate thanks for having me, and thanks for listening.
• 1:02:42
  I appreciate it. See you guys.
• 1:02:44
  Be safe.