Latest / Agentic Conversations (formally mlops.community) / [Bonus Episode] Practical AI x MLOps // Demetrios Brinkmann, Mihail Eric, Daniel Whitenack and Chris Benson

Transcript

[Bonus Episode] Practical AI x MLOps // Demetrios Brinkmann, Mihail Eric, Daniel Whitenack and Chris Benson · Agentic Conversations (formally mlops.community)

• 0:00
  What's going on everybody? This is another edition of the
• 0:03
  MLA Ops Community podcast and it's a very special edition.
• 0:06
  I am your host Demetrios and today I'm joined by none, other
• 0:09
  than mr. McHale Eric, what's up, dude?
• 0:13
  Hello. Hello, it's been a while.
• 0:15
  I'm glad to see you again it's going to be here so very very
• 0:19
  special as I just mentioned because we not only had each
• 0:24
  other's company today. So that's that's enough most of
• 0:28
  the time actually. Yes, it is for me, I don't know
• 0:31
  about yes. But for me it is and today in
• 0:35
  this episode, it was a collaboration with the Practical
• 0:39
  AI eyes. And so we had Chris and Daniel
• 0:42
  on it was a little bit of us asking them questions.
• 0:45
  Then I'm asking us questions and then jam session.
• 0:48
  I don't know what you thought about it.
• 0:49
  I enjoyed it was great. It's great.
• 0:51
  Yeah. So super, super sharp, guys.
• 0:54
  Really like what they're doing with the product cast, really?
• 0:56
  We touched on so many different topics from just ml Ops at a
• 0:59
  high level. All right.
• 1:00
  What was the goal of ml Ops and Liberate?
• 1:02
  The history of a laps through to?
• 1:04
  How does that play into modern-day generative Ai?
• 1:06
  And what do we, what's going to stay the same?
• 1:10
  Or what's going to change with ML UPS?
• 1:11
  Now that we're entering this new generative, AI hype cycle and
• 1:14
  just touch on a lot of fascinating topics.
• 1:16
  Hopefully will give the listeners a lot of food for
• 1:18
  thought, dude. One thing that I just can't stop
• 1:20
  thinking about is this new generation of ml Ops tools for
• 1:27
  the foundational models for the large language models and how
• 1:30
  How there's going to need to be infrastructure.
• 1:32
  And these guys they brought it up and they made so many
• 1:36
  articulated points that I could not have done better myself.
• 1:40
  And what they were talking about is how there's going to be
• 1:44
  infrastructure for foundational models and large language
• 1:47
  models. And we're always going to have
• 1:49
  this need for infrastructure. Even if we think that we can
• 1:52
  prompt engineer it away, it's still there.
• 1:56
  It's still going to be there. And what is that going to look
• 1:58
  like? What kind of knew it?
• 2:00
  For structure. Paradigms are we going to have
• 2:02
  that aren't going to be the same as what we're used to and what
• 2:07
  the majority of the questions that are being asked right now?
• 2:10
  And then mobs Community are so that was one of my favorite
• 2:13
  pieces. I really appreciate their views
• 2:15
  and they've been in the trenches doing machine learning.
• 2:19
  It's quite obvious, they have Battle Scars and so I love
• 2:23
  talking to them and seeing their way of looking at machine
• 2:26
  learning in ml, Ops in general. Yeah, in case.
• 2:30
  Case, you have not heard about their podcast practical.
• 2:32
  A, I encourage you to go over and subscribe to it will leave a
• 2:36
  link to it in the show notes or just type it into your favorite
• 2:42
  podcast player. And, you know, it would mean the
• 2:45
  world to us is if you rated and left a comment, gave us a
• 2:51
  review, even if it's a bad review, we don't care.
• 2:54
  I mean me, Kyle actually really enjoy those bad reviews.
• 2:57
  He wants to know how we can get better.
• 2:59
  I enjoy it. That negative feedback and get
• 3:02
  hit. It's what I look at in the
• 3:04
  morning when I'm getting pumped and getting ready to start the
• 3:06
  day like I will make this better.
• 3:09
  Exactly. Exactly.
• 3:11
  I love that Spirit. That's what we want to hear.
• 3:13
  And you know, I do the same. That is how hopefully we
• 3:19
  encourage you to leave us a review without further Ado.
• 3:24
  Damn, I am just on a rhyming spree right here.
• 3:28
  Did you hear all three of those? I didn't mean to do that.
• 3:30
  That anyway, let's get into this podcast, man.
• 3:40
  Well, welcome to a very special episode.
• 3:44
  I'll say this is an episode of a podcast actually of to podcasts
• 3:49
  Chris. Uh-huh, is a different kind of
• 3:52
  episode than we normally do, because we're privileged to be
• 3:55
  joined By Demetrios and Mikhail from the ml Ops Community
• 4:01
  podcast. Welcome guys.
• 4:03
  Yeah, we're hiding over here in the shadows, in the ml, Ops
• 4:07
  Community, the Shadows that the ml Ops community.
• 4:11
  We don't really have a guest this week.
• 4:13
  We have 4 host. That's what we have.
• 4:17
  This will be interesting. A lot of chefs in the kitchen
• 4:19
  right now. Yeah, that's right.
• 4:21
  Which is obviously what listeners want more of.
• 4:27
  We're just gonna answer each question with more questions.
• 4:30
  I don't know if we have the host with the most, but we have the
• 4:32
  most hose. This is everything over.
• 4:34
  There we go. There we go.
• 4:37
  The real question is, who's going to ask the first question?
• 4:39
  Oh, I was gone. On for it but no.
• 4:41
  Now you can but technically that was a question.
• 4:45
  I'm not sure. So maybe I'll answer your first
• 4:48
  question with my first question, mli, Ops and Community.
• 4:52
  I love everything you all are doing with the community over
• 4:56
  there. Of course the podcast which you
• 4:58
  know will link to and the meetups and the content that
• 5:02
  you're putting out there. My sort of basic question though
• 5:06
  is like we have heard ml Ops however, many times on this
• 5:09
  podcast Ask practical Ai and we've heard even conflicting
• 5:15
  sort of statements about what that even is.
• 5:18
  So like you're branded the Emma Ops community.
• 5:21
  So from your perspective like what does that mean who's in the
• 5:25
  community? How is it defined all that
• 5:27
  stuff? So so that I really high-level
• 5:30
  ml Ops, it's an abbreviation of machine learning and operations
• 5:34
  and one really quick easier, way to think about what Emma UPS
• 5:37
  does. It's, it's fundamentally about
• 5:39
  the question of How do we take machine learning that you
• 5:42
  typically see in like research and deliver that in the real
• 5:45
  world? So until relatively recently, I
• 5:48
  want to say the last few years machine learning was starting to
• 5:51
  get really hot maybe like a really kicked off in the 2012
• 5:54
  2013, right? As when we had this Resurgence
• 5:56
  of deep learning that really really picked up and people got
• 5:59
  excited about image recognition systems speech, you know, ASR
• 6:03
  systems and for a long period of time.
• 6:06
  A lot of machine learning, even Cutting Edge of machine learning
• 6:08
  still happened in the lab right in the research.
• 6:10
  Other big institutions or even just at actual academic
• 6:13
  institutions. And so it was around, you know,
• 6:16
  I want to say where I went and of tensorflow came out, when we
• 6:18
  started really getting this effort to standardize the
• 6:20
  processes of going from something that you see in like
• 6:23
  research new models, new AI systems, and try to actually put
• 6:27
  out in front of people. Right?
• 6:28
  How do we get these systems to the point where they can start,
• 6:30
  delivering real value to real people?
• 6:32
  And ml Ops is pretty much a set of Frameworks and tools and
• 6:36
  systems that people have developed their based a lot in
• 6:39
  software, engineering as well. Well, as devops to make that
• 6:42
  translation process from Research into production much
• 6:46
  easier. And so it's really just a set of
• 6:48
  principles and techniques and tools that a lot of the
• 6:51
  communities developed to make that translation possible.
• 6:55
  Yeah, I think around the time like we started this Chris, I
• 6:59
  think one of our feelings was an hour, you know, a good way is
• 7:04
  in, but I still feel this way that like it's not an easy leap
• 7:08
  to sort of just Learning like oh I'm going to do the fast a i
• 7:12
  course or something. And then like okay well let's
• 7:15
  say you get a position as a data scientist or like whatever the
• 7:20
  title is like okay. Whatever is happening.
• 7:24
  Like in that position like there are overlaps between what you
• 7:28
  learned in like a fast AI or whatever like certainly that's
• 7:31
  like wonderful stuff but like it is only like a very small
• 7:35
  component of a larger like system of like practicalities.
• 7:40
  That are not that easy to access and learn about totally and
• 7:44
  they're still trying to standardize.
• 7:46
  Like the other piece is depending on your use case ml
• 7:51
  ABS might be one thing or another thing to you it's really
• 7:54
  not super clear right now. Like, if you're using deep
• 7:57
  learning, that's one type of ml Ops that you're doing.
• 8:00
  And if you're doing recommender system and you like to play in
• 8:04
  the trees and the decision trees, those random Forest, then
• 8:07
  it's a different type of ml upset you So, like, I think we
• 8:12
  should dive as we go here and we should dive into the Weeds on
• 8:15
  what those are. But I actually want to throw a
• 8:18
  hand grenade into the conversation and say, like, oh,
• 8:21
  Dove devops or Dev seconds depending on where you're coming
• 8:25
  from, you know, in like the differences and ml Ops.
• 8:28
  How does it integrate? Is it some of the same?
• 8:31
  This is the show where we can clarify.
• 8:33
  All of these things between us here.
• 8:35
  We should just Dive In by devops.
• 8:38
  Do you mean a Docker? Chris Yeah.
• 8:42
  What do you what do you mean by devops something for me heiner?
• 8:46
  Well, once again I mean I don't want to put my my definition.
• 8:50
  These are the things that have opposite.
• 8:52
  The end that everyone is talking about if you were out there in
• 8:56
  the world and you're doing models of various types of might
• 8:59
  be, do you learning might not be deep learning, you know, Dimitri
• 9:02
  those points but, you know, it also integrates in with
• 9:05
  software. And so where does One Stop and
• 9:08
  the other star? You know, if we can start
• 9:10
  Sorting through what we think these things are.
• 9:12
  I think that would be useful because you put a bunch of data
• 9:17
  scientists and software people in a room.
• 9:18
  And they still today, don't necessarily know what each other
• 9:21
  means when they're having these conversations, I aired a bit of
• 9:24
  my dirty laundry and my pet peeve of like, people saying a
• 9:28
  Docker or Dockers, I deployed a Docker, but yeah, I think that's
• 9:33
  like the minds that you hit when you put a software engineer in a
• 9:37
  data scientist together. I think it touches on a really
• 9:40
  Astute point, which is that the definitions are a little fuzzy
• 9:43
  that they are to fit me. Perfectly honest, there.
• 9:45
  They're very fuzzy in terms of, where, where does the domain of
• 9:48
  one begin in the domain of another?
• 9:50
  Like, how do we cross these territories, where the fiefdoms,
• 9:52
  where the the boundaries set up and fundamentally ml Ops is has
• 9:57
  been. An is a very interdisciplinary
• 9:59
  field and so maybe one of the easiest visuals that we like to
• 10:03
  sort of discuss or we try and explain what I'm office is like
• 10:05
  imagine like a three circles been diagram, where one is data.
• 10:10
  The other is software engineering and the third is
• 10:13
  devops and ml UPS, its kind of squarely, right in that little
• 10:17
  intersection area of those three circles because there's a lot of
• 10:20
  the stuff that makes ml Ops. And the way people use, it does
• 10:23
  touch on pretty much every single one of those three
• 10:25
  circles. There's modeling, right?
• 10:27
  There's fundamental, your building models your training
• 10:29
  systems. You have these data concerns
• 10:31
  that really falls out of the data science category you have
• 10:33
  software engineering, right? Because you're trying to
• 10:35
  actually build out good systems, that have to be engineered, you
• 10:38
  have to write code that has to be well.
• 10:40
  Stood, unit tested code. And then the devops is real, but
• 10:43
  how do you streamline that process of going?
• 10:46
  You know, doing this once putting it in a Docker and then
• 10:48
  doing it multiple times and then just repeating that over and
• 10:50
  over and over and doing it systematically so that hey, I
• 10:54
  have 100% ideally up time on the machine Learning Systems, I put
• 10:57
  out into production and that's really the devops component.
• 11:00
  So really, it borrows a little bit from every single one of
• 11:02
  those circles. See what you did there and just
• 11:06
  shout out like to Docker compose, because you can do so
• 11:10
  much. With that.
• 11:11
  Let's just not forget the unsung hero of this whole thing and
• 11:16
  give a huge shout-out to Docker compose or the bash script.
• 11:21
  The truth but ruo jie hero. Exactly the unsung hero really
• 11:28
  in reality. But there's, you know, like me
• 11:31
  Kyle was saying to, I think that it's not only one person that
• 11:37
  should be tasked with this ml Ops.
• 11:40
  A title. And a lot of times you can get
• 11:44
  yourself into a lot of trouble. If you're trying to build out
• 11:48
  something that is going to productionize your machine
• 11:52
  learning and you expect your data scientists, to be able to
• 11:55
  do everything, or you have a very like naive sense of what it
• 12:01
  takes to actually productionize machine learning and and not
• 12:05
  just like productionize it in a way where you get a model out
• 12:09
  but really make Process out of it.
• 12:11
  And I really enjoyed an idea which is ml obvious is not just
• 12:17
  putting a model out there. Once it's the act of n plus 1.
• 12:24
  So ml Ops is a machine learning model in production but not just
• 12:30
  one n plus one and that's when you get into how are the
• 12:34
  processes around this and what do we really need to be thinking
• 12:38
  of when we're production izing? Our machine Learning.
• 12:41
  I have a question for you, just straight off of that.
• 12:44
  Can you talk about like when you're talking about getting
• 12:47
  that out in that in plus one model?
• 12:49
  How is ml Ops in that capacity like kind of late in the process
• 12:54
  there? As you're kind of getting it out
• 12:56
  into wherever your environment, the cloud or whatever it is that
• 12:58
  you're putting the model. Where does that differ from the
• 13:02
  devops world? Because it's easier to see that
• 13:04
  the front end because you're kind of starting with two
• 13:07
  different things that you're trying to put out there in the
• 13:10
  world, you know? One case you're starting by
• 13:12
  training, a model of some type, and other case you're working on
• 13:15
  software and all that and you're, you know, you say, okay,
• 13:18
  we're ready to go, but somewhere down those pipelines, they start
• 13:21
  looking very similar in a lot of ways.
• 13:24
  How do you look at that? How do you differentiate what
• 13:26
  those are? Do they merge.
• 13:28
  Do they not merge, how do you say it?
• 13:30
  So it's the fun one that we talked about a lot in the ml Ops
• 13:34
  Community. Right.
• 13:34
  Like isn't analogs. Just devops and the dirty secret
• 13:39
  I think Is that? Yes, it is just devops but then
• 13:43
  you sprinkle a little bit of data on it and so magically it's
• 13:47
  not Dev Upson. I think me cayó can really like
• 13:50
  add a lot of great insight to this but we were just talking
• 13:54
  about this probably like a few hours ago when we met and it was
• 13:57
  very much like ml Ops is more about software engineering than
• 14:03
  it is about the modeling piece, I think like it's much Or for
• 14:10
  someone who knows about software engineering to get into ml Ops
• 14:15
  then it is for like a machine learning researcher to get into
• 14:19
  ml Ops and that's what I think we've seen over the years and we
• 14:23
  really have recognized in the past couple years that there's
• 14:28
  been a very clear engineering discipline that comes with ML
• 14:34
  Ops as opposed to in the beginning it was it wasn't so
• 14:37
  clear. It was like oh data scientist
• 14:39
  should be doing this. And now it's a little bit like,
• 14:42
  hmm, maybe it's not the easiest thing in the world to teach data
• 14:45
  scientists how to code. You know, it's funny that you
• 14:48
  say that and I'll leave the company a name, but I few years
• 14:51
  ago, I worked at a different place and I had a team for a
• 14:56
  first AI team. And I had a team of data
• 14:58
  scientists and I was like the only one to come from kind of a
• 15:02
  software development background and everybody else was kind of
• 15:05
  straight data science and Atlas. What we found, it was a
• 15:09
  struggle. A little bit to try to figure
• 15:11
  out how to create that ml Ops align.
• 15:15
  And I was like there's an entire engineering skill set that I
• 15:19
  think people come into it without realizing it up front.
• 15:23
  I think there's a it's like an Insight that they have, you
• 15:25
  know, that you just stated that takes them a little while to
• 15:28
  realize that an interesting is when we look at how Emma laps
• 15:33
  that evolution of Emma, laughs over the last few years because
• 15:35
  it seems to have been born a little bit in the wrong way, in
• 15:38
  that data, scientists started learning.
• 15:40
  These principles and these soft engineering practices, right?
• 15:43
  So we had these data sizes, like, oh yeah, like unit testing
• 15:45
  for data, that's super novel. And then the moment you had any
• 15:48
  personally came from devops or traditional software
• 15:50
  engineering, start entering these conversations.
• 15:53
  Are like, what you guys talking about?
• 15:54
  Like this has been a thing for like decades.
• 15:56
  Like what do you mean testing your model code?
• 15:57
  What you mean testing like different parts of your software
• 15:59
  engineering stack. Like what do you mean using a
• 16:01
  Docker? You know like what do you mean
• 16:03
  by like it making it seem like this is some profound Insight
• 16:06
  you guys have really found and yeah in a way there.
• 16:10
  Don't say there was tension but there was definite like a WTF
• 16:13
  when you guys talking about like this is not this is not as
• 16:16
  especially the guys make it seem.
• 16:17
  Yeah, you have the different sides of the spectrum, right?
• 16:20
  We're coming at it from this engineering point and they're
• 16:24
  like this is just devops and then you have the data scientist
• 16:28
  coming at it from the opposite end of the spectrum where
• 16:31
  they're saying like wow, this is so much different than anything
• 16:36
  that's ever been out there before.
• 16:38
  And so we need to build a whole Our new discipline around it.
• 16:42
  Yeah. Also I think that the part of
• 16:45
  the confusion is like just like everything else like labeling
• 16:50
  and naming like some of I think there's been data scientists and
• 16:53
  researchers over time. They're like, yeah, this sounds
• 16:55
  great like machine learning plus Ops and what that ends up being
• 17:01
  is experiment tracking, right? Which actually is something that
• 17:05
  also is not new because like people have been doing this in
• 17:09
  high performance computing. For like decades and decades,
• 17:12
  right? But this sort of solution around
• 17:15
  like, let's say, weights and biases.
• 17:18
  Clear ml. Like I love these tools we use
• 17:21
  Clear Mel like the value. I get out of that is
• 17:26
  experimenting at scale tracking experiments making sure I know
• 17:30
  like what assets are, what like maybe logging data sets and like
• 17:35
  input/output, all of that stuff but that's all like experiment
• 17:38
  tracking and One of the model. I sort of tend to think about ml
• 17:44
  Ops as like everything after that.
• 17:46
  Right. Like you have a model that's all
• 17:48
  great. Like whatever happened before
• 17:50
  even if you want like lineage about what data was input to
• 17:54
  your training which output what model like that's all track.
• 17:57
  That's great. There's still some huge hurdles
• 18:00
  to get over in terms of like this model being called from
• 18:05
  within a software application that has like real users on the
• 18:10
  Other and and like all of the potential implications around
• 18:15
  that. So in my mind, that sort of
• 18:18
  Distinction is really confusing for people and I don't know,
• 18:21
  maybe you can comment on this because it's maybe something
• 18:24
  that's come up on your podcast. I'm not sure.
• 18:27
  Have you also seen that confusion of like Ops versus
• 18:32
  experiment tracking and the fact that a lot of data scientists
• 18:37
  might might think there it's like I think that word that
• 18:41
  you're saying doesn't mean what you think.
• 18:43
  It means really good. Herb.
• 18:45
  Pretty little princess bride over there.
• 18:46
  Yeah exactly I think that word doesn't that's so classic.
• 18:52
  Your reminds me of the meme to where you have the little girl
• 18:57
  who's standing in front of the burning house and it's like
• 19:01
  worked fine in my jupyter notebook.
• 19:03
  It's an Ops problem. Now, I love Jupiter and all
• 19:07
  that, but when I like, when someone told me there was like a
• 19:10
  Export to python script function and jupyter notebooks like it.
• 19:14
  Absolutely frightened me to like no end.
• 19:17
  Like I'm like why does this feature exists?
• 19:19
  This is like I guess I understand like where it could
• 19:22
  have come from but it just like frightens me that maybe it's
• 19:26
  that evidence of like you know the disconnect between.
• 19:30
  Oh I wrote some code and did this thing between that and like
• 19:34
  software development and integration.
• 19:37
  I think it's evidence of that gap for sure.
• 19:40
  Is Skyler. I lead machine learning at
• 19:43
  Health rhythms, if you want to stay on top of everything
• 19:46
  happening in ml. Ops, subscribe to this podcast
• 19:50
  now. Now, now, now now now Now, I've
• 20:02
  kind of been perusing your goods on your podcast which is
• 20:06
  awesome. You know, various inflammatory
• 20:09
  titles such as air flow sucks for ML.
• 20:12
  Ops we learn from you guys though.
• 20:14
  Let me just tell you that I learned from you on the title or
• 20:19
  the naming conventions when Louis was on and it was like at
• 20:23
  mobs doesn't exist or something like that or ml Ops is a lie.
• 20:27
  Oh yeah. That was like these guys went
• 20:29
  and did it. Man like my life but it's
• 20:34
  amazing that we went from that title to having the mo Ops
• 20:38
  community on the podcast. That's why, yeah, I knew we
• 20:42
  needed to have a conversation because it was like, all right,
• 20:45
  I like it. And I think most people that if
• 20:49
  anyone knows me, they know that the most critical of ml obvious
• 20:53
  is myself, and I feel like I kind of have to be and that's
• 20:57
  why I love when Lewis says that kind of stuff.
• 20:59
  And we Went and we grilled them hard when he came on our podcast
• 21:04
  to because it was very much like okay, what do you mean by that?
• 21:08
  And why is that? And I'm very much trying to see
• 21:13
  at every juncture. What is going on with Hema labs?
• 21:17
  And is there something new like my new favorite thing to ask
• 21:22
  people is is ml Ops going to get her duped and that is like with
• 21:28
  chat GPT or with foundational. Those is ml Ops going to become
• 21:32
  obsolete, or it's going to become something that old Legacy
• 21:38
  companies use. And so, that's my big question
• 21:42
  right now and what I love thinking about and those thought
• 21:45
  exercises that are going through my head and so I almost feel
• 21:50
  like the pressure of needing to do that.
• 21:54
  By the way, you realize that there's a very very current
• 21:57
  version of Hadoop to, you know, that a certain Each company has
• 22:01
  a big fear of right now. I think the big fear is instead
• 22:04
  of duped, would it be Googled? Yeah, because of chat gvt.
• 22:08
  You know, you've seen all the stuff in the last few days
• 22:11
  about, you know, is the search algorithm get to be gone?
• 22:15
  Is it done for done? For I've got a bit of a
• 22:18
  contrarian stance on that. Okay, go for it when I see that
• 22:23
  coming up a lot. I'm just kind of like there's no
• 22:26
  way it's going to be replaced. Because first of all, I said it
• 22:31
  once and I'll say it again on this podcast.
• 22:33
  Like, my New Year's resolution was to be as confident as chat
• 22:37
  GPT. That's all that salt.
• 22:41
  Because let's be honest, like it's the amount of bullshit that
• 22:45
  is spews out is incredible. I want that kind of confidence
• 22:49
  when I'm spoon push have people, you know, like, give me that
• 22:53
  please. And so that's the first thing
• 22:56
  like you don't know if you can trust it but the other thing is,
• 22:59
  there's a lot of stuff. If on there like I just see them
• 23:02
  as two completely separate uses and it's going to definitely
• 23:05
  take parts of what you would Google and that is for a good
• 23:10
  cause because a lot of the stuff that you do Google right now, it
• 23:14
  seems like the user experience, isn't that good?
• 23:16
  And so I would like to have a better experience when I'm doing
• 23:20
  that. Like for example if I just want
• 23:21
  to know the recipe of whatever my favorite vegan, bean burger I
• 23:27
  can ask chat G PT and I don't have to get like this verb.
• 23:30
  Those fully SEO optimized with a ton of ads and pop-ups and
• 23:35
  everything that user experience is horrible.
• 23:37
  And so, please, like, don't make me have to go through that, but
• 23:40
  I'm feeling some role of here, by the way, as a vegan.
• 23:42
  I'm just saying, thank you for using that example.
• 23:45
  There we go. Nice.
• 23:46
  But yeah, then there's going to be a lot of other use cases
• 23:49
  where it just isn't the right medium in my opinion.
• 23:54
  I don't know about you all, but that's my take on it.
• 23:56
  Just bringing it back to the Ops perspective.
• 23:59
  Here. It's this Weird.
• 24:00
  Like, because even with like fine-tuned or like transfer
• 24:05
  learned models, where you have this, like Foundation model,
• 24:09
  type of workflow. Generally up until recently, the
• 24:14
  thought process was okay, well, I'm still performing a task and
• 24:19
  that task, like I can generate like a table of tests for that
• 24:22
  task, right? And like, minimum functionality,
• 24:25
  I can put that as part of, like, my automation for when I release
• 24:29
  this model, like, however, you do That whether that's like, you
• 24:32
  know, whatever ml Ops tools you want to use for that.
• 24:36
  And now you've sort of got this, like everybody's thinking about
• 24:39
  these generative models and like, okay, well, if I'm
• 24:43
  completely open domain, how, how do I mitigate risk in those
• 24:48
  situations with these sort of open domain models?
• 24:51
  And I think there's so I think there are ways and I think that
• 24:56
  some of the same workflows will apply, but I do think it does.
• 25:00
  Was kind of tweak your mindset a little bit to think about some
• 25:04
  of these things. I don't know if that's that's
• 25:06
  something that you all have been thinking about with the kind of
• 25:09
  most recent wave of like generative models.
• 25:12
  It's interesting because they do change the Paradigm quite a bit,
• 25:17
  in a lot of ways, there was a hackathon.
• 25:19
  So I'm based in the Bay Area, there was a hackathon.
• 25:21
  It was like, last week hosted by scale where people were just
• 25:24
  hacking on different things with a, I but a lot of them naturally
• 25:27
  because Jen AI generative. AI is a hard thing to do.
• 25:30
  There were Out of applications built around generative.
• 25:32
  A I use cases, and one of them was actually someone tried to
• 25:35
  basically build an entire back-end using nothing but LM
• 25:39
  calls. Like they were basically just
• 25:40
  making calls and updating literally data structures in the
• 25:42
  backend fully using generative models naturally.
• 25:46
  This wasn't like amazing, right? I mean in the sense there's a
• 25:49
  little bit hacky to try and update dictionaries or you know
• 25:52
  like literally objects via these API calls using things like
• 25:55
  codex and what not. But it does force you to ask
• 25:58
  this question of like, well, okay, Now, it's maybe not great,
• 26:02
  maybe in a little bit brittle, but we see this out, like, let's
• 26:05
  let you know, imagine for a second.
• 26:07
  What? 10 years from now?
• 26:07
  It looks like, I mean the 60s computers took an entire room,
• 26:10
  right? And there was only IBM that had
• 26:12
  them but here we are with them, like literally on my wrist, you
• 26:15
  know, we're in my pocket. And so it's not unreasonable to
• 26:18
  expect that like some of the ways we even do some, these
• 26:20
  workflows are going to be touched by this, right?
• 26:22
  It's going to change how we operate with our software
• 26:25
  engineering systems and by extension buyer machine,
• 26:27
  Learning Systems. I mean it, what can be done away
• 26:29
  with once we have The only interaction being like language
• 26:33
  to language style, kind of translations.
• 26:36
  I also wonder if there's like a metal layer here.
• 26:39
  We're like part of the question is how do we test and like set
• 26:46
  up the Ops around large language models and generative models?
• 26:51
  I think that's one question that can be asked, and are those the
• 26:54
  same? Or are they different?
• 26:55
  Is one of the I was going to ask that.
• 26:57
  Anyway, how do they relate? How are they not the same?
• 27:00
  Sure. And how can one feed into the
• 27:03
  other, right? Like, what you were talking
• 27:05
  about just now Mikhail. Is that there is this idea of
• 27:08
  like, how could generative models help me with my ml Ops,
• 27:13
  right? Like, let's say I'm trying to
• 27:15
  put a model into production and I'm trying to test that well or
• 27:20
  like I'm like searching through logs and other things and trying
• 27:23
  to parse that out like if I can ask an agent to like, help me do
• 27:28
  some of those things which to be honest or A predictable if I've
• 27:32
  seen them before. Yeah.
• 27:34
  It's very meta. I know.
• 27:35
  But there's like the one side like, how do I put these things
• 27:38
  into production? And as you say Chris, like, are
• 27:41
  there differences with that and the second place, like, what
• 27:44
  could I actually like bring those around the other side and
• 27:47
  help them do my, have those models?
• 27:49
  Help me do my Emma Ops tasks, right?
• 27:52
  So, I am 100% in the same boat as you.
• 27:56
  I know the CEO of you.com. I Heap.
• 28:00
  Post on Twitter, like, what is the best use of a large language
• 28:05
  model in your mind? Of course, I instantly thought
• 28:09
  like yeah, more fluency, that would be incredible if it could
• 28:15
  do that. Like, let's just be honest, how
• 28:17
  much time would it save if it could just go and set up my
• 28:23
  kubernetes cluster for me, and I've talked to a lot of smart
• 28:28
  people about that, and I think, A lot of people are like, no,
• 28:32
  that's not going to happen. Like that's too far but I also
• 28:36
  am a little bit like yeah, well, there's a lot of other stuff
• 28:39
  that we thought wasn't going to happen and it's happening right
• 28:41
  now. So I don't think that's too far.
• 28:44
  I like that is my pain Point like we're this far into
• 28:47
  kubernetes and that still hurts. Look, if you're not just using
• 28:51
  someone else's implementation, you're going to set up your own
• 28:53
  cluster. It's still painful and like that
• 28:56
  this is known stuff. It's just a pain in the butt.
• 29:00
  That's a great thing, right? I don't think that's too far at
• 29:03
  all and I want it now. Yeah, exactly.
• 29:05
  Maybe it's just like a fine-tuned away.
• 29:07
  Let's be honest. Maybe that's all it is.
• 29:09
  And somebody needs to think about that.
• 29:11
  I'm sure I'm not the only one that has thought about that.
• 29:13
  And so, but all that mr. Eric chime into, I know, he has
• 29:18
  some strong thoughts and all like, preempt his thoughts on
• 29:23
  this with his idea of an incredible app on top of
• 29:29
  generative models. Was stealing somebody else's IP,
• 29:33
  you want to tell us about what you've created?
• 29:35
  Mr. Eric, that sounds perfect. And by the way, just disclaimer.
• 29:40
  I'm not implicating myself in any of the following
• 29:43
  conversation. This is not hacking advice.
• 29:48
  Oh, goodness. This show.
• 29:50
  Just took a turn. Oh, boy, I didn't realize I was
• 29:54
  gonna be basically a kidding myself criminally by coming on
• 29:58
  this podcast, but we're going to edit this out, right?
• 30:00
  This is edited out sure. Sure will say that now, but we
• 30:03
  didn't attract of any type, so and there are no intelligence
• 30:07
  agencies listening, I promise, absolutely not, right?
• 30:11
  Absolutely not. Yeah.
• 30:13
  I mean, I guess I'll maybe spend a little bit of time talking
• 30:16
  about the use case that I think Demetrius is referring to, which
• 30:18
  is a little bit of a toy use case.
• 30:20
  We didn't do it because we were trying to get sued.
• 30:23
  That wasn't the goal but hopefully it should spur a
• 30:26
  little creativity. What a start.
• 30:31
  Well okay maybe maybe like ten percent we were trying to get
• 30:33
  sued. No use case to give an idea of
• 30:36
  what has been come possible outside of you know yes the
• 30:40
  really practical use cases generating yeah Mille generating
• 30:44
  unit tests. I think these are all in a very
• 30:46
  on a very serious note. These are things that are going
• 30:48
  to be possible. I actually don't see this being
• 30:50
  as too far off and I don't know, not even like a decade.
• 30:52
  I see this being like, next, few years, being able to spin offs
• 30:55
  things like that, is just going to be totally feasible.
• 30:57
  You know, this is like language-based kind of
• 30:59
  generation. Of modeling.
• 31:01
  The other one that we haven't really touched on as much, which
• 31:03
  is also really capture, people's interest has been more of the
• 31:06
  image based right like Vision based whether that's a single
• 31:09
  static image or that's entire video orienting some cases even
• 31:13
  audio just like different modalities beside pure text and
• 31:16
  so one that has certainly become really interesting because of
• 31:18
  the rise of things like Dolly from open Ai and then stability,
• 31:22
  you know, stable diffusion as these been these like incredible
• 31:25
  photorealistic images in different styles that you can
• 31:27
  just prompt with literally human.
• 31:30
  Straight, which is never been possible for we're at least not
• 31:33
  at this level of quality. And so one of the things that my
• 31:36
  co-founder I actually becomes. Yeah.
• 31:39
  Yeah that we were working on was a little toy application that we
• 31:43
  called Rick and mortify. And the basic use case was we
• 31:47
  were big fans of the Rick and Morty TV show and you know we
• 31:50
  love it I think it's super like it's very great.
• 31:53
  Great great content that did not stop a cease and desist letter
• 31:56
  from coming. There there's a There's a back
• 32:01
  story here that I found out after we released the
• 32:03
  application, which if you guys are curious, I'll go into.
• 32:05
  But we were really trying to test this hypothesis.
• 32:08
  Like, okay. So you have these Vision based
• 32:10
  models that are incredible. You have these language based
• 32:12
  models that incredible, how can we like merge them to actually
• 32:15
  try and do something between a kind of at the intersection of
• 32:18
  them too. And so we came up with was like,
• 32:19
  well, can we personalize episodes of Rick and Morty?
• 32:22
  Like if I as a super fan of the show, one to imagine a new
• 32:27
  episode, if I provided a premise, if I provide You know a
• 32:30
  set of characters that I wanted maybe myself eventually but
• 32:33
  let's just start with the basic characters that are in the show
• 32:35
  already. Rick Morty summer you know poopy
• 32:37
  butthole mr. Meeseeks like all these folks
• 32:41
  that is an actual character named.
• 32:42
  I just want to point out for the audience I did I just make up a
• 32:45
  name to try and be profaned. That is a character stupid
• 32:48
  butthole and and and true we've ever had that word on the show
• 32:54
  before I actually is I did owes a word until the show so you
• 33:00
  know, We're both in this world new to this game.
• 33:03
  What we essentially built out was, you could come with this
• 33:05
  application. You could provide a premise of
• 33:07
  what you would like the episode to look like.
• 33:08
  So Rick and Morty, go to the Practical AI podcast and have a
• 33:11
  great conversation about gender of AI and then you pick your
• 33:14
  characters and using a combination of vision-based
• 33:18
  genitive systems stable, diffusion Dolly, and then the
• 33:21
  GPT 3s, right? The, you know, these kind of
• 33:23
  latest generation of GPT models. We were able to Journey, not
• 33:25
  only the visuals were able to generate a script of effect,
• 33:29
  effectively a storyboard. New episode.
• 33:31
  This is like a first use case like well we're literally
• 33:34
  getting new episodes, you know? Maybe there's like five ten
• 33:36
  frames of this episode but you're seeing some flavor of a
• 33:40
  plot with dialogue with accompanying visuals and you
• 33:42
  know sure it's a little bit rough but extend this out to the
• 33:46
  Future, right? Like what happens then like this
• 33:48
  has become literally a fully-fledged episode and that's
• 33:52
  where this could become, right? And so no cease and desist yet
• 33:55
  but people did play with it. People had a lot of fun with it.
• 33:57
  We're not making any money on it.
• 33:58
  So, you know, don't worry, Dan And Justin like it's it's just
• 34:02
  for fanart Friday. We're just trying to show our
• 34:04
  appreciation for the show you raise something.
• 34:07
  It's kind of funny because when it just in day-to-day
• 34:10
  conversations about AI topics and like new model comes out and
• 34:15
  you see all the media stuff where people are just kind of
• 34:17
  bashing it until you all the problems with it and everything.
• 34:20
  And to your point, you just said this, that's why I'm bring it
• 34:23
  up. It's like, but think of what we
• 34:25
  can do with this tomorrow. Like today we have this and it's
• 34:29
  imperfect. Yesterday, we didn't have any
• 34:32
  such thing, think about. If today, we have the imperfect
• 34:35
  thing, tomorrow's could be pretty amazing.
• 34:37
  And to your point, they're like, what you're doing?
• 34:39
  There is like, way out there, but it's not far from being very
• 34:43
  mainstream. So my question about that is
• 34:46
  like you have the training data and you have the base show that
• 34:52
  a human made, right? And so, without that, like, I'm
• 34:56
  trying to extrapolate 10 years in the future and if it's Just
• 35:00
  going to be us. Remaking, a bunch of shows that
• 35:03
  we made back when we used to do it all by hand.
• 35:07
  Or is it like that's that's kind of my question there, right?
• 35:12
  I think that's a generative function right there.
• 35:14
  I don't think you have to start with where you were in the sense
• 35:17
  of part of that function at the very front end of that workflow
• 35:20
  is going to be generating different possibilities that are
• 35:24
  not directly linked to the training data and then like
• 35:27
  carrying it from there. And I think, I think it's a few
• 35:30
  Sure entertainment. I'm Emmanuel, Mason machine
• 35:35
  learning engineer, a stripe and author of building machine
• 35:38
  learning, powered applications. And if you don't want your
• 35:41
  machine learning models to explode, you should subscribe to
• 35:45
  this podcast. What I think is kind of
• 36:01
  disconcerting is the right term. I don't know what the time scale
• 36:05
  on this or whatever. But like imagine like all these
• 36:08
  data sets around language models, especially are scraped
• 36:10
  from the internet, right? And like a lot of these image
• 36:14
  data sets are scraped from the internet, right?
• 36:17
  So there's a proliferation of these models and The internet is
• 36:21
  being filled with computer generated content, right?
• 36:25
  So the next scrape versions of the internet, like the next
• 36:29
  common crawl, the next whatever. Like, What?
• 36:32
  Proportion of that data set is coming out of generative, AI
• 36:36
  models. Now, I think there's interesting
• 36:39
  things going on, of course, with like detecting, what is a i
• 36:43
  generated and what isn't, I think?
• 36:45
  I've seen like what open a I came out with for their own
• 36:49
  system. And of course, like you Take
• 36:50
  Chris, everybody has like criticisms of that already, but
• 36:54
  there's like many, other people exploring this as well, like, G,
• 36:58
  PT 0, and like other things that are exploring like, how do we
• 37:02
  determine? What is a i generated?
• 37:04
  And not. And so, that's where I like, I
• 37:07
  don't know what the implications of that are for these.
• 37:10
  Large scale data sets down the road, but that's sort of where
• 37:14
  my mind is going more. So than, oh, should we pop be
• 37:18
  populating the internet with a lot of this generative?
• 37:20
  Of Art and other things. Even if it's terrible, well, I'm
• 37:24
  having a lot of fun with it right now, but there's sort of
• 37:26
  second order effects. I guess.
• 37:28
  No worries than man. Yeah, well I'll be dead by that
• 37:32
  time. I don't know, maybe no no, no,
• 37:36
  this will be soon. It's a really interesting
• 37:39
  thought experiment and again, one that is closer than I think
• 37:42
  we believe this idea of what happens when the book of these
• 37:46
  data sets are actually generated by systems and I can see you
• 37:50
  Some positive effects and some that you would be like, okay,
• 37:53
  this might be a questionable downside, on the positive side,
• 37:56
  you know, these models like at the generative ones especially
• 37:58
  the language. Variety are like very good.
• 38:00
  I mean grammatically correct right there.
• 38:01
  Like very semantically good in terms of what they output does.
• 38:05
  This mean that now we the bulk of the data that's being trained
• 38:07
  on is just going to be like way higher quality writing, then you
• 38:10
  would typically find in a typical Reddit post, right?
• 38:12
  Or something, just in the deepest corners of the web and
• 38:15
  then after training on data, that's significantly better.
• 38:17
  You have this compounding effect of like, well now the daytime
• 38:20
  training on This better. So the models only real even
• 38:21
  better at writing of some kind and then we mix it in with some
• 38:24
  other more diverse writing and then is going to continue to
• 38:26
  pound on itself in terms of quality, the downside.
• 38:29
  I mean, one you could hypothesize is that like these
• 38:32
  models are only able to generate certain kinds of distributions
• 38:34
  of data, right? Only certain kinds of things
• 38:36
  they can write about or talk about.
• 38:37
  So now when you're injecting all these training sets with like a
• 38:41
  very skewed distribution of topics of ideas on these topics
• 38:45
  etcetera, how do you ensure that actually you're still giving
• 38:48
  your model enough of a versatility and what it He's is
• 38:51
  next Generations that are then trained on these data sets to
• 38:53
  ensure that it's still a general-purpose model right.
• 38:56
  Like what if 90% the content that's put out there is just
• 38:59
  really bad marketing copy, you know, Facebook ads or something
• 39:03
  over fit. Yeah, it'll over fit exactly to
• 39:05
  that. And then what happens to these
• 39:06
  next-generation system? They actually might be heard in
• 39:08
  the long run? Yeah.
• 39:09
  It's like we're in the Golden Era.
• 39:11
  Yeah. It I think that you can have an
• 39:14
  Mo Ops community in my opinion for a long time because there's
• 39:17
  such a wide variety of problems that people are.
• 39:20
  Are still dealing with so on this like very far.
• 39:23
  And we're talking about generative, AI being part of
• 39:26
  like my day job, as part of an international NGO, like we're
• 39:30
  dealing with problems still like we're, hey, there's no internet
• 39:34
  in this place and like, we're running our model on like an
• 39:37
  Android tablet, or like, whatever.
• 39:39
  Like there's this range of like, okay.
• 39:43
  What's the problem that happens when I scrape the whole
• 39:45
  internet? Again, all the way down till
• 39:48
  like, how do I run the small? On an Android tablet and I don't
• 39:53
  actually see that changing for some time.
• 39:56
  Yes. The world like changing a lot,
• 39:58
  but there's still like such a wide variety of issues.
• 40:02
  And I think fun challenges to like wrestle with around this
• 40:07
  concept of like, operations, plus Ai and machine learning.
• 40:12
  And so, yeah, it is sort of hard to Define in that sort of way
• 40:15
  but it's also really exciting because of the wide variety of
• 40:18
  things that you could be involved with.
• 40:20
  With like, as a devops person or software engineer, or data
• 40:25
  scientists, like, there's plenty of problems that have to do
• 40:28
  with, like running even like a much older language model on
• 40:34
  your phone and like other problems that have to do with
• 40:38
  like various different scales or different modalities of data,
• 40:42
  all these things. It's, yeah.
• 40:44
  And that goes back to kind of in the pre-show that would our
• 40:48
  listeners were not. Therefore, we kind of talked a
• 40:50
  little At about that level of diversity.
• 40:53
  Like how do you guys see that? So I mean, Daniel, you've kind
• 40:56
  of talked about almost an extreme case of kind of edge
• 41:00
  concept, you know, being your target, you know, and having
• 41:03
  this slow Android thing. And yet we're used to being just
• 41:06
  like gluttonously resource in the cloud, you know, everything
• 41:10
  you could possibly want their ramp it up, ramp it up, you got
• 41:12
  all the stuff you want there and you have this incredible
• 41:15
  diversity. So, for you guys thinking about
• 41:18
  ml Ops, you know, so So much. And how do you deal with that?
• 41:22
  Like when you go from big company to one person struggling
• 41:27
  to work at all? How does ML Ops look when you're
• 41:30
  talking about diversity of use cases, plus diversity of users
• 41:34
  and that you're serving, how do you make it a thing?
• 41:37
  You know, how do you keep it all together?
• 41:38
  There, if you a few answers this question, actually.
• 41:41
  So the first part of the diversity of use cases and in a
• 41:44
  sense, we've gone a little bit backwards from something we said
• 41:47
  before. We're on the one hand were like,
• 41:49
  oh, generative systems are just going.
• 41:50
  Do away with all of this. And then now we're back at like,
• 41:53
  well but it's, there's like enough stuff that we still need
• 41:55
  to solve that it'll probably be around both and we're not quite
• 41:58
  there yet. Exactly.
• 42:00
  Exactly. And to that point, I actually I
• 42:02
  was actually asked this in a podcast once a few weeks ago
• 42:05
  where I was asked like, what do I believe will be the position
• 42:08
  like the role that will exist 10 years from now?
• 42:10
  And I was asked that will be like a data scientist, a prompt
• 42:12
  engineer and Emma Watson jr. Like where would I put my money?
• 42:15
  And I still answered ml Ops engineer like machine learning
• 42:17
  engineer. You know, like kind of in that
• 42:19
  in a similar category. Because I do believe that these
• 42:22
  same problems will persist whether or not there for
• 42:27
  old-school decision tree based models.
• 42:29
  Distributed models that we use, you know, maybe five ten years
• 42:31
  ago even pre deep learning or these new GPT 500, whatever will
• 42:38
  come later models, stable, diffusion 10,000, you know, I
• 42:41
  mean, like the same problems will persist.
• 42:42
  How do we, how do we operationalize it?
• 42:44
  How do we make it scalable? How do we keep up time on
• 42:47
  models? That people interact with this?
• 42:49
  These are all questions that. Let you know that I'm Ops is
• 42:51
  fundamentally trying to solve and address and whether or not
• 42:55
  you're using kubernetes today or some prompt engineering-based
• 43:00
  kubernetes tomorrow or you know, make it even more concrete like
• 43:04
  airflow today versus like an air flow for prompt engineering,
• 43:09
  right? Which is what people are
• 43:10
  actually developing today. The same sort of principles and
• 43:13
  the same concerns are going to apply.
• 43:16
  And so I don't see that going away anytime soon, as long as
• 43:18
  there's a machine Learning System as long as you know, we
• 43:20
  We I would assume all of us here in this room in this virtual
• 43:23
  room, anyone who's listening here believes that AI is going
• 43:25
  to be the future? It's going to be here for
• 43:27
  decades then. The same questions will still
• 43:30
  have to apply. And so that's like the first
• 43:32
  part I want to just, like, throw out there is.
• 43:34
  There's a mops is here to stay whether or not it's kubernetes
• 43:37
  or whatever comes after kubernetes.
• 43:39
  We hope it comes soon. We hope it comes soon.
• 43:44
  Someone needs to get it soon. I have a tip there that I'll
• 43:48
  provide later. Yeah, I don't know if you You
• 43:50
  all have seen, what Eric Bernards in is doing with a
• 43:53
  modal and modal Labs. I've been playing around with
• 43:56
  that recently. I've been pretty floored by it.
• 43:59
  But anyway, that's a whole nother side topic and episode,
• 44:02
  which, hopefully we can have soon, but totally, but I think
• 44:04
  to me the second part of the question, which was how do
• 44:07
  different organizations, think about, maybe the M Ops question
• 44:10
  and something that we do want to address here which is it does
• 44:13
  depend? It really does depend in terms
• 44:15
  of the the maturity over the organization and budget time I'm
• 44:20
  Etc. These are all different axes.
• 44:22
  That fundamentally Define how a team, or a business should think
• 44:26
  about its approach to ml Ops and there's different axes that we
• 44:31
  can go into exactly what they are.
• 44:32
  But you know, open source versus not open source.
• 44:36
  Like are we gonna use something up here?
• 44:37
  You can stitch together bunch of Open Source tools or you can use
• 44:39
  Sage maker out of the box. How much money do you have?
• 44:42
  Can you get by with just spinning everything up on your
• 44:45
  own? All these different, axes,
• 44:47
  different organizations, have to think through and it becomes Not
• 44:50
  a one-size-fits-all you really is like it's a function of all
• 44:54
  these different parameters to really tailor the right solution
• 44:57
  to the organization. And there you go, you didn't
• 44:59
  realize that mr. Mihael has a little consultant
• 45:04
  in his blood. Do I get to put another plug
• 45:07
  here? Another Shameless plug?
• 45:08
  Is that what this is and imitate?
• 45:09
  Now? It's many as you want.
• 45:11
  As many as you want, that is the most Consulting answer you can
• 45:16
  possibly give to any question is like, well, for this rate.
• 45:20
  And I can tell you more details on what comes after.
• 45:23
  Yeah, exactly. You want to go at it?
• 45:25
  But I think what, I just will add a quick piece to that, which
• 45:29
  is, there was a time in the nlrb's community that like a
• 45:33
  week wouldn't go by in our slack where someone would not share
• 45:40
  the you are not Google blog post and it's like the amount of
• 45:45
  people that try and go at it and try and get that especially
• 45:49
  because It's out so much great thought leadership on ML Ops and
• 45:53
  they have the 0 level 0 level 1 level 2.
• 45:57
  Or they have the like ml test score, all that stuff.
• 46:01
  And people think that straight out from 0 to 1, you need to be
• 46:07
  creating everything. Automated it needs to be like
• 46:10
  the most high performance bulletproof system that you can
• 46:15
  think of. And it was just setting up a lot
• 46:19
  of people for failure and I think we've moved past that
• 46:22
  because I haven't seen the you are not Google blog post being
• 46:26
  shared as much in the community which makes me think people
• 46:29
  recognize and they're a little bit more self-aware when they're
• 46:32
  trying to create their systems at their jobs, you don't think
• 46:35
  they're just off in the corner, crying.
• 46:38
  Well, or, or they just don't think being Google is a good
• 46:40
  thing anymore. Just objectively given give it.
• 46:43
  Yes, I might be into, I think I was listening to, I don't know
• 46:47
  if you all. Listen to Indie.
• 46:50
  Ha. His podcast, but recent episode
• 46:52
  at, like, I think they made this good point they're talking about
• 46:55
  like, because they talk about a lot about like, bootstrap
• 46:58
  startups, and stuff like that. And and one of the points they
• 47:02
  made was like, you know, whenever like base camp was
• 47:05
  around, or like, starting up in that sort of thing, like they
• 47:09
  made such inflammatory statements about like, like
• 47:12
  taking venture capital is stupid like why would you ever do it?
• 47:16
  Like there was a need for that voice to be in there right now.
• 47:20
  Voice is still there with the founder, you know.
• 47:22
  Yeah. But now like you hear less of
• 47:26
  that, I think it is sort of like normalized and that like,
• 47:29
  they're still different perspectives, but there's not
• 47:31
  like as much like it's become more normalized to have these
• 47:35
  like, more nuanced discussions. I think with Emma Ops there's
• 47:39
  sort of like, you know, we've all made our sort of
• 47:43
  inflammatory like podcast titles, like, whatever it was ml
• 47:47
  Ops is dead or, you know, I forget what it was.
• 47:51
  And so there is probably like still this we're kind of feeling
• 47:55
  out where like the normal is and where things settle down to and
• 47:59
  providing like a balanced perspective, where?
• 48:02
  Yeah, like, you probably shouldn't be doing ml Ops the
• 48:06
  same way. Google is, but you also should
• 48:09
  probably be doing Mo Ops. It's just like, we're on that
• 48:13
  Spectrum, do you land and what's type of tools make sense for
• 48:17
  you, you know, in real life, I don't think it's, you probably
• 48:20
  know, Should not be I think it's you can't do ml Ops the way
• 48:24
  Google does. I mean just from a resource
• 48:26
  standpoint the you know most companies don't have that team
• 48:30
  available and that set of tools and you know I like the fact
• 48:33
  that we're talking it in a more realistic context here you know
• 48:37
  for the vast majority of us out there that are not you know
• 48:40
  accessing the best of the best in all the categories, it's not
• 48:44
  possible. For many of us, we have to
• 48:46
  settle for something that's doable.
• 48:47
  Kind of going to Demetrius is point.
• 48:49
  You know you have to And that level that you can do it.
• 48:52
  You can sustain it. And yet it's still incredibly
• 48:54
  productive even if it's not the Google version and one thing I'm
• 48:58
  fascinated by just because there is almost this open source,
• 49:04
  you've got one person trying to hack something together and
• 49:09
  looking out there and seeing what's on the market that they
• 49:12
  can get for a price point between free and cheap.
• 49:17
  And then you've got the Googles that have built everything.
• 49:20
  A thing and have so much time and ability to do that.
• 49:25
  And in between those two points, you have a lot of companies that
• 49:30
  popped up and they popped up in the last, like, three to five
• 49:33
  years. And as me Kyle was saying, we
• 49:37
  came at ml Ops from this data science perspective.
• 49:40
  And so I think in the beginning a lot, we're trying to cater to
• 49:44
  them and then some were like, whoa, wait a minute, there's
• 49:47
  like platform engineers. And so they try to Cater to
• 49:50
  them. And then you're seeing now, I
• 49:54
  almost wanted to, like, change my title for what I do on
• 49:58
  LinkedIn as I'm just going to be like the I ride hype waves
• 50:03
  because ml Ops was a complete hype wave and we felt it and
• 50:08
  especially like I just got lucky because the pandemic hit and
• 50:12
  right when it hit, I was working for a company that was trying to
• 50:15
  sell ml Ops tools and that company went out of business but
• 50:19
  I was in the Ml Ops World. So I figured I would start this
• 50:22
  slack community, and then it took off, right?
• 50:25
  And so, I was able to ride that hype wave and we really felt it
• 50:28
  over the last 23 years. And now it's like, all right,
• 50:32
  now there's this generative, AI hype wave.
• 50:35
  And so, if you talk to VCS, and the ones that poured a boatload
• 50:39
  of cash into the ml Ops Market there now, like, yeah, that's
• 50:43
  kind of not really that big of a deal anymore.
• 50:45
  We're not going to follow up on all those Investments.
• 50:48
  What we're going to do is invest in the Next day I too.
• 50:52
  And so I love thinking about that and how now what is the new
• 50:59
  hype cycle of the AI and generative AI link large
• 51:03
  language models like where's that going to play out?
• 51:06
  You know, we've been talking about it if there's a lot of
• 51:08
  potential out there but where we going to go with that and is it
• 51:12
  like just a bunch of money pouring into it and who knows
• 51:15
  what's actually going to happen? It's kind of funny when you say
• 51:18
  that and it's like there's An over simplification, you know,
• 51:22
  when the market observes these things, you know.
• 51:24
  And you know ml Ops will now we're past that, you know,
• 51:27
  that's been solved. You know, we're going to web
• 51:30
  three, you know what, whichever hype cycle it is.
• 51:33
  But I think when you look at what's happening, it takes all
• 51:37
  these things to have gotten you right now we're saying
• 51:39
  generative because that's kind of the sexiest part of the
• 51:42
  puzzle, but it's not just generative.
• 51:44
  It's the fact that you have ml Ops now, that's matured a little
• 51:48
  bit and is supporting all that. Have the large language models
• 51:52
  that people like Daniel have been working on.
• 51:54
  If you have all the Transformers that are well established, if
• 51:57
  you didn't have all of those components, the current height,
• 52:01
  that's now being attributed to generative would not be
• 52:04
  happening. And so, it's once again, an over
• 52:07
  simplification by the market on the sexy piece.
• 52:10
  But it's that whole ecosystem, that's evolved over the last few
• 52:13
  years that enabled all that to happen.
• 52:16
  And so we're seeing a really cool moment, definitely But it's
• 52:20
  really the fusion of it all as opposed to just being
• 52:23
  generative, you couldn't have generative today, without that.
• 52:27
  I think that's such a fantastic Point, like, as one likes it on
• 52:30
  that a little bit longer because when the gbd3 paper came out,
• 52:34
  this was like a toward the end of 2020, maybe middle to end of
• 52:37
  2020. This is like long thing that,
• 52:40
  you know, I remember reading and the most interesting in my mind,
• 52:45
  achievements of that whole system was not even anything
• 52:48
  about modeling, right? Fundamentally the building
• 52:51
  blocks of the AI architecture. If you want to call it, that was
• 52:53
  just we've been using these systems for years in some sense.
• 52:57
  It was really the fact that they had built this incredibly good
• 53:01
  software, infrastructure to build out, train large-scale
• 53:05
  systems at the scale over this many gpus at this latency to
• 53:09
  make sure that, you know, the updates for the gradients could
• 53:12
  happen fast enough. That this wasn't a take 50 years
• 53:14
  to train and they did that all which fundamentally was real
• 53:18
  like an ml Ops challenge. I mean, at its core I agree.
• 53:20
  You know, being able to Archive that kind of a system is the
• 53:23
  complexity. It's not the fact that there was
• 53:25
  this new scientific achievement that we really came up with it
• 53:28
  was really like an engineering achievement and so in that way,
• 53:30
  it totally was a layering on of things that we had seen before,
• 53:33
  but now they're like old hat, right?
• 53:35
  Like now we have open source repos that can literally
• 53:38
  approximate the same effect. Like deep speed from Microsoft
• 53:41
  is beating really fast. And then being very used, you
• 53:44
  can train large systems at that scale.
• 53:46
  But the people that pioneered those systems really had to
• 53:48
  solve some MLS challenges in their own right?
• 53:50
  It's all right there, they haven't gone away.
• 53:53
  You know, it's still right there.
• 53:55
  It's a team effort for all of those different parts,
• 53:58
  constituent parts to put the whole together.
• 54:00
  But at this moment, the word generative is the front, man.
• 54:03
  Apparently, you know, as we record this today, it's pretty
• 54:07
  amazing. That is that ml Ops take that
• 54:10
  has put that all together and kept it together, kept them in
• 54:14
  lockstep so that you can create the new things of today.
• 54:17
  Yeah, I mean, another example of that is like All that we're
• 54:20
  hearing about about language model chaining and that sort of
• 54:23
  thing. Like this is like if you look at
• 54:26
  each step of that process right length chain or whatever it is
• 54:31
  those are things. We've been talking about for a
• 54:32
  long time like there is an operational burden, though to
• 54:37
  like Cheney things together and make them work well like in
• 54:40
  concert, right? That's really like one of the
• 54:44
  fundamental things that we learned, like, when we talked
• 54:46
  about Chad GPT, right? Like the, like the language
• 54:49
  model, This said reinforcement learning existed human feedback,
• 54:53
  existed like all of these things existed like the chaining of
• 54:56
  them together in a certain workflows, the real interesting
• 54:59
  piece. So I do think that's really
• 55:02
  exciting where a lot of those sort of chaining operations and
• 55:06
  like, bringing things together in unique ways that is a lot
• 55:10
  more possible because the tooling has gotten better, it
• 55:13
  doesn't like move us past Emma Ops.
• 55:15
  Actually it just like we use Emma Ops slightly differently.
• 55:19
  And if anything I think It becomes more crucial because
• 55:22
  there's so many moving pieces, right?
• 55:25
  Shoulders of giants all the way through.
• 55:26
  Yeah. You combine that with the
• 55:28
  Einstein, Turtles. All the way down kind of thing.
• 55:30
  It's shoulders of giants all the way down.
• 55:31
  Its yeah, the one thing to add me to that is very often.
• 55:35
  When I have described sort of the progression of a lapse over
• 55:38
  time. Like, people often, like, and
• 55:40
  use the Gartner hype cycle, right?
• 55:41
  To describe Trends in technology, right?
• 55:43
  You have a kind of the initial hype, then the waiting in the
• 55:45
  kind of the trough of disillusionment and then this
• 55:47
  gradual climb upwards, right? I've often said the right now
• 55:51
  where we are. We're at that gradual climb
• 55:52
  upwards like the Emma UPS systems and the technology
• 55:55
  developed are like standardized and they're becoming more
• 55:57
  commonplace and people are using them.
• 55:59
  But the thing is that linear climb upward, while it is where
• 56:03
  a lot of values. Extracted is not really what a
• 56:06
  VC would look for. They would want this, they want
• 56:08
  something that's like exponential and growth and so
• 56:10
  they're never going to ride a hype train.
• 56:12
  That's like a linear climb upwards even if that's where
• 56:15
  people are actually deriving value.
• 56:17
  They're like, well, what about how do we get to the next thing
• 56:19
  in the next Gardner? Hi.
• 56:20
  Cycle where there's the next big inflection point up that we need
• 56:22
  a ride because that's where you get real 100x gains.
• 56:25
  If you know that's the kind of thing they're looking for.
• 56:27
  Yeah it is. Ironically the place where you
• 56:29
  find all that value there. After that initial height is
• 56:33
  from the trough of disillusionment, you know,
• 56:35
  that's where you kind of go. Now I understand now I know what
• 56:39
  we really need to do and you get that really good growth after
• 56:42
  that, that's a great Point. That's a great Point.
• 56:44
  Yeah, this has been amazing. Yeah, this has been awesome.
• 56:48
  I have not needed. To carry the conversation at
• 56:52
  all. So I appreciate you all doing
• 56:54
  the heavy lifting for me. Usually, I am constantly
• 56:58
  thinking about that. And you made it very easy today
• 57:02
  so I appreciate that. This was a fun conversation that
• 57:05
  this is awesome. Do this more often.
• 57:07
  Yeah, we should. We should Circle back and see if
• 57:10
  any of the things that we said actually are true next year
• 57:15
  which may be some of them. I'm going for like 25 percent.
• 57:18
  If I get there, I'm good. Well, I didn't put my money on
• 57:21
  any of these predictions so I don't really care.
• 57:24
  That's right. You've already explained that
• 57:26
  you're not culpable or I'm not gullible on the hook for
• 57:30
  anything, Rick, a trick, and mortify, it's just a toy
• 57:33
  project. Don't worry about it.
• 57:35
  I come in, peace. I keep waiting, you know,
• 57:39
  because we can see each other on video already.
• 57:41
  I know it's an audio recording, but I keep waiting for the feds
• 57:44
  to busting behind you on your screen there.
• 57:46
  You know, you know, the door goes flying back the different
• 57:49
  sort. BBC kid moment.
• 57:53
  Yeah. And then my bike just goes out
• 57:55
  and it's like well he'll be back.
• 57:57
  Probably maybe he'll be back. That's right.
• 58:00
  But then Demetrius would have to carry the conversation.
• 58:03
  Yeah, that would not be good. I'd be come looking for you.
• 58:07
  Well next year, we'll see if this or any of the other things
• 58:10
  come true on practical, AI plus ml Ops Community.
• 58:14
  Definitely check out the show notes for those practical AI
• 58:18
  listeners out there. R.
• 58:20
  We're going to include links to all the great things going on in
• 58:23
  the ml Ops Community, slack Channel, podcast, events
• 58:27
  newsletter, all sorts of amazing stuff.
• 58:30
  And yeah, thanks guys. It's been awesome to have you on
• 58:32
  the show like what? Yeah, pleasure.
• 58:20
  R. We're going to include links to
• 58:22
  all the great things going on in the ml Ops Community, slack
• 58:25
  Channel, podcast, events newsletter, all sorts of amazing
• 58:29
  stuff. And yeah, thanks guys.
• 58:31
  It's been awesome to have you on the show like what?
• 58:34
  Yeah, pleasure.