Latest / Agentic Conversations (formally mlops.community) / The Long Tail of ML Deployment // Tuhin Srivastava // #161

Transcript

The Long Tail of ML Deployment // Tuhin Srivastava // #161 · Agentic Conversations (formally mlops.community)

0:00
All right, listeners, I am so excited to announce that the MLS
0:04
Community has partnered up with Quantum black, the AI branch of
0:08
McKinsey & Company to show you some of the incredible things
0:13
that they're working on. And for the next couple moments,
0:17
I brought on a partner from Quantum black närcon to come and
0:22
talk to us a little bit about what they are doing.
0:24
What they've been up to. All right.
0:26
Your official title neither is partner.
0:29
But Your unofficial title is professional data nerd.
0:33
I want to know exactly what is quantum black.
0:37
I mean, if I got a little bit about our Origins, we started
0:40
off in Ahmedabad. We realized very early on that
0:43
we could use. They are an AI in very smart
0:46
ways off the track to have big impact on track times.
0:50
So then we took that same mentality of using data and AI
0:53
very smart way to help companies across Industries on some of the
0:57
toughest, a I use cases to really drive in And just some
1:01
examples like that could be in sports and that could be for
1:04
example, team composition or injury, prevention or even
1:07
racing could be natural resources, for example, oil
1:10
fields or energy production, optimization health, and Life
1:14
Sciences is it's a big Topic in terms of R&D of new molecules
1:18
but also optimizing clinical trials.
1:19
So instead of 10 years driving all the way down using data in
1:23
AI to fraction of the type. So we're about 2,000 colleague
1:27
globally we've expanded now to over 40 Countries.
1:30
And when we talk about the kind of profiles we have is data
1:33
science data engineering if standard ones but also devops
1:36
software Engineers product design, SRE clouds a whole
1:41
football, team of different technologists and and skills to
1:45
come together to really solve some of the toughest AI problem.
1:48
And when you say football, you mean American football or you
1:51
talking about the American Film? Because I know you're an
1:53
American football fan. So, I'm okay with that.
1:57
So, how does Quantum black Approach scaling AI.
2:01
I know you all are very, very deep in this subject.
2:04
I think sometimes always distill down to just technology but it's
2:07
more than that and I would argue that it's people and process to
2:10
which is sometimes even more important.
2:13
But because of your listeners, maybe, I'll just talk in a tech
2:15
first. One of the nice things with to
2:18
be right now, today's that with technology agnostic.
2:20
So our clients are the organization, we work with.
2:23
They have a myriad of different types of technologies that could
2:25
be on premise or that could be on the cloud.
2:29
Tech gets you any so far. So really, it's helping
2:32
organizations optimize the process of building Ai and
2:35
bringing this, this football team of talent.
2:37
I mentioned earlier, but also helping organizations build
2:39
their own football team about it and trying to standardize and
2:43
automate the process in to win like removing the friction
2:46
between teams. So, in a really simple way,
2:48
taking organizations that take, maybe months and months to build
2:51
something and putting to use his hands, but actually getting it
2:54
down to date. That's a goal one of the final
2:57
points, I'll just mention Don this like I'm having this one of
3:00
our really strongest value propositions.
3:02
And some things we redouble down is actually building the right
3:05
thing. So not, you know, building Tech
3:08
in and hoping the users become but actually, we're focusing on.
3:11
Is this a problem? What is the problem?
3:12
We're trying to solve with AI and data and then working our
3:15
way backwards to what tech do we need?
3:17
What data do we need talent ways of working, etc, etc, listeners,
3:21
you all heard it here first Quantum, black is partnering
3:25
with the emilybs community. If you want to go deeper into
3:28
Anything that they are doing, we're going to leave a link in
3:31
the description below for you to go down the rabbit hole and also
3:35
they're always looking for incredible Talent.
3:37
So if you feel like you fit the bill will leave an email for you
3:41
to reach out and get in touch with them.
3:45
Having me. Let's get back to the show.
3:47
Yeah, my name is Dewey and srivatsa house.
3:50
I'm the one of the co-founders and CEO of base 10.
3:55
That basically means I do a lot of Of nothing and a little bit
4:00
of everything. And the, how I drink my coffee.
4:04
I drink a black eye, but, you know, a straight like I really
4:10
enjoy coffee. So I would drink any formal
4:12
coffee. I like espresso.
4:15
But, you know, being in America long enough, you end up just
4:18
drinking a lot of black coffee, which is a very American thing.
4:22
Hello, everybody. And welcome back to the
4:25
envelopes Community podcast, I am Demetrios.
4:29
I am joined by none other than Abby.
4:32
What is going on a be recording the card and show today.
4:37
So the Is great. Where's the disk that you
4:40
engage? That is true listener.
4:43
If you were wondering we just pound it out like three
4:47
different intros and we're getting pretty good at it.
4:50
Right now this introduction though we just got off the call
4:55
or the recording with 2 Min and I loved it.
4:59
I really was fascinated by some of the stuff that he was talking
5:03
about. Did you have any key takeaways
5:05
that I can add on to? What were you what did you like
5:07
from the car? Or Sation actually, when I was
5:10
looking at base 10, I can't give built a platform for machine
5:14
learning engineers and Engineers to not have to do the boring
5:19
stuff, which is the interesting stuff.
5:21
According to some people boring stuff, this is especially their
5:24
cyclist, but it's like he had a very different perspective which
5:28
was based on their research with a lot of Stanford students.
5:32
And I'd like to what he said which is eventually have to
5:35
become an engineer and you have to think about Britain from very
5:39
productive perspective and that is the key audience that they've
5:43
been building the platform for, which is essentially a first
5:46
platform. Yes, I have.
5:49
I love how he said. I mean, this was probably the
5:51
hottest take of the conversation which was Data scientists are an
5:56
artifact. It was like a blip in time and
5:59
he's not sure that they're going to be around much longer, or
6:03
they're going to be here for in 10. 20 years.
6:06
I thought that was. Whoo, it's going down.
6:08
A limb there. But yeah, the whole conversation
6:10
that we had around basically empowering the software
6:14
Engineers with machine learning and AI.
6:17
I thought that was awesome and really liked hearing his
6:20
viewpoint on it. I think the honor Thurgood could
6:23
pop for me or the Highlight is going the liver into
6:26
microservices and how he thinks about using microservices to
6:31
build products of the future, especially as we are working
6:35
with bigger models, it has become locavore office.
6:38
Prep at a fraternity were doing it before it was cool.
6:44
Yeah yeah he has been at it for a while they started in 2019
6:48
they've been plugging away and now the space that he's in just
6:51
absolutely blew up so it's good to see.
6:53
I'm glad that they're finding success and hope you all enjoy
6:58
the conversation. Of course.
7:00
The biggest thing for us that could help right now is if you
7:04
know, one person that would enjoy this episode and if You
7:08
can share it with that one person that would mean so much
7:11
to us, right now, I have one question for you to make shoes.
7:15
Did we get any reviews are coming?
7:18
So we did. I think we did.
7:20
Hold on. Oh, shoot.
7:22
I gotta know. Give me a sec.
7:23
I'm gonna go. Find them.
7:24
What do we got? Okay, oh, we got some awesome
7:29
reviews. This is cool.
7:30
So Wait, this one might be from you.
7:34
What is this from you? Did you write a review?
7:36
No. No you did it.
7:37
Alright, I just saw the name. It's from go Ali.
7:41
Alex. Okay, I can't pronounce that but
7:44
this is the most recent review. I'm a senior ml engineer who
7:47
deals a lot with in a lobster related items.
7:49
Since we don't have a dedicated real for that owner team.
7:52
This show and the slack community in parentheses have
7:56
been great resources for inspiration and staying up to
7:59
date on the constant evolution of tools and the best practices.
8:02
It's very useful to hear from other practitioners as we all
8:06
try and navigate this landscape together.
8:09
Whoever you are go Alec did you could have whatever I get
8:14
tongue-tied. Oh that is so cool.
8:17
I love seeing that is absolutely too kind and I appreciate it.
8:21
If you want to leave us a review that would be super cool.
8:24
It is very helpful to see this stuff and I love reading these.
8:32
That's the reviews and it would mean the world to us if you left
8:36
a review. Also share this with one person.
8:39
Leave a review. Show us some love and let's get
8:42
into it with to eat. heat. You've got an interesting
8:52
background nude, you were Investment Banking, and now
8:55
you're in machine, learning infrastructure.
8:58
Give us the breakdown. How the hell did that happen?
9:00
Yeah, I was an engineering College.
9:02
I went to work in banking, spend two years in banking.
9:07
Go bored out of my mind. You know, honestly, like was,
9:10
was a lot of good but just also just so boring.
9:16
The, the real positive investment banking is that they
9:20
just teach you to work very hard.
9:22
When you hate your life and you're not having a good time
9:26
and that translates really well to when you're having a great
9:29
time and you're working on interesting thing, if you like,
9:32
I could work wherever it's a really interesting Dynamic,
9:34
right? So, like, Every, the side, not
9:36
like every night at 6 p.m. your director would come to you and
9:42
say, I need this by 6 a.m. tomorrow at 9 a.m. tomorrow
9:46
morning and you're like, okay, cool.
9:48
And so you stay up all night, you stay up all night, get the
9:51
work done and they didn't need it at 9 a.m. the next morning,
9:53
but you have to do it anyway. And if you do that every day for
9:57
two years, it is the one of the most horrible things.
10:01
But well, it's just like work for the sake of work and so but
10:05
You still have to do the work. I'm assuming you're making good
10:08
money but you're just getting grinded down.
10:12
Yeah, this is one common complaint people to have about
10:16
Finance industry as well as about law.
10:19
Both to have that very similar culture which is like hey, make
10:23
this associate work crazy heart. Yeah.
10:26
Yeah, no, I think that's right. I their apprenticeship
10:29
businesses, right? So it's like, like the the
10:32
general consensus is that you learn by doing Tech time with
10:35
medicine. And as well, you know, we're
10:37
just like, we'll just work you to the ground and, you know, you
10:40
you appreciate you, appreciate it off to it and then if in-law
10:44
way is actually like maintains the hierarchy of these these
10:47
professions. Yeah, it's like you have to do
10:49
the work. His arm is pretty fascinating
10:51
but I did that for a bit and I was just like, I don't need to
10:55
do this anymore and so I actually moved to Boston to work
10:58
on some biomedical research using machine learning, just
11:02
like 2012 2011. Using machine learning to
11:06
predict the prognosis of neuromuscular disease.
11:12
So we're trying to figure out really, really early on in
11:15
someone's life if they were going to have, you know, one of
11:19
the diseases and for early intervention published a few
11:22
papers, your machine learning that was very different.
11:25
It was very much like the support Vector machine here are
11:29
like gradient boosted. Trees were just becoming, we're
11:32
just becoming popular, but really, Great time.
11:36
And then yeah just through a series of twists and turns just
11:39
kept getting deeper and deeper lost it or something right
11:42
there. Real fast.
11:44
Which like yeah, the sleeping under my desk and my boss was
11:47
asking me for these reports by 9 a.m. and then I just like went
11:51
and did this super Advanced shit with machine learning.
11:55
How did you know how to do this advanced stuff with machine
11:59
learning in 2010 2011? Yeah, so I studied the signal
12:03
processing in college. So I had like a like, a
12:06
probability background and if had I did a bit of information
12:10
Theory here and there. And so I think, like from like a
12:14
technical perspective, it wasn't, you know, different
12:17
probabilities different statistics as opposed to, you
12:21
know, something really, really new.
12:22
And so, I actually like, spent like the last three or four
12:26
months while I was at that job, just learning really, we don't
12:29
have statistics, I'm in a coffee shop.
12:32
On weekends. I was like, alright, sure,
12:34
sounds good. I'll go and work at a lab for a
12:38
bit. Yeah, I'm not.
12:39
A, what else am I doing selling my soul out sleeping under desks
12:45
classic red. That's so cool.
12:47
So then you got into ML and and you started getting really into
12:51
it. I'm assuming like we're happened
12:54
then. So I just started working with
12:56
machine learning in like, a very applied setting.
12:58
So, what worked on fraud detection content?
13:00
Moderation? The kind of, like, I'd say, 2010
13:05
early 2010's use cases of machine learning, you know, lead
13:08
scoring, you know, those types of self and I think eventually
13:14
it went to start another company that was acquired in 2018.
13:17
But like it was the prettiest pretty interesting because I
13:22
think in 20 2010 to 2020 like no one really had any expectations
13:29
of machine learning. And so like most people would
13:32
Treating it like a research function for the most part,
13:33
right? Even in companies, or just like,
13:36
you know, unless you're a Netflix and working on
13:37
recommendations or you know, who burned working on, you know, ETA
13:42
prediction and those types of things you're going to place it
13:46
that were pretty under invested. So they're like, oh, we have a
13:48
machine learning problem. We'll just hire will.
13:51
Hire a few people will hire hire a few people and then, let's
13:57
hope something comes out of it. And, you know, ultimately
13:59
nothing did. But for me.
14:02
At least, I really didn't want that to be the case.
14:04
So I ended up just becoming an engineer as well to be able to
14:07
support myself as a machine-learning engineer.
14:10
So I was like, all right, if I'm going to ml, you know, I need to
14:12
understand how to actually get this to value for someone as
14:15
opposed to training model in the jupyter notebook and showing a
14:20
confusion Matrix to someone like that wasn't the yet.
14:22
That wasn't the that's all at all for me.
14:26
And so it was very much like, how do you how do you build for?
14:28
How do you, how do you get this into products?
14:30
That was just kind of did. Which is the transition towards
14:34
ml engineering / envelopes with this at Camp Road, or it was a
14:38
tent, the next company shape. Yeah, guide, gumroad.
14:42
It was very much that we didn't have the resources to have,
14:45
like, engineering help on the ml side.
14:47
So the founder guy could start whole was very much just like,
14:54
you know, You can do whatever you want because I have to do it
14:57
by yourself. And so I just didn't, I just
15:02
figured I'd end up learning a bunch of service, I did
15:05
engineering, but a front-end engineering.
15:07
We hide another machine learning guy, who's a friend of mine and
15:10
one of my co-founders this business, he had a PhD in
15:13
mathematics and he was just like I could train, some models
15:16
should base, it should, you know, should have become an
15:19
engineer, so he became an engineer and so we kind of ended
15:22
up with I'd say at the time I differentiate.
15:25
The skill set in that we were, we are machine learning people,
15:29
but we also knew how to build products my journey was very
15:33
similar, which is worst. I work as a data scientist and
15:35
then it was like, Hey, nobody was there to do.
15:38
Engineering, learn everything by yourself.
15:40
Somebody had to one person on the team and eventually
15:44
everybody starts picking up. Yeah, and I think that's like,
15:48
what super interesting we can maybe talk about this.
15:50
Is this around, like, Maybe like the last six or seven year like
15:55
last 10 years about like, you know, the data scientist who
15:58
became engineers and now like I think the world the power is
16:02
completely shifted and now it's like hey, every engineer needs
16:05
to learn a bit of machine learning and I want that means
16:08
we're still trying to figure it out but we're it's flicked in
16:11
like it's crazy because I think this is flipped in like the last
16:14
six months. I haven't flipped it like it
16:16
flipped very, very quickly but it is a paradigm just very, very
16:21
different now You know, I think that's right.
16:25
Which is like you either go to a team where you have support or
16:28
you figure out how to support yourself, there's no data
16:31
science for the sake of design or you had a hot take their that
16:35
I think is awesome that I want to go back into which is that it
16:38
went from data scientist needing to learn engineering.
16:41
Now it's been flipped on its head and Engineers need to learn
16:44
machine learning, like, keep going with that one.
16:47
I want to pull on that thread about more Yeah.
16:50
Yeah, this is fascinating like this is, I'll be like, I've just
16:53
like been thinking more and more about especially with our own
16:55
customers, as we've seen this like, develop and see who's
16:58
having impact. Is that my Take right now, is
17:04
that data scientists? And I was one of these people
17:08
and, you know, like, I know a lot of your listeners might be
17:10
healed. He decide to do a data scientist
17:12
or a relic of the past. I think, you know, we, it was
17:15
just this, like, flashing time moment where, you know, we
17:19
didn't know how to actually get value from machine learning and
17:22
it was like this research e function.
17:24
And so, what we actually did was we like, I like and I think I'll
17:29
be said this To which is like, you know, we looked at the
17:34
people where we thought this damned out of them are like, oh
17:36
they must be data people, you know, like the data, people must
17:39
do the data science. And so you'd basically have
17:42
these people who traditionally Alice going, you know, read
17:47
something. If they're working in awe, you
17:50
know, a lot of the initial machine learning, libraries were
17:53
not, I guess, I could learn what they adapt ation of a library it
17:59
in in our and so, you know, they They'd go inside doing machine
18:03
learning turned out like, you know, analysis and machine
18:07
learning actually like very very different things and they just
18:10
happened to stem out of this and we built a whole industry for,
18:13
you know, from 2012 to 2020 around supporting data.
18:18
People who became data scientist to do machine learning and even
18:23
the skillset, even Asia, even the product that we built three
18:26
years ago was built under this premise, turned out that like So
18:32
much more, leverage the so much more leverage.
18:34
When you engine is use machine learning because they could
18:38
build their own products. They can actually think about
18:41
prioritizing it and then not reliant on getting other people
18:44
to build for them. So my take is actually what
18:48
we're going to see now is that these data people who became
18:51
machine learning people, I think that's actually going to shrink
18:53
back. They just going to go back to
18:55
being data people and machine learning is going to become just
18:59
like a massive, massive part of Every Engineers toolkit.
19:03
I think I want one thing that's super interesting, is that, so
19:06
what I want of our investors who eat Demetrius, who we met
19:10
through gridlock, they have a pretty like active recruiting
19:15
effort, especially on campus. And we do a little all caps
19:18
recruiting at Stanford. And one thing we get it, like
19:21
out of this is that we get like what like for every Edge
19:24
software engineer who's graduating because part of the
19:26
career fair what their primary focus is and you know we've been
19:29
building building this company for three and a half years now
19:32
So we have like three or four kind of data point here like
19:34
snapshots in time, every May what do the engineers and do and
19:38
what do they want to do? And I swear it's gone from like
19:41
the top priority, a top Focus being machine learning, probably
19:44
about 20 percent in 2020, 19 to like do it like last year.
19:50
It was like you know, 70 to 80%. I'm guessing this year's go
19:53
back. 95%, there's no doubt in my mind that there should go
19:56
knife at the end. I think like that is just like
19:58
the Ark of the, the Ark of the skill set which is like
20:01
Engineer's learning how to adapt these things that, you know,
20:04
maybe not even having to understand like how deep do I
20:07
need to go into this model? Being able to choose like a
20:10
black box, like taking taking something and understanding how
20:13
to appropriate it as opposed to starting from scrub.
20:17
Did you settle out there? And that's I guess, the next
20:21
when is How deep do the software Engineers need to go?
20:26
Because that is something that I think people ask a lot in the
20:30
emeralds community and one thing that I think about is how we
20:35
just had this survey, right? And it was all about people
20:39
using large language models and how they're using them and what
20:42
tools they're using around it and you do see that the barrier
20:46
to entry has been lowered. Its not been lowered it's just
20:50
been absolutely different. Yeah.
20:53
There is no very now, if you even can write in English,
20:58
you're good. Like you can figure this out.
21:00
So one thing know is that how like how much value you can get
21:07
if you know, how to go into tensorflow and really turn some
21:12
knobs and host your moon-like open source model and maybe
21:18
bring it down, maybe you don't need this large language model,
21:21
you can just use a smaller model and training.
21:23
It on your data, your company data and so maybe we go into
21:27
that a little bit on how there is a bit of a juxtaposition.
21:30
It's like there is this gateway drug which now is like GPT and
21:35
you can go into machine learning and you can get a feel for what
21:40
a I can get you. And then if you want to go
21:44
deeper down the rabbit hole, you can start to figure out.
21:48
All right. Well, how do I play around with
21:49
these open source models? Yeah.
21:51
II can start with like an anecdote.
21:53
It's like Ellen Dimitri. What we talked about this guy,
21:57
this project. Last time we talked was like
21:59
this, this refusion project at the end of 2020 22.
22:04
Like I took off which was, you know, it was based off stable,
22:07
diffusion and it's fine tuned to create spectrograms.
22:11
That's fine. Tune our grades back to rounds.
22:13
And, and so, basically, what you could do is you could still do
22:16
it right now, you go reviews, on.com, you can type in like,
22:19
you know, Taylor Swift dancing to ball, like making Indian.
22:23
Music and I will spit out sloppy that sounds like, you know,
22:27
someone that's Taylor Swift and I, you know, you got to use your
22:29
imagination a bit. The I'd say that that like, to
22:34
me, that's like a really good example.
22:36
I got the the engineer who all the engineers who built that,
22:41
you know, they didn't have a ton of machine learning experience
22:43
or what like, they have a lot of Music Experience.
22:45
They have a lot of domain experience for our music and
22:47
they have a lot of intuition around Music Creation.
22:51
But they were able to more or less use the machine learning
22:55
part of it as a blend. The end like in mind you, they
22:58
weren't just using the input output stuff like they were,
23:00
they were still fine-tuning it. Like there were going and you
23:03
know, they had a weighted weights and biases are but I
23:06
don't know if they had like a deep conceptual understanding of
23:11
what was happening under the holiday.
23:14
And you know, I don't mean this as a disc to them because you
23:16
know I think very highly of those guys but really like
23:21
they're able To create value by. They give it the product
23:24
problem, not as much as like, the research problem.
23:26
And so, I think we're like, thankfully, we're gone in from
23:32
this explore to this exploit state in the last year.
23:36
And in the last year, where we've gone from like, hey, like
23:39
the research section with the Explorer part and the research
23:42
stuff is still happening, but we have got the abstraction than
23:46
the understanding around these models.
23:48
Now that a lot of folks can exploit it without understanding
23:51
Standing, you know how how like you know what's going on on the
23:56
end, you know on the 27th layer as you know like what do I need
24:00
to change that? And I think that's fantastic.
24:03
I think like they're still going to be room for that.
24:08
Explore that explores state. But I think you could just go
24:11
really, really far, you can just go really, really far without
24:15
having to go that deep and we, you know, we see this with our
24:20
customers to just like The amount of Engineers who are
24:23
taking like with fellow who are taking stable diffusional taking
24:25
control net, or taking a llama and and be like, what can I do
24:31
with? This is just, it's astonishing
24:34
it's unlike anything I've seen before which is like, we just
24:36
kind of like we we built the abstraction high enough.
24:40
All the value was was was high enough that we could just try
24:44
to. We could finally treat these
24:45
things as black boxes. If it babies I love, we heard
24:49
Cambrian explosion. Is what's happened?
24:52
That is the best way of explaining or describing, what
24:56
is going on right now? And when you talk about the
24:59
software Engineers that are now being able to exploit ml or AI,
25:05
one thing that comes to my mind is how Everybody's kind of on
25:10
the same playing field and so you don't really have any
25:13
defensibility. It feels like you know I look at
25:17
those photo. Yeah.
25:19
There was like as soon as one person figures out that there is
25:23
demand for a certain type thing. Like the profile picture
25:27
generated by stable diffusion. Then there's copycats that pop
25:31
up and so I hear how you think about that and again should we
25:38
really dive deep under the Bird and figure out what is going on
25:42
to try and give us a leg up on the rest of the world.
25:47
Yeah. I think, I think, I think that
25:52
was still exist, right? Like, I think that's like it
25:54
just like the the long tail of use cases will be this.
25:58
Kind of generic commoditized, us has Eric Schmidt matej, zaharie
26:04
out from dinner. Breaks Eric Schmidt, from
26:06
Google, obviously, and my truck heading Metrorail go.
26:12
They recently had like a blog post on this about, you know,
26:16
how much do you think that this this GPT kind of Gateway?
26:19
Drug large, language model, off-the-shelf models of his
26:21
provider, will be able to solve as opposed to like, how much
26:24
will you have to, you know, fine-tune how much would you
26:27
have to go deep into? And I think, like, the truth is,
26:29
is that for the low value. I personally think I think this
26:32
is their point as well as that like, for the low value, your
26:36
long tail of use cases. Like the, the commoditized, the
26:40
commoditized, like GPU TPT for API, call wall become huge.
26:47
But, you know, to to turn that and make This hate this.
26:50
It works really good for medical for this Medical Imaging.
26:53
Hey, with this look like what's really good for the lawyers?
26:57
Hey, this really looks works really well for like no design
27:05
is. I know like foot video generate,
27:07
I think you're gonna have to go deeper and either come up with
27:10
your own models. And you know just all just like
27:13
tune them or appropriate, those molds for your context and I
27:18
think, you know, And like, I think there is a case for like
27:22
smaller, more potent. Targeted models there.
27:25
As opposed to this general purpose, I do everything.
27:29
I do everything model. I'm kind of like against Pan
27:33
Asian restaurants because, like, you know, it's like I did, they
27:35
can make Thai food and Japanese Food equally good.
27:39
I think when you do want to get the best, when you do want to
27:43
get the best Thai food, you'll still go to a Thai restaurant.
27:46
You want to go to that restaurant.
27:47
That makes Thai food Japanese food. and, you know, Chinese
27:51
that and there is something I think that is It's been, you've
27:56
opened up a door in my mind. It's like there's so many
27:59
different use cases out there that even if we try, the
28:04
majority of people that are hacking around on stuff, are not
28:07
going to find those little use cases and that is where the long
28:11
tail is. And for those use cases for the
28:16
most part, A GPT is going to be good enough because there's not
28:20
so many people searching around in that part of the universe.
28:25
Yeah, but then when you go to the use cases that have these
28:28
huge unlocks like you were talking about like doing stuff
28:31
for for lawyers, or for medical imaging, etcetera, Etc.
28:35
That's where you need to really add a lot more value than what
28:40
you can just get off the shelf. Yeah, II think.
28:42
So I think when you need concentrated, concentrated value
28:46
and you care about speed cost and performance which is like
28:49
the whole the holy Trifecta. You going to have to go smaller
28:53
more targeted. Yeah I mean it's the Idea that
28:56
everybody's talking about right now, which is how will taint
29:00
with the generalized models be in the long run, while they
29:04
would be great at, you know, giving you some sort of
29:07
abstraction initially and solving problems at that level
29:11
past that we have to think of different things, which is RV,
29:15
using honor the model, and using it in like an actor critic
29:19
setting, where we are able to see first is to be able to set
29:23
the guardrails. For the results generated from
29:27
these bottles and of the second is, the question around, who
29:31
feigned to do prompting versus going to do fine tuning?
29:34
And where is the general kind of practice around knowledge,
29:37
distillation? Yeah, I think that's right.
29:40
That that team right to me. Which is that, you know, you
29:43
gonna the like the first the first question you're going to
29:49
almost make is like, is there an objective truth here?
29:52
And what will I be able to you know, this is like I I think
29:58
Dennis from ma'am. May I know if you guys have seen
30:01
me, may I had to go to Coal Company.
30:03
I'm they and you should have Dennis on the show cause I think
30:07
he thinks about this in terms of product like really, really
30:12
unique. Yeah, that's awesome.
30:13
But one thing he thinks it's just like, you know, if there is
30:15
a task to be done, like if there is a specialized tasks to be
30:20
done, the General, South will break down for a number of
30:24
reasons and Then you almost have the secondary question as I.
30:27
What do I need to do to solve this?
30:30
It might be better prompting. It might be fine-tuning might be
30:33
completely different model, maybe it's maybe like you need
30:37
this response within 100 milliseconds and you need and
30:41
you don't want it to be very, very expensive because it's
30:42
happening your, every three or four seconds and like you got to
30:46
have to get a smaller more concentrated model.
30:49
But I think that at least to me intuitively that that seems
30:55
right. I think like you know like we
30:57
can I can just say it because I think it's true right on some of
31:00
the reasons. Like, hopefully I'm also really
31:03
good right now. Like scary, good like scary
31:05
better, you know, then even like the more targeted stuff right
31:09
now and I think like did a bit of catch-up that needs to happen
31:11
right now? For the things that we are
31:15
talking about to be true, I think you know what I like but
31:20
intuitively it does make sense to me.
31:23
I think I agree with you which is these are all the limitations
31:27
of lme's and you know because again you know you don't have
31:32
their backtracking accessibility that we did have in the
31:35
conventional models but let's move a little bit towards the
31:39
compute which has the bigger elephant in the room.
31:42
Any model that you deploying. Now that yeah the fact that
31:46
we've already proved when in a very that bigger models might be
31:49
a little bit better and not like Ian parameters, but just just
31:56
tiny bit bigger than we are using today.
31:58
Could have their performances so I want to.
32:01
Yeah, you had very interesting plug post, which is how to
32:05
choose the right horizontal scaling setup for high-traffic
32:09
bottles. And why I want you to talk a
32:12
little bit more about in terms of the context, because as you
32:16
scale board, the things increase, which is the computer
32:19
as well as the storage. Yeah.
32:21
Totally I didn't write that blog post, I think was written by
32:25
someone else on our team but I think like what you're saying is
32:33
like a center-right which is like the These models get as his
32:37
muscles, get bigger they and yeah, this is, this is a great
32:41
plug for our company because the basis for our for our company,
32:43
which is that, you know, a whole host of infrastructure problems
32:49
show up. And so, you know, whether that
32:51
be, you know, right now we're trying to deploy a, you know, I
32:54
think it's like a 60 billion parameter 60 billion parameter
32:58
model with like floating Point 32 f p 32 or something like that
33:03
which is, you know, huge and I think like the The challenges
33:06
from like a scaling perspective. From a scale, 20 perspective
33:10
from a cost perspective, from a latency perspective, from a
33:15
scale up scale perspective like this.
33:18
All these you know there's all these massive barriers that that
33:26
you know stop you from using these models in some setting to
33:29
build really, really great products.
33:32
And I don't and I think you're right, I be like this stuff is
33:36
Solved. Like now that now we're getting
33:37
into territory where you know, like when Lama came out like
33:41
four weeks ago, whatever, like we were the first people to be
33:43
able to serve that, it fish like you know really really fast.
33:46
We created this chat Lama thing coach at Longwood.com.
33:50
You know, you could you could play around with it and you know
33:52
you got to top pills at the top of how can use you know, for
33:54
most of the day and what you realize is that, you know, these
33:58
models now that are bigger and like I think at least the
34:01
current understanding is that the the bigger models are
34:05
forming better than the smaller more specialized models, you're
34:11
talking about deploying things that or serving things that are
34:15
in a way that would be hard to serve in the past but now and
34:20
like because the lat the largest use case is almost like a
34:23
consumer use case right now is that like a scale that we
34:26
haven't thought about and I think you know again a lot of
34:30
these things I'd say from my Mo else perspective and from an
34:33
infrastructure perspective like they'd be on my pay grade and I
34:36
Clint tell you how we're doing it.
34:38
But you know, we all, you know, there are lots of interesting
34:41
things. They were doing from Micah
34:42
scale, 20 perspective from like a autoscale Pacific from a cold
34:45
start perspective like how like when my, if I have this massive
34:50
model and I only needed two hours a day, do I need to have
34:53
it running all day long and when it scales down, how long will it
34:57
take for it to come back up what's acceptable?
35:00
And I think these are all the unsolved challenges and to
35:02
getting these gaiety and and like this is I think What
35:06
opening? I just chose good has done so
35:07
well like you know like is that you know know they made their
35:11
models usable like they didn't just create the best model they
35:15
made it usable they gave you fine-tuning infrastructure.
35:18
They gave you serving infrastructure.
35:19
They gave you great docks, you don't have to think about scale
35:23
up and scale down, you know, I don't think the rest of us and
35:27
also included and, you know, a bunch of other products like we
35:31
have. We haven't created that same
35:32
moment of Magic For Engineers, just yet to be able to use these
35:36
All those very effectively and efficiently on at the one of the
35:38
bigger challenges that has been persistent and all our
35:42
conversation would different level Engineers as well as being
35:45
a scientist is documentation, how frequently do you document
35:49
your code and how do you document your models as well?
35:54
And yes there are options which is Wars and controlling and
35:57
model, container version isolations.
36:00
So those are the options out there but you have something
36:03
interesting which is darks as code.
36:05
Do you want to talk? A little bit more about that as
36:07
well and tell us what it is about.
36:09
Yeah. Yeah, it's a totally, which is
36:12
like, what? One thing we've thought a lot
36:13
about is, you know, how do we maintain a product that has like
36:19
active living documents that all Engineers can contribute to?
36:24
And so we, you know, we thought we thought a lot about, you
36:27
know, being able to create be, you know, creating some
36:31
scaffolding. So that Engineers could write
36:34
one like doctors code. So it's hosted in a general
36:38
purpose, doc Annotation system that you're contributing as
36:42
code, but also integrated with the development process, and I
36:46
think like, you know, Phillip and I both assign both of those
36:49
posts. I think you're talking about,
36:51
were written by word and by Philip, he's very, very
36:53
passionate about this, which is like, how do you build docks
36:56
into the life cycle of engineering?
36:59
I think this is even more true from like a formica machine
37:04
learning perspective, right? Because these artifacts, they
37:06
kind of disappear. They kind of these moments of
37:08
time and who knows? Knows what my who knows?
37:11
What my what my model was four weeks ago on.
37:17
You know, my third laptop at work and like how and we haven't
37:23
solved this yet. I think anyone's done a good job
37:25
here. But like that documentation
37:27
perspective, in that version managers perspective, is
37:30
definitely a massive open problem and machine learning.
37:32
I'm still in like the stop your reference to Gabby is very much
37:36
from like just our engineering work flow perspective as opposed
37:40
stir within our product and I think like that's something we
37:43
hear a lot from customers, which is You know, I had a machine
37:47
learning researcher or engineer, they came and train the model
37:52
they disappeared. You know, what does that version
37:55
do, which version matters? You know?
37:57
How do I find the other? I'm sure you guys talked about,
38:00
you know, you guys hear about so much.
38:02
Yeah. From the coma Community as well,
38:04
which is like you know and like this stuff like weights and
38:08
biases which is making it easy from a training perspective.
38:11
But still like I don't think most customers of ours and you
38:14
know, although we talk to and most folks are you talk to would
38:18
be able to tell you like which artifact is running in
38:20
production II bet. I think it seemed that the thing
38:25
which I found interesting looking at based a, I feel like
38:28
you guys are an advocate for micro service based
38:32
architecture. And I don't know if that's
38:35
across the stack or just across the models itself because across
38:39
the models, I feel like you're ahead of you type reason being
38:43
because now with lme's, now everything is you have product
38:47
buildup on top of apis? Yeah, I think with everything, I
38:51
think we think that like, you know, so Based on as coal is
38:55
this way for, you know, it's like the way I like to think of
38:58
it like a toolkit for creating value from your models and like
39:04
you know like I think like the a good analogy there.
39:07
Something. Like the cell which will sell
39:09
took your like front end package and they said, hey we're gonna
39:12
build a bunch of utilities and extend it from that to the real
39:16
world. I think for us like it goes like
39:19
all the model stuff definitely has to be api-based and, you
39:23
know, Like it's everything that severe microservice.
39:25
Like we not only does that model that model itself and every
39:33
single version have to have its own API but the logging for it
39:37
has to have an API so you can access the logs for I think, you
39:40
know, the place where this gets super interesting is that we
39:43
also have the serverless functions that you can write on
39:45
top of your model. So think about your traditional,
39:49
like I'll take you back to like, one use cases like one user had
39:52
this massive Language model gets hit a lot, the tries to
39:55
Performance optimization, so they need to build a caching
39:58
layer on top of it very, very typical product thing where you
40:02
going to Build That Wall. You can go and build that in
40:03
your own mono repo somewhere but based on gives you a survival as
40:06
function and access to a database.
40:09
So you can build that caching layer into base 10 but those
40:12
Services functions that do the caching and call the model they
40:15
have their own apis as well. And they by separating these
40:19
concerns we can scale up differently so I may be your Or
40:23
your model actually needs a lot less tracker to support a lot
40:26
less traffic than your overall service, maybe vice versa.
40:30
But it gives the the the user, a lot more control, over how these
40:35
this model will be used in the application.
40:37
And that's just one of the things I was concerned up.
40:39
Our been be say, hey, we're going to advocate for
40:42
microservices or AP solution across the entire stack.
40:47
Is how do they scale? Because Microsoft versus again,
40:50
to have their own limitations, if you have too many Apis
40:54
versus, you know, testing and production becomes a little bit
40:57
more complex. Second thing, is it just
41:00
increases operational complexity?
41:02
Yeah. Managing dear consistency is
41:05
another issue. How do you work past those
41:07
challenges? Yeah, I mean, I think the
41:10
pre-charge but like I think he's gonna treat it just like you
41:14
would with a it's like a traditional API, right?
41:16
Which is that, you know, it's, you had you have your CI CD
41:19
system. You have, you know what one in
41:22
the way you learn do, Now that we are working towards is to
41:25
have a much more. Get based workflow is like, you
41:28
know, you don't ever think when you have a, when your web app is
41:33
running the user never has to think about, hey, what version
41:36
is running, right? It just want you just one URL.
41:39
And then behind the scenes, they give you access to what, you
41:42
know, what is what that is pointing to.
41:45
And that's what we're trying to get to as well, which is a you
41:47
have burdens on your model but there's only one primary version
41:51
and that is the thing that gets used.
41:53
By everything else. If you want to call another
41:55
version that still works but you need to keep track of what is
41:58
primary and what is the main version?
42:00
I think the same thing with what we call work led to several as
42:05
functions is just like there is a production version and there's
42:08
like everything else and there by maintaining this bifurcation.
42:12
This yeah, this separation between what is being served
42:17
live traffic by your app and what is not.
42:20
I think you can get around those things but I think This is also
42:23
one of the benefits of having Engineers as the primary
42:27
archetype now, right? Because the, these are the
42:28
workflows that they used to, they used to, you know, see ICD.
42:33
They used to maintaining the production version.
42:36
Everything will be. I'm they understand.
42:37
It's like very traditional good Pace workflows staging
42:41
production, you know, Master whatnot.
42:43
So, yeah. Like I think I think it is.
42:46
I think it's reducing it more into an engineering problem.
42:49
Then a data science, problem is like, what I would say to that.
42:53
Which Is the complexity, dude. I mean, just think about how
42:55
many times I just know because over the last three years, it's
43:02
been like so many engineers and even like X data scientists, or
43:06
some people even brand themselves as recovering data
43:09
scientists, they talk about how painful it is to get some of
43:13
their teammates, just to learn get.
43:16
And hopefully, that is going to be the problem anymore.
43:19
Hopefully, we're not going to have to figure these things out.
43:23
Out or push data scientists to do those things that they don't
43:26
want to do because they're just going to get eaten by the
43:29
software engineering world. Yeah, I have I I admit my maybe
43:33
not. I think it's a it's an evolution
43:37
that simplifies things a lot more in that it's, you know, and
43:41
like I think like you know, Abby you mentioned this earlier as
43:44
well, which is that you will you and I said she was like, you
43:49
know, we found ourselves to be more effective when we became
43:52
engineers and Had to learn that. It's like I don't think it's a
43:55
bad thing that you know, you have really clean worth.
43:58
But around this I think by that by the Persona being kind of
44:02
like squashed a bit into, hey, you are a, you are also an
44:06
engineer and you machine learning is one of the things,
44:08
you know. I think that is like her.
44:11
It does make life a bit earlier holistically, at least.
44:16
All right, man, I want to switch gears and I have to ask because
44:20
I love the fact that you have some incredible people that have
44:24
backed base 10, how did that happen?
44:28
I mean, did they just see in you through this shining star and
44:32
they wanted to go deeper or what?
44:35
Like are you incredibly well-connected or are you?
44:39
Great salesman or a little bit of everything.
44:41
Not so I think like at the end of the day, like we're just been
44:46
around for a while until we have meaningful connections and like,
44:50
you know, some of the people were buried money for, you know,
44:53
we've known for, you know, RZ close to a decade like Sarah 0,
44:57
grow from Grail of now, conviction, who has run a number
45:02
of checks into us. Like we started, we met her when
45:05
we were at our first company in 2015 and is she Backed Us in
45:10
2019. And so I was just like a lot of
45:12
relationship building. I think the like overall like
45:16
for us like what we got lucky with was that we were pretty
45:19
technical team that would that could build products.
45:22
You know, we We will working in a market that had a lot of
45:28
oxygen. We're working in and knock the
45:31
hell out of oxygen. And yeah, of course, like we
45:34
just knew we just been around long enough that we knew a lot
45:39
of great people who showed early conviction and I don't think
45:43
there's like, you know, anything particularly special about us
45:47
like this. This stuff is very, very
45:49
difficult. This is very, very difficult and
45:51
he's got to get lucky at, you know, a number of different, you
45:56
know, Junction Junctions. And so I I love the humility
46:00
where he says, I've got no secret sauce.
46:03
It's just timing is just perfect timing because I had nothing
46:08
else. Exactly?
46:09
That's the right place, right time.
46:13
You're out. Let the right coffee shop at the
46:14
right time, you know, ordering the right thing.
46:18
Well, and you're working exactly.
46:19
Working on a hard problem. And the problem got very hot.
46:23
Look on how proud the problem. No problem popular.
46:25
That anything like that was like a, you know, you guys, you guys
46:29
like me. But that's like three years ago,
46:31
two years ago, is that, you know, those so many machine
46:35
learning Ops companies. And so many of them and still
46:39
are like, we're just so Enterprise focused to her.
46:42
We're just like, Hey we're going to sell their and prize.
46:43
We've got top down, way to sell it to the CIO.
46:47
I'd say that. You know, those like a small set
46:49
of companies that were focused on the end user and like, you
46:53
know, building for the engineer of the data scientist and, you
46:56
know, obviously and like, I think those Those are to me, at
47:00
least they're they're interested companies.
47:02
And you know, we're like all these companies, not even don't
47:07
grab is 10 right now, fantastic. Really hugging bait is a great
47:09
company. Is building for the end-user
47:11
replicate and all those guys would like been like awesome
47:18
people. Working on really interesting
47:21
problems. I with a differentiated point of
47:23
view, but I think for us like, you know, we were, we were
47:26
thinking about machine learning from our product perspective and
47:29
we have been Continue to be, you know, we haven't really made.
47:32
We still have a long way to go. Obviously we're just getting
47:34
started, but the like it is it is somewhat.
47:40
It was somewhat different at the time.
47:43
And you know, you see like now companies like that definitely
47:47
getting fun to write old making a lot of Splash, like, you know,
47:50
Liang Chen, just doing so great right now, it's fantasizing.
47:52
See. But again, it's focused on the
47:54
engineer. Its focus on the its focus on
47:57
using ml is an API almost. Yeah.
47:59
Yeah, yeah. Well tell us about I mean, I
48:01
know you guys are hiring and who you looking for you, looking for
48:05
machine, learning Engineers. Yeah, exactly.
48:06
So we're looking for, you know, folks who could do ml, we're
48:09
looking for. Honestly, like machine longing
48:13
for is, you know, this is why your podcast is so great, in
48:17
your audience, is so great in your community.
48:18
So fantastic is that like, there's just not that many MLM
48:21
for people out there who are really good.
48:23
And I think like that this is like such a massive opportunity.
48:25
That's why it's so hard for us to hire him.
48:26
So any mrm for people like, you know, No, we will value.
48:31
You have a good time. You're working hard problems.
48:34
We'd love to chat, you know, but I think in general, you know, we
48:39
take a pretty What's arrival? I we look for more utility
48:45
players. Then, specialized base 10 and
48:49
like, you know, if you like to build stuff, you liked it like,
48:51
you know, reach out to me or anyone else on the team.
48:54
We'd love to chat with pretty open-minded about who we really
48:57
are. And will leave a link to the job
49:00
description, and the hiring page and the description of this of
49:05
this podcast in case anyone wants to check it out.
49:07
One quick thing, I'll ask as a follow-up on that, are you
49:10
hiring like in a particular location?
49:11
Ation as well. Our is it across States?
49:16
I did it anyway. You know, it's just a it's, you
49:18
know, we have a team and queer people in Canada.
49:20
We have people in Armenia, you know we had people in the US
49:24
like we're pretty open one location.
49:26
So nice building remote teams is hard but we're committed to it.
49:30
Yeah. Well dude I got one last
49:33
question for you and hopefully this doesn't destroy our
49:35
relationship but first time we talked you told me that what you
49:40
were born. You were born In India.
49:42
Are you born in Australia? I can't remember exactly.
49:45
But I thought on this straight. Okay.
49:46
Yeah. So then that destroys the
49:48
question that I was going to ask.
49:49
I was going to ask if you were to well you should ask it.
49:52
I'll ask it. Anyway, no, I was gonna ask you
49:54
if you were the guy that that movie was based on Lion.
49:59
That's why I love. Oh, he was bullied.
50:04
Yeah, no II. Wish that.
50:05
That, that's a good start. Now do I wish?
50:07
I don't think I wish but I was a good move the Great.
50:12
I was just thinking while I was talking you, I'm like, I yeah,
50:15
the Aussie accent is coming through and so I had to ask
50:18
anyway, this has been awesome, man, I appreciate you coming on
50:20
here. I appreciate you.
50:22
Chopping it up with us and teaching us.
50:23
A few things is always a pleasure chatting with you and
50:27
look forward to doing it more. Awesome, thank you so much for
50:30
this. I am happy and to Beatrice.