Latest / Agentic Conversations (formally mlops.community) / Adventures in Building CLIP & Other (Largeish) LMs // Sachin Abeywardana // #180

Transcript

Adventures in Building CLIP & Other (Largeish) LMs // Sachin Abeywardana // #180 · Agentic Conversations (formally mlops.community)

0:00
Hello everyone my name is Sachin Abeywardana.
0:03
I am a senior machine learning engineer at Canva.
0:07
Fun fact about me, I don't drink coffee, but I do lots and lots
0:11
of saucer. And I don't mean the dip, I mean
0:13
the dancing saucer. Thank you.
0:18
Welcome back to the mill OBS Community Podcast.
0:20
I am your host Dimitrios and today I'm flying solo.
0:23
But I was just talking to Sachin.
0:26
Wow, I have so many notes from this conversation.
0:29
There was so many different threads that we pulled on and we
0:33
talked about his journey, but we also talked about the trade-offs
0:37
that you get as a machine learning engineer and having to
0:40
think about the broader product and.
0:44
The fun stuff that these machine learning engineers like Sachin
0:48
enjoy doing, which is that? Deep down, hard math problem.
0:54
So we get into that first thing. I knew it was gonna be a great
0:58
conversation because he started it with yeah, I did my PhD and
1:02
it was a complete waste of time. Bam.
1:05
That's how we set the tone for this one.
1:07
I loved that, but he was very open and I loved.
1:12
Another thing in this conversation, which was how
1:15
forthright he was with the downfalls of not understanding
1:21
the broader product and bringing machine learning capabilities or
1:26
AI capabilities into the product.
1:30
So we went down a bit of a rabbit hole on how he has been
1:34
remediating that and why he feels like it is important, and
1:40
we finished it off. One thing that he mentioned?
1:42
A little bit of a spoiler alert. But one of the last things he
1:45
said was. It is so important for machine
1:49
learning engineers to not only understand the metrics of their
1:53
machine learning projects, but also to understand the metrics
1:57
that the product team is looking at because that will help guide
2:01
you as you are creating your different AI ML initiatives
2:06
within the company. All right.
2:09
I hope you enjoy this conversation with Sachin.
2:12
And even if you don't, you know what would mean the world to me
2:15
is if you share it with one friend and like or review or
2:20
give us some stars wherever you are listening to this episode.
2:30
sometimes that's because I have too much shit open.
2:32
So I'm going to close like every other app.
2:36
Yeah, I'm going to like, I don't need Spotify open right now as
2:40
much as I love it. Or signal or.
2:44
Like 20, Mozilla, Brave and Safari.
2:48
I just need Chrome right now. Have you got chachi PT
2:52
installed? I don't.
2:55
Do you not I mean I just paid for the premium one today so
3:00
little church today. So that's that's that's been
3:03
amazing. Yeah.
3:05
Yeah it's yeah. But I mean like if if you're if
3:07
you're a programmer, you you must have it so that the
3:10
copilot. Yeah, yeah, yeah, totally.
3:13
It has become something where at least like to stay up.
3:19
To date and to be in with the times, you have to be using it.
3:23
It's one of those like. Early adopter things, I think,
3:27
yeah. Yeah.
3:29
I mean like it's a I'll probably describe myself as a power chat
3:33
GTA user because I'm really bad at writing shell scripts, right?
3:38
So I've just gotten decent at writing Python.
3:41
Michelle scripts is it's a completely new world.
3:43
So what I do is go check do D and say write me a code to
3:48
download MP3 files. It will do that for me.
3:50
So yeah, so the small things like that, it's really good at
3:56
and I'm sure it'll be, it'll be much better things as we go
3:59
along too. Have you?
4:03
Have you messed around with some of the?
4:06
Tools that help you validate the output and make sure that it's
4:11
real scripts? Or are you just plugging them in
4:14
and then once they work, you're good?
4:16
Or I am hoping for the best. And look, I do test it, I do
4:20
test it myself, right? But in production, these are
4:23
production code that I'm writing.
4:24
I mean, because I'm all engineer.
4:26
That is the quality of the code. I am very bad at being test so
4:32
I'll admit. But saying that, you can get
4:35
Chachi 50 to write the test as well, so.
4:38
Yeah, it's yeah. So you can get around that.
4:42
That's a easy thing to get around.
4:44
Actually, one of the hackathons that we had recently was in
4:48
Berlin and one of the submissions.
4:52
I voted for it to be like one of the winners, but the actual demo
4:57
broke. So the other judges, it wasn't
5:00
me, it was the other judges. I'm gonna just throw them under
5:02
the bus right now. They were like ohh it was good,
5:05
but the demo didn't work and the submission was that you have
5:11
code generated and then if there's errors in the code you
5:16
have agents that go through and they grab the errors and then
5:19
they paste it back into ChatGPT. Again, they figure out what the
5:24
error is and then the agents go and they update the code.
5:27
So yes, it's the full cycle where you can be a little bit
5:33
more. I don't know if confidence the
5:35
right word, but just lazy. Yes.
5:38
Yeah. I mean, I mean that is valuable
5:40
because right now what I do is if there is an error, I manually
5:44
go back and paste it and then it goes, oh sorry, you're right.
5:48
And I'll go and correct it and then place it back and then just
5:51
keep going back up. So if that process is automated,
5:54
yeah, sure, I can see that. I can see the value in there on
5:56
that on its own. Yeah, I think the hard part
5:59
right now is just dealing with agents that are reliable.
6:03
That's where I think in all of the hackathons that.
6:07
I've seen they kind of fall over at a certain point and so I know
6:11
there's a lot of hard work being done by a lot of people out
6:14
there to make agents more reliable and that's active.
6:19
Cutting edge piece of this machine learning and AI
6:22
industry, but yeah, so I mean I only have a very basic
6:27
understanding of agents. So what I understand of agents
6:30
is that they it's prompts if I understand correctly, but very
6:36
detailed, very specific points so that you basically define the
6:40
boundaries of what open the eye or whatever it is can do.
6:45
And I'm not sure that kind of agent is going to last for too
6:51
long because the my hypothesis, again, my hypothesis it's an
6:59
uninformed hypothesis is disclaimer.
7:03
Yes, that's right. Yeah.
7:06
Is that it's already good at understanding language.
7:09
I mean compared to five months ago, it's really good.
7:13
So I mean, when I tell I'm an agent, I know exactly what I
7:19
want. I'm assuming that exactly what I
7:22
want is only going to get better, more training, more
7:27
data. So like even right now ChatGPT
7:31
has this thumbs up and thumbs down thing.
7:33
So I'm assuming that they if if I give a thumbs up it will, it
7:37
will keep that as a training data, go back and feed it feed
7:40
it back in, especially if it gets it wrong, right?
7:42
So then especially if it gets the wrong ones are the most
7:45
important bits right? So that's what you need to
7:48
feedback and improve it. So yeah, if it's a prompt
7:53
engineered agent, I can see it's getting better.
7:57
But if it's an agent that actually I don't know if this is
7:59
a web or access a pipe and terminal, those things, yeah,
8:02
those things are, those things are amazing.
8:05
Yeah. So because that just extends,
8:07
extends the capabilities, right. So.
8:10
Yeah, exactly. So alright dude, let's talk a
8:14
little bit about what you're doing and what you're up to.
8:16
I've got your whole history here and I mean you're you're so
8:22
humble. You were like, I don't know,
8:24
because all the other people that you've had on the podcast
8:28
are big names and big hitters. But you're in your own right.
8:34
Like you're also quite a big hitter.
8:37
What's your story? Give us the breakdown.
8:39
I know you got a PhD. You went to what that was?
8:43
Yeah. Let me, let me start with the
8:44
heart. So I I I did my pH.
8:47
D in Bayesian machine learning. And if, if I'm being honest,
8:51
that's the biggest waste of my time that I've ever had.
8:56
So. No way.
8:57
Why do you say that? What's So I spent I spent 4 1/2
9:00
years doing a PhD right in November.
9:02
It was one of the earliest times look like, so I do not recommend
9:07
doing that. Unless, unless of course you're
9:08
rich and you don't have anything better to do anyway.
9:12
So the reason that I personally regret is like when I was doing
9:16
my one, mine was very theoretical.
9:19
I was doing in an area called Gaussian processes and at the
9:24
time, the best that we could fit was like 1000 thousand data
9:28
points, right? And that was the best that we
9:30
could do. This was like 2010 between 2010
9:35
and 2015. So when neural networks are just
9:37
starting to take off, right? So I I kind of missed that
9:40
neural network bus as opposed during my PhD at least.
9:45
Um, and yeah, I think, I think we got to a point where it was a
9:49
million points, which is amazing for for a Bayesian for a
9:52
Bayesian thing. So anyway, once I finish with
9:56
PhD, I did this this course on basically it was just an MNIST
10:01
data set height. So it's 60,000 data points.
10:04
I have system amazingly fast. Just yeah, so we just did a few
10:08
layers and like we've got this new 100% accuracy and that was
10:13
like whoa, OK, I'm not going back to Bayesian Hurricane.
10:16
So but look it's it's not, it's Bayesian sites have come back
10:23
like all the stable diffusion models that you see have some
10:26
Bayesian foundation in them, right.
10:27
So it is grounded in the Bayesian side.
10:30
But the point is, in terms of speed, in terms of what you can
10:34
do with the model, the the neural networks capability is
10:38
just so much bigger than what you would do as a from just
10:43
using Bayesian tools O after that.
10:46
So after that I got, I got lucky enough to start at a small
10:50
startup that had just been bought out.
10:53
I started there as a data scientist, mostly just analysing
10:55
data. I didn't actually know any
10:57
Python at that point at all. So I think I got lucky lucky
11:02
getting hired to mostly software company.
11:05
So so yeah, so that's where I started doing like Python
11:10
learned about Docker, about learning, about deep learning.
11:13
So that the next six months, it's it's in that six months, I
11:18
feel like I learned way more than what I did in those four.
11:20
Years. So that's again a reason to say
11:24
Ohh man, PhD really wasn't for. Me.
11:26
So when you say you learned a bunch more in those six months,
11:30
it's you learned useful information for the industry as
11:35
opposed to this deep dive on the research side of things, To be
11:41
honest, actually both, right? So the thing with doing a PhD is
11:45
that you go deep into the rabbit hole that might not matter to
11:48
the rest of the community in a few years, right?
11:50
So that that there's always that risk.
11:53
But the good like, so I'm still obsessed about the math side of
11:58
things. So I still try to keep that side
12:01
as well as the technical side. U and the good thing that I saw
12:06
was that that deep learning like did all these weird tricks to
12:11
apply their problems to very different domains.
12:15
I think like one of the early examples I can think of is like
12:18
Gantz, like generating images. Just the way that they did that
12:22
was very, it's very different to how like statistical machine
12:26
learning had done things because they were just going he's a
12:29
classification problem, he is a regression problem.
12:33
You know, the standard stuff and let's try and fit it.
12:36
Whereas this thing came along and said, hey, let's try and
12:38
generate images, right. So it's still using very similar
12:41
tricks, very similar loss functions, but used it in a very
12:44
different. Way and so then you realise
12:47
though maybe I need docker, maybe I need Python skills and
12:52
you you kept up with the math and from there.
12:55
So like. You've done some awesome stuff
12:58
and especially what I love about your story is that you've stayed
13:03
on this deep learning path and you've really gone down.
13:08
I don't know if it's down the rabbit hole, but you've you've
13:10
stayed true to deep learning. When?
13:14
Deep learning. A lot of times the criticism
13:19
that we have from other. Engineers that come on here is
13:24
like. For 90% of the problems, deep
13:27
learning is overkill. Why not just use the?
13:33
The stuff that's passed and lightweight, that's so OK, let
13:37
me let me talk about the lightweight thing.
13:39
Right. So it's 2023 now, right?
13:41
So to say that you can't deploy a deep learning model in
13:45
production because it's too heavy, it's just simply wrong,
13:48
right. So that's that's throw that side
13:50
out because like these days there's Docker, now there's you
13:54
don't need a GPU as well like there's plenty of things that
13:57
you're going to apply to without without just using this normal
14:00
CPUs. That and it will be fast enough.
14:04
O yes, like extra boost is still better for tabular data with the
14:10
with the tabular data, obviously that's that's the one thing that
14:13
non deep learning models have a big edge on South.
14:17
They've tried to beat it like I think Tab Net was one of the
14:19
first papers that tried to beat it and it's still not there yet.
14:23
But again, I do think it's getting closer and closer.
14:27
The gap is skin is smaller, right?
14:28
So I do think it will overtake at some point, but that's,
14:34
that's tabular data, right? So I feel like again, we're not,
14:39
we're not thinking far enough. So again disclaimer, my my
14:43
opinion O. For example, a lot of tabular
14:48
data have text rows in them, right?
14:52
So most of the time when you're doing analysis, you just go.
14:55
Well, we'll just extract a few keywords from this.
14:58
Or yeah, so do some simple NLP type things and then let's do
15:05
the treatise as a normal tabular problem where you have one
15:09
column to predict. But the better way I think is to
15:16
take those text data, use your favourite I know your NLP
15:21
transformer, and then use those extra features and then do your
15:25
predictions. Yeah, so you can go beyond what
15:28
you used to do if you can combine deep learning into this
15:34
whole problem space. So it's basically a step in the
15:39
pipeline and allowing for being open minded about using deep
15:44
learning as one of these steps in your.
15:48
In your orchestration, Yeah, that part.
15:51
I think it depends on the problem.
15:53
So again if if it's a, if it's a something traditional like I
15:57
don't know classified the risk of this customer then or the
16:02
churn models that kind of thing. Yeah, sure.
16:04
Like we we, we know that we have a known process to do this but I
16:11
think what I would like to see happening in industry is go
16:14
beyond the just the journal models just to your.
16:17
What you consider like recommendation or lead scoring?
16:21
Let's talk about search actually rather than recommendation
16:23
because I'm not I'm not as familiar with that.
16:26
So with search, the traditional way of doing it is to extract
16:30
keywords or stem stem words and then search a library and then
16:36
recommend them. But you would obviously rerank
16:38
the results based on frequency of how many people picked on
16:42
them. So I'm trying to get our company
16:46
to look at clip for image search, just simply because when
16:51
you type in a longer query, it makes far more sense.
16:55
The context might matter, right? So so if I say but sitting on a
17:01
tree like the the sitting part is not.
17:05
I mean it's, it's it's, yeah, it compared to both sitting on a
17:09
bench or put under a bench. So that context really matters,
17:13
right, Or the order of the words matter.
17:14
So that gets thrown away as soon as you stem the words and you
17:17
just take a frequency approach to finding it.
17:20
So that's a very good way of getting vector databases in and
17:25
trying to do vector search, especially now that there's so
17:29
many patients in that space. So I know elastic search is
17:33
suffering that now. Everybody I I was at the.
17:39
Databricks and AI Summit and. Every database out there has
17:45
some kind of story around how they are supporting vector
17:49
databases. They're now like a vector
17:50
database. I was at the Mongo DB booth and
17:53
they were like, ohh, yeah, we do vector database too, yeah, yeah,
17:57
it's like, of course you do. Why wouldn't you, wouldn't you.
17:59
Yeah. And it's.
18:00
I think it's important right. So I think in the next few years
18:04
that they will. I'm not sure about overtaking
18:06
like there's certainly there's certainly a reason to use the
18:10
classical things as well. So the reason the recent this
18:14
clip that I did, I I searched for the results are looking
18:18
good, right? I searched for Jesus.
18:20
Hey, now obviously if you get this wrong, like religious
18:23
things, it can be very offensive, right?
18:25
So and like there was one image of Jesus, but the rest of it was
18:31
like, ohh, it's a bit questionable right.
18:32
So. So if you.
18:35
Yeah, so there are cases when you do get it wrong, you get it
18:38
wrong. Spectacularly badly right.
18:39
So, so this is the perfect case of using deep learning as a step
18:45
in your process, right? So use your keyword stems to
18:50
gather a large larger than usual set of images and then you can
18:54
use the deep learning to rerank it.
18:55
So use both the combination, so just just to get much better
18:59
results. Break down this whole clip.
19:04
Yep, blog post that you wrote and what you're doing.
19:07
I mean, we should probably take a step back, say what the use
19:11
case is, say what you're working on at CAMBA, because I think
19:13
that's probably important to give a bit of context around
19:17
what the goal was, what you're trying to do there.
19:21
Yes, so when it comes to multilingual search, we we some
19:28
poorly resourced languages. For example, we don't know how
19:32
to stem a word right? Or we might not have as a bit of
19:35
a library of knowing hey this word map.
19:37
So this when it comes to stemming, so the results when
19:40
it's non-english especially it can be, it's definitely worse
19:44
off than the English search results, right?
19:46
So one way to get around this is to use deep learning to try and
19:53
match U. Here's my query.
19:55
Here's my here's the best image that's possible.
19:58
So how do we do that? What we do is we take the query
20:02
and we encode it into embedding. So embeddings is just a set of
20:06
numbers. And we would also do that on the
20:10
images as well. So what we're going to do is
20:13
instead of matching words now we're going to try and match U.
20:17
Here are the closest numbers that you can get with the text
20:20
encoding. What you can do is there's
20:23
actually a lot of multilingual models right now.
20:25
So if you go to hugging phase social modelling, well there's
20:28
so many out there and the good thing that they can do is they
20:31
can map. So if I if I say cat sitting on
20:35
a bench and it all mapped to the same or same or very similar
20:40
version of that in French as well, so the numbers will match
20:43
up very similarly. And the image embedding it's
20:48
exactly the same, that doesn't matter because it's just an
20:50
image, there's nothing multilingual about the image.
20:53
So what you need to do is I need to search in whatever language
20:56
that you want to. You'll map to the same number
20:58
set. You use a vector database to go
21:00
and search for that in there in your library and Bola, you've
21:04
got your image. And then So what you were doing
21:08
at Canva is you were saying, hey, how can we bring clip into
21:11
this, right? Yeah, so, so the clip, OK, so
21:14
the clip model is is really clip is 2 models, right?
21:18
So clip is how can I use the text model to project it into a
21:23
certain set of numbers and how can I take image model And again
21:28
project didn't set of numbers right?
21:30
So clip is all about trying to line up those numbers so that
21:34
they match. So if you think about one
21:37
dimension like if you can understand the two is close to
21:40
one right? So now The thing is these are
21:43
multidimensional numbers so imagine that we are on this huge
21:47
planet but the distances centre is 1 kilometre right?
21:53
So and as people randomly spread out across the across the planet
21:59
and what we're going to do is we have a second planet and that's
22:05
representing images, right. So now what's really important
22:07
is that this is the direction from from the centre to this
22:12
person on the text Annette, and the same direction from the
22:16
centre to in the image planet. So we have this little magnet
22:20
that comes and says oh OK, this a cat image is pointing towards,
22:26
I don't know, Australia. OK, so and then you go and look
22:30
on the on the text anet, hey, which one?
22:34
Which one of you is a cat? And I'm going to try and drag
22:37
you to towards Australia and but at the same time it even while
22:45
that magnet is doing that, it kind of has this pushing away
22:48
effect of everything else right? So it's not as strong with the
22:52
non cat people, but it does have a small.
22:57
In fact I'm pushing away. So the point I'm trying to make
23:00
is it's trying to line U the direction of the cat image and
23:06
the cat text has to roughly point in the same direction and
23:12
I hope that explain explained that decently well enough
23:14
because now take that 3D planet and burst that into like 100
23:20
dimensions or plus, right. So that's that's effectively
23:23
what, yeah, clip is. I just want to know more about
23:27
these cat people. Tell me about them.
23:30
Yeah, well, that's that's a lot of cat images, right?
23:32
So that's that's that's that's somewhat somewhat of a problem
23:37
because obviously we will do really well with we're trying to
23:41
cap images but when we're trying to find non cat images that that
23:45
so I don't know what's what's a random word a a fabled danceable
23:52
right the Barnstable. So those are very rare search
23:57
terms are very rare. Yeah it's it's barely it's never
24:01
hardly a caption right. Not a possible maybe a tablet
24:04
because it hasn't appeared in a long time.
24:06
Yeah. So yeah, there's very few of
24:08
those training examples, so we don't get to see them.
24:10
Now what I try to do in my blog is to go.
24:14
That's true that we haven't seen a fabric in a long time, but
24:18
what if we take a pre trained image, a pre trained text
24:22
encoder, right? So if I take that, freeze that
24:27
model and if I just say OK, I'll take the last output of that
24:34
model and I project it. So projection is just means
24:37
putting a linear layer on top or a few linear layers on top?
24:42
What if I can take a frozen text embedding a project it to
24:46
certain certain dimensions and and I do the same on the on the
24:52
image side right? So it again freeze the freeze
24:56
the original image model. And the reason that I tried to
24:59
do that is to say let the projection try to move those
25:04
people around to the right place, because the text model
25:08
already understands language, right?
25:11
Yeah. So that way it won't be a
25:13
problem that it hasn't. I specifically haven't seen
25:16
phablet because that projection should just guide it along.
25:21
O let me see if I if I understand that correctly.
25:24
Because of the understanding and text, it's like it it leans on
25:29
that. It's almost like a support
25:31
mechanism and so it can give the direction what it gives us
25:36
starting. OK, so it gives, it gives us a
25:39
starting point, right. So think of that this projection
25:43
layer that I keep thinking of, think I'll pick up that as a
25:45
map, right. So, but this map is more of a.
25:48
It tells you. It tells you the directions,
25:51
right? So it tells you the directions,
25:53
but it does not. It does not care about your
25:56
starting point, right? It doesn't care where you start
25:58
off from. It don't always say left, right,
26:00
left, right until you get to that point right?
26:03
So yeah, so that's really the main advantage of.
26:09
Getting done. So I guess what?
26:11
What's the obvious? Conclusion for me what I have
26:16
been beaten senseless over the last three years when it comes
26:20
to using machine learning from the community is.
26:25
What were the results of this? How did you make a business
26:28
case? Like how did you show that this
26:30
was actually worth doing and worth pursuing.
26:34
So the business case in this case is more for non-english
26:38
search terms, right? So I want to try and see if we
26:41
can improve the non-english and so I should, I should say that
26:46
we haven't tested this in production so we don't know the
26:48
results yet right? So it's more of a question of
26:51
can we can we do that for non-english one so that someone
26:55
searching in say Sri Lanka where which is where I'm from will
26:58
have the same experience as someone searching in Australia
27:01
in an English speaking country. So that's the, yeah, that's the
27:06
business case. We didn't use the original Clip
27:11
image which is also a cannot occupy surfaces everything
27:14
right? So you can download whatever you
27:16
want to from hunger pace. So the reason we didn't use it
27:20
was because it's relatively bulky.
27:22
So my personal goal was to try and see if I can build a smaller
27:27
version of Clip, smaller and multilingual version of Clip.
27:33
Canva is. A.
27:37
Very visual company, right? And so I think about all of the
27:42
different use cases that you are facing at Canva.
27:46
And when you think about what you can spend your time on and
27:51
how you can tackle these different problems, how do you
27:54
look at what different things you can bring machine learning
27:59
to? So that's actually changed a lot
28:04
in the past six months, right? So with this, right, just
28:08
because of the explosion of of AI models, like we really change
28:11
focus and try to yeah, we're trying to use as many models as
28:17
we can as as everywhere that we can.
28:21
Obviously that's not, that doesn't always work.
28:23
Well, you should like there's certain things where you
28:25
shouldn't shove machine learning into.
28:28
I mean, what you've got is. What you've got is perfectly
28:31
fine. I can think of one example where
28:35
I think I got it wrong. Right, so, so we have a creators
28:41
programme and we pay. We pay according to uh, I think
28:47
how much they, how much time they spent on a certain template
28:53
or the the market that they're in like depending on the demand
28:56
and so on based on clicks basically.
28:59
But we do have another like to make things a bit fairer if if
29:03
you think, hey the presentation even though the number of clicks
29:05
below they would have spent three hours on that.
29:07
So let's pay them a little bit more compared to Instagram post
29:10
which is just one one page. So now the the point where I got
29:17
it wrong was to I suggested as if we treat this as a money
29:23
making problem, we got the data on the clicks we got data on
29:29
whether that led to people paying for camera pro or not.
29:35
So we can use that data and machine learning to say, let's
29:40
divide the money up according to what the benefit was to Canva
29:44
compared to fairness, so to speak, right?
29:47
Or what drives up more, what drives are more revenue?
29:51
So you can treat that as as a machine learning problem, but
29:55
that's also it's also unethical in some ways, right?
29:58
So you're you're you're forcing, you're forced.
30:02
Well, it's unethical and it will drive.
30:05
It will make cannabis off. I should, I should.
30:07
That's probably a better point. It'll make can worse off because
30:09
that means people can gamify it and they can focus on Instagram
30:14
posts, which are one page gets in more money.
30:17
South Yeah. So that would have been a bad
30:20
use case of machine learning. Yeah, basically because.
30:26
It incentivizes the wrong activities, correct.
30:30
Yeah, that's right. So it it incentivizes, yeah, so
30:36
like you could probably get a VPN and go to the to the the
30:39
bigger markets and so on. So yeah, it it gives you a way
30:43
of producing low quality templates as well more
30:47
frequently because again, I'm assuming frequency will matter.
30:51
So I think I feel like YouTube is probably one of the places
30:54
where they kind of got that balance right, right.
30:56
So in terms of with the creators, they like when I look
30:59
at my recommendations, it's always very relevant, high
31:02
quality. So, yeah, so they got their
31:07
payment system right in, in my opinion, yeah.
31:09
So I'm not sure if they use machine learning in their entire
31:13
pipeline or if it's just certain bits of that.
31:16
So basically from the last. Six months.
31:21
Till now you've been. Trying to add AI and machine
31:26
learning into everything that you can not not OK.
31:30
It's added cameras a huge like there's a huge machine learning
31:33
team right. So I I'm probably, yeah, I'm
31:38
certainly not one of the most yeah, there's plenty of more way
31:40
more successful engineers at Canva.
31:43
So I think that's one thing. One thing I struggled in the
31:47
past is probably not understanding enough or product
31:52
right? So I've I've always gone behind
31:55
these mathematical problems like Clip.
31:56
Like I just mentioned, I've been obsessed with what was the
32:01
reinforcement learning problem where you have bandits, right?
32:05
So did the payment system. I was trying to think of your
32:07
bandits at that point. So yeah, I spent way too much
32:11
time thinking about maths and not enough about product, so
32:15
that's that's a weakness of mine.
32:18
All right, real quick one. The folks over at Prem AI have
32:21
been kind enough to sponsor this episode, and they wanted to let
32:24
you know that they just got done compiling the most comprehensive
32:29
ebook on the Internet about the state of open source AI.
32:34
If you were wondering what the hell is going on in this space,
32:38
you've got it covered with this ebook.
32:41
It goes over everything from evaluation datasets and
32:44
leaderboards, fine tuning, vector databases, developer
32:47
tooling, platforms, hardware, and of course a deep dive into
32:51
the models, all of which are open source.
32:55
And so how did they do this? Well, the Prem AI team has been
32:59
working exclusively with the open source stack and open
33:03
source models for the past year and they compiled all their
33:06
learnings and what trade-offs you need to be thinking about.
33:09
If you are adding some of this open source material into your
33:12
stack and that can be found at book dot Prem AI dot I/O, you
33:19
can go ahead read it for free. It's out there and it's also a
33:22
living document. So if you feel like something is
33:25
missing, you can submit a PR and get it added to the ebook by
33:29
going to book dot Prem AI dot IO and we're back to the pot.
33:37
Yeah. So I think I'm trying to get my
33:39
head around where where should I put machine learning, like you
33:44
said. And that's I don't have a good
33:46
answer to that yet, right. So maybe ask me in one year's
33:49
time, I'll probably have a better answer.
33:52
Nice. Well, but dude, that's something
33:55
that. Has become so apparent to me
33:58
over the last six months because of this.
34:02
Now, ability to jam AI into so many things that were much
34:07
harder to incorporate AI into before.
34:13
Now you have a lot of people that are able to just get U and
34:18
running and these people are coming at it from different
34:20
backgrounds and I see it. Some incredible product talent
34:25
that. Basically unlocked a new level
34:29
to their ability of designing product and.
34:34
So the flip side of that is now thinking as a machine learning
34:38
engineer or someone who is very deep in the AI and set into AI
34:43
and it's it's inner workings. How do I also make sure that I
34:49
level up and understand the product as opposed to staying
34:54
very, very deep on the mats or the the other pieces of of
34:58
whether it's just orchestrating the data or etcetera, etcetera
35:02
and not thinking about that product.
35:03
So are you doing anything? Specific to try and get you
35:06
there, unfortunately not enough but I think what I what I'm
35:12
trying to do is really talk to more more on the product side
35:16
now the engage a bit more on the with the product managers.
35:18
I think that's really mean. That's really key.
35:23
I am noticing a pattern. Ohh, the product goes still
35:29
remaining the same and you're trying to shop in AI into that
35:34
product which is which I think is is a bad way of going about
35:38
it, but I would. Hope.
35:40
And what do you mean? What do you mean by the product
35:42
growth staying the same? So chum model it's it's, it's
35:46
done and tested like we we know how to do it.
35:48
But the good thing and the bad thing that Open AI has done is
35:54
that it's really, it's really let people imagine, right?
35:59
So there's a lot more that we I should I'll have I'll have a
36:04
little take on that as well but it's it's made non machine
36:10
learning people think hey maybe we can do this and just shove
36:14
machine. So if if there was a sudden
36:16
black box that we couldn't do in the past, Oh yeah, let's just
36:20
use AI to solve this problem. I fully understand what you're
36:23
saying there when it comes to. Hard problems that.
36:28
Aren't easily solved and thinking that AI is your escape
36:33
hatch to be able to just OK? Well, we haven't been able to
36:38
solve this with other methods, so we might as well try AI and
36:43
throwing that in and seeing if that works.
36:46
But. A lot of times I I fully agree
36:49
with you is. Is it needed?
36:52
Is it useful? I think one thing that AI is
36:55
going to help you do is to really sidestep all these
37:00
intermediate goals and then treat whatever the customer
37:05
needs straight away, right. So I think that's that's really
37:07
the capability that we're going to be having.
37:10
So for example, like, OK, let's talk about, let's talk about
37:14
search, right. Let's.
37:16
When people come to Canada, yes, you search for images.
37:20
Yes, you will search for certain fonts.
37:23
But another day, that's not really the goal.
37:26
That's that's like a minor step to get to that goal, right?
37:29
So what AI is going to given us is the capability to go, OK,
37:34
what they come in to do is a birthday card for grandma.
37:38
Just give them that right straight away.
37:40
And yeah, so all these intermediate steps become
37:46
unnecessary. I mean, look like I am these
37:51
these are extreme goals. These like you still you still
37:55
you still need to safeguard your main product, right.
37:58
So I'm not encouraging anyone to just just go and do this
38:01
straight away right. Just be mindful of what can go
38:04
wrong because things again can go wrong really bad.
38:08
But the point is, like you, you can jump.
38:10
There's a lot of cases where you can't jump straight to the final
38:13
product, final need, rather than hey let's tiny steps, baby steps
38:18
and all all the way. Yes.
38:22
I understand what you're saying. There is that.
38:26
We. Again, it goes back to
38:28
rethinking the product. If you know that these.
38:34
Capabilities are possible. How can we eliminate unnecessary
38:39
steps? Yeah.
38:40
Yeah, that's right. Yeah.
38:42
Or or or just, it could be a completely new product
38:46
altogether, right. So, but yes, yes.
38:48
So if you can go, here's what we have here is what my customer is
38:51
asking for. How do we get from A to B rather
38:54
than to ZED rather than green BCD, right.
38:57
So. So, so then talk to me how
39:00
you're interacting with your product team now that you have
39:05
this? A desire to level up on the
39:09
product side of things and be understanding.
39:14
Of the new capabilities, my personal goal has been to do to
39:19
have more meetings with product managers, right.
39:22
So for me, Michael was just just be to talk to my immediate group
39:27
to talk a bit more about what what our goals.
39:32
I'm still, yeah, I'm still not there in the sense that I'm
39:35
still not. I'm still not a good enough
39:39
product engineer, so to speak, right.
39:41
So I've been, I think, and I'm trying to change that.
39:43
I'm trying to stop reading papers and try to focus on this
39:47
but. But you love that match.
39:50
Yeah, it's like a drug. It is a passion.
39:52
But let's let's go back to actually let's go back to
39:55
something that you mentioned before about things have changed
39:59
within the past few months I. My personal feeling is that when
40:05
it comes to machine learning, like it really hasn't, nothing
40:08
has fundamentally changed, right?
40:10
So what's changed is that Open AI came with this amazing
40:14
product and it needed a lot of people's imagination, which is a
40:18
good thing. But, and I think a good way of
40:24
putting this is say if you can pose your problem as a text
40:28
problem then you are going to get far right.
40:32
But if you can't pose the problem as it takes problem.
40:34
So for example, risk modelling, that's still going to do your
40:39
traditional models, that's that you can't, you can't use it in
40:42
our model to do the same, the same problem.
40:46
So that's when the maths becomes important, right?
40:48
So when you try to do those other problems, which is not
40:53
next word prediction, so yeah, so it did that.
40:58
There will always be a place for the maths right?
41:00
It's it's just a render of convincing product to go and
41:05
explore those other things like I don't expect that to happen in
41:08
the next six months. This is just such a huge
41:11
opportunity to waste. Like I think we will be in this
41:14
NLP hub for the next five years. But yeah, there's other problems
41:22
will continue to exist in in different different formats.
41:24
So I think the two cans right now in the machine and space is
41:28
like tree based models and deep building models right?
41:33
The thing is to again coming back to the mats, the loss
41:36
function is still the same right?
41:37
The console the same metric that they're trying to optimise over.
41:41
So when it comes to marrying up maths and programming, deep
41:46
learning as actually I personally think is doing a much
41:48
better job because you can do a lot of wonderful things with
41:53
these loss functions and and come to a conclusion whereas
41:59
extra boost I just found it a bit more awkward to define this
42:02
loss functions. So will they come together OK?
42:07
I I hope so. I hope so.
42:09
But I don't. I don't have any formed opinion
42:12
about whether they can, but I do.
42:16
I do think that the planning will overtake it at some point.
42:19
Our board, Bryan who has something to say about that as
42:23
as he always does. But yeah, so I do think there
42:29
will be an Inflexion point where the tool generally it will take.
42:32
And I do think that will happen in the next, if I had to, if I
42:35
had to bet, I would bet in the next two years.
42:37
Yeah, and boy, on for people who are listening and not super
42:41
familiar with it is a loud Twitter voice.
42:43
Yeah. But but like is, is, is is a
42:46
genius, right. Like I yeah.
42:47
I very much yeah. He's he's a Kaggle grandmaster
42:50
as far as I know. So it's.
42:51
Yeah, he, if anything has, yeah, he's, he's got things to back
42:57
him up, right, like. Yeah, 100%.
43:01
So then the the other question that I was thinking about just
43:07
you having your head in papers and thinking about where things
43:12
are going. I've heard a lot of criticism,
43:16
and you've kind of hinted at this earlier on the current
43:20
architecture and it having its limitations.
43:24
Sure. Do you feel like we're going to
43:26
go past Transformers? Have you seen anything that's
43:28
interesting? Is that even a thought?
43:31
Is that on your radar? Beyond Transformers, it's it's
43:37
doing OK, right? So like I've seen.
43:41
So the problem with Transformers for people who don't know is the
43:45
length of the text, right. So the computational complexity
43:50
is Owen squared, meaning that magnified the complexity of
43:55
almost sentence. That's 4 words.
43:57
Is is like OK, it's 4 ^2 compared to a single word which
44:04
is just one squared. Anyway, point is just squared
44:07
right complexity. Yeah.
44:08
You don't have to do public math where you let listeners figure
44:12
that one out on their own. Grab your calculators.
44:14
Yeah, we're like Chad GBT in that regard.
44:17
We don't like to do public man. We don't do public not.
44:20
Yeah, so, so that's that's the biggest limitation of Transform
44:24
is. But they they do have
44:26
workarounds. Some papers are coming I think
44:30
flash attention. It's probably the latest thing
44:32
to try and deal with it. But the point is like I've seen
44:36
a paper with a million tokens that it could deal with.
44:39
So yeah, there's people arch that tackling the problem,
44:42
right, so. It's it's a version of
44:45
Transformers underneath all of that, so will that go away?
44:50
I can't see anything major coming U anytime soon, but
44:56
again, that might just be me with my limited depth.
45:01
Yeah. Vision, I guess.
45:02
So. Point is, it's it's doing it's
45:05
doing quite well on its own, right?
45:06
So do we need a new architecture?
45:08
I don't. I don't think so.
45:09
Not yet at least. What does excite me though is
45:14
it's this graph ML site. So that is far more exciting
45:18
because the thing with text or tabular is that you can only
45:24
look at data that's linked to it, right?
45:28
So within a row it's always the same row, within text it's just
45:31
it's similar, it's a line of text, right?
45:34
But Graphml, it can pull in images, it can blend text, it
45:38
can pull in tablet data. So that is I think going to my
45:43
personal got feeling is that that will prove far more
45:46
interesting than just Transformers on its own.
45:51
But it can do just marrying up all this data.
45:53
We're still not using all possible data.
45:57
So if I'm understanding that correctly, the idea is.
46:01
Transformers we haven't sufficiently explored, and we
46:05
haven't pushed the limit to where we feel like.
46:08
We're bored of them yet, so we've got a lot of surface area
46:11
to cover, but at the same time we can incorporate things like
46:15
Graphml into what we're doing right now with Transformers.
46:20
Would echo that far like I I'd like to see that happen that the
46:23
the problem with FML is other libraries, right?
46:27
They're not mature enough yet, uh?
46:32
The the guy who was at Pinterest for a while, I forget his name.
46:36
I think he had one of the the leading libraries.
46:39
But the point is like not many people are working on it.
46:41
So everyone's excited about Transformers.
46:44
So there's a lot of brain power going into that area right now,
46:48
not so much this other area. And that's fair enough.
46:50
It's it's It's not a lot of value on its own right, but.
46:55
I do think, yeah, when it comes to and let's you even think
46:59
about SQL databases, right? So you can link up a whole bunch
47:02
of data types and even missing data types which.
47:07
Which current mill struggles with?
47:11
Yeah. So if we if we can figure out a
47:13
way to link data together, I think that will, that is where
47:17
graphic novel can come in and play together.
47:19
I think Transformers will still be a part of that, right.
47:22
So there's nothing special about Transformers.
47:25
It's just this attention mechanism that it has.
47:28
So if I'm looking at a word cat, like how do I know which word
47:33
it's paying attention to? That's that's really what's
47:35
special about transform is this. Everything else is just just
47:39
linear layers. Can you explain to me exactly?
47:42
What graph is? Because I only know it through,
47:46
I'm pretty sure I've just seen it.
47:48
In reference to databases, maybe and then also.
47:54
Like different teams on different database companies
47:56
that I know are working on Graphml.
47:59
OI think we need to make the distinction between Graph Mill
48:02
and Graphql, right? So I think a lot of people when
48:05
they say we're doing data science with graphs, they
48:07
usually mean this graph QL thing which is somewhat mature
48:11
product. But Graphml is doing inference
48:15
on graphs. So for example, I think Twitter
48:18
did this. So one way of saying whether an
48:22
article is fake data is literally train a classifier on
48:26
natural language, right? No, you can get plenty of data
48:31
on that On national language have people labelled things as
48:35
going fake or not fake. But what they did differently,
48:38
which is which I thought was really unique, was instead of
48:41
looking at the individual tweet, what they looked at was the the
48:45
network of how these things spread, right so.
48:50
Fake fake news. What happens?
48:53
The tendency was that these bots would come and retweet them,
48:57
right? And it would grow at a faster
48:59
speed pace compared to normal news O that looking at the
49:05
network, looking at how it's spread that and doing that
49:09
classification on the network actually led to far more
49:12
reliable results of saying what was fake was not.
49:16
Rather than saying, let's look at the content of the off the
49:20
tweet. Because again, like what's fake?
49:21
News to one person might not be fake news.
49:24
See others. Or it may be fake in. 20. 20 And
49:27
then in 2023 it's not fake and so you learn new information or
49:32
whatnot. So it's really about and and
49:35
graph is just figuring out how how like how does graph play
49:40
into that. So what was graph about that?
49:44
So the graph about that was. So if you think of the the the
49:49
original Twitter as a note, you would think of the edges as who
49:55
would read it or who would who would read it.
49:58
And the the other side of the note is who would retweet it,
50:01
right? So in that case, they were
50:05
looking at this like look at that you piece of information,
50:09
which is a pace of transmission on that thing.
50:13
But yeah, so that's. Where you get the edges and the
50:16
nodes and yeah how do I marry U these all this data so you
50:24
mentioned. Graph QL.
50:25
But then there's also Graph Mill.
50:29
Is Graphml, does it have anything to do with databases or
50:32
was that whenever I heard graph look?
50:35
It can as long as you define what the nodes are, right?
50:38
So if you, if you think of a database as a node, right, a
50:43
table now no, no no. I'm not doing justice for does
50:47
the graph and health community. But this is again in my very
50:50
limited knowledge they're going to riot.
50:52
You're gonna see the. Castle quick, So yeah.
50:56
So when I add SQL queries I'm really joining up to libraries,
51:01
right? To points of data.
51:03
So sometimes that data might be missing, right?
51:08
So you might get the first few columns populated and the next
51:11
few columns missing. O.
51:13
The good thing about Graphml is that it can have a varying
51:17
number of nodes around it, right?
51:19
So when we're doing this machine learning on all graphs, if
51:23
there's missing data, it's equivalent to not having a node
51:26
at all that is joining to, right?
51:28
So we can have variable amounts of data points, so this
51:32
information that comes into that node when you're doing the
51:34
machine learning over this graph, this entire graph depends
51:38
on what it's connected to and what's already populated, so we
51:41
don't. We might not have this missing
51:44
data problem at all when it comes to Graphml, so I know
51:48
Xgboost and stuff is always quite good at dealing with
51:51
missing data. But I'm very much talking about
51:54
OK, what? What if I treat this as a bunch
51:56
of nodes coming in together? Awesome.
52:00
Graham is the. Future You heard it here first.
52:03
It's, it's. It's untapped.
52:05
It's untapped pressure. That's that's my that's what I'm
52:07
saying. Ohh, I like it.
52:09
Yeah, yeah. We can make sure it's an I'll
52:12
send one over to you and that is well I'll say ohh because I'll
52:17
send you the link of the person I've talked to like he's he's
52:19
he's I think he did a Stanford series on graph melas quite yeah
52:23
quite fond of the way how he did it so nice.
52:27
Yeah, we can add that. To the show notes, so anybody
52:29
listening can also. Have a gander at it and now.
52:35
There. Is one thing I want to talk to
52:36
you. About also, which I find
52:39
fascinating when it comes to. How long you've been playing
52:44
around? With GPT, Yep, because you
52:48
mentioned that you fine-tuned GPT 2.
52:53
So a what was that experience? How did you go about that and
52:57
like, tell me more? I think there's probably a
53:01
handful of people that were playing around with GPT before.
53:05
There's a boom, yeah. So I think what a lot of people
53:08
don't realise is that GPT is quite old, right?
53:11
It's about like the original GT is about four or five years old
53:14
and people have been playing around that for a long time.
53:16
So the two years ago I wanted to do so I was working this user
53:20
voice team. We basically we get all these
53:23
problems from users saying hey, can you help me with the
53:25
payment? Can you help me with my design?
53:29
Can we have automatic agent having automatic replies or at
53:33
least at least do a portion of our users so and what I was
53:38
going to do was train. Here's the input of here are the
53:42
tickets that came in. Here's the output of how our
53:45
agents are like human agents responded to to those tickets.
53:51
So you have your input, output and yeah, what?
53:55
And I want to train a GPT model now it didn't at the point like.
54:00
My knowledge in training those models wasn't too great, so I
54:02
didn't really get too far, but. So these days it's a lot easier.
54:07
There's a lot of tools that you can, you can quantize it, you
54:10
can reduce the like. Right now 30 point is 4 bytes.
54:16
They receive 4/4 bits recently as well.
54:20
So the point is there's a lot more, there's a lot more tools.
54:23
I didn't actually use the compass tools, I just did the
54:26
normal fine tuning in my little blog post where I did some
54:28
grammar correction thing but the.
54:32
Point is that you can you can. You can do that already using
54:36
only face, just using manual without having to resort to
54:39
using Open AI. Yeah, so Open AI is easy, quick
54:46
way of doing things now. Like it's such a it's such a
54:48
knowledgeable model that like I don't, I don't need to train a
54:52
very specific model for grammar correction, It can correct the
54:55
grammar for you, right? Because it knows about the
54:57
entire world, so to speak. So you can get around that
55:02
without any training with Open AI, but if you want to save a
55:05
bit of money, if you want to do things in the house, I think, I
55:07
think it's still valuable to have those skills within your
55:10
company. That's the huge trade off that
55:13
I'm seeing. Because people are able to get a
55:17
quick. Feature out there and it goes
55:20
back to that product discussion that we were talking about
55:22
before and see if there's any value and then if there is value
55:25
they can bring it in house and they can figure out phase two of
55:29
it. I think that's that's a that's
55:32
an amazing strategy, right? So test, test it out with the
55:35
Open AI. Because another day, yes.
55:38
It is like if you if you already set up the tools, yeah it is
55:42
cheaper to. Train it in house.
55:44
But you know, factoring in your engineering time, right?
55:47
So you know, factoring maintenance costs and all that,
55:50
so. Yeah, by all means test out.
55:54
Just start to put off using to open AI API and then come back
56:00
to train training model. Yeah, I mean it.
56:06
If you can, right? Because I know there's a lot of
56:08
use cases out there that. Are not allowed to send data or
56:13
do anything outside of their little walled garden.
56:15
So in those cases it makes a lot of sense.
56:18
How have you found? I guess you probably are.
56:23
Very opinionated about. The control that you have.
56:27
With models that you are working on in house as opposed to
56:31
sending out to some third party API's, have you looked at those
56:35
trade-offs? I am guessing just from talking
56:38
to you for the last hour that your opinions are that it's it's
56:42
better to be in house always, although you're not.
56:46
No, no, I'm not. I'm not that that strictly
56:49
around that because end of the day there's only that camera we
56:54
have. I think now we have about 50 or
56:57
60 analyse, but we have a lot more teams, right?
57:00
So. So.
57:03
If you don't have an in house mill Imagineer, obviously you
57:07
can't. Like you can't borrow other
57:09
teams engineers. So yeah, by all means go ahead
57:13
and do this because yeah, so a lot of problems that ML
57:19
engineers can do have been, especially if you're an LLP
57:22
side, which is what our words can be done by Open AI, right.
57:26
So it's a back end problem now. So it's no longer a no problem,
57:30
say. It so it's a very much a
57:33
resource problem, right. So do you.
57:35
Yeah. Do you want, do you have miles
57:38
to do the miles have different problem to work on which is far
57:40
more important, right. So and the third one I guess
57:43
like the very important thing that you mentioned is speed like
57:45
speed to market, right. So which which is really going
57:48
to be crucial for a lot of companies in the next few
57:51
months. There's a piece.
57:53
That I wanted to go. Down the rabbit hole, a little
57:55
bit of leading other ML engineers and leading teams and
58:02
how you look at that and how, what have you found?
58:06
To be successful and where do you feel like?
58:09
Maybe others. Can.
58:13
Make missteps. I think I've made a lot of
58:15
mistakes, right. So I think the biggest mistake
58:18
that we've done in the past is really not aligning or
58:21
understanding product, right. So I would too often be
58:25
interested in a mathematical problem, whereas the problem
58:27
really that brings in the money is way very different to to
58:32
that. So that's that's been a huge
58:35
weakness of mine and I think and I think maybe I'm brushing
58:40
everyone with a with a broad stroke, but I think that that is
58:43
going to be a problem of MLAs especially if you come from
58:46
academic background, right, trying to make that transition
58:48
from academia into industry where the motivations are very
58:51
different, so. Yeah, I think.
58:56
I think people, if I was to advise my younger self, what I
58:59
would tell is engage this product a little bit more often.
59:04
And faster, simpler models and that would maybe.
59:10
Not simple models, so I still think.
59:12
I still do think that all these cool new toys do make a
59:16
difference, right? So let's just tougher with the
59:19
simple morons for a second. But fast.
59:21
The speed? Speed is crucial, right?
59:23
Speed of how you how fast you finish your model and move on to
59:28
the next is very important. Umm.
59:32
Yeah, just so you can show off a few more toys, right.
59:34
I think that's that's the best way of putting it, yes.
59:38
And talking about that, the 1% of these complicated models and
59:44
this these hard math problems. How do you go?
59:48
About. Looking at the trade-offs of
59:52
that because it it seems to be a theme of hey, there are very
59:57
valuable problems that you can hit.
59:59
You just have to choose your battles.
1:00:01
Yes, you have to choose the battles.
1:00:03
Yeah, I think that's a very nice way of putting it.
1:00:06
If I was to put a concrete number, I would say try and get
1:00:09
a turn around time of Immortals of three months, I think that
1:00:12
should be that should be the absolute Max.
1:00:15
If you if whatever you're doing is taking is or, you can see
1:00:19
that it's heading down that direction.
1:00:20
Maybe it's time to look at a different solution.
1:00:25
Work on. Something else?
1:00:28
Because end of the day what you're what the business really
1:00:31
needs is you to deliver, right. So coming back to Harding phase,
1:00:35
the good thing about phase is that and all these other faster
1:00:38
libraries is that they've given you a lot of tools to do that,
1:00:41
right. So you don't need to compromise
1:00:45
on the fancy stuff if you can use these tools O that's that
1:00:50
has changed, right? So yeah, by all.
1:00:53
Means go for bigger, more complicated models, but use the
1:00:57
tools that are available rather than trying to do everything
1:01:00
from scratch. So again, this is me to switch
1:01:02
to myself, but I think, I think that a lot of people will
1:01:06
resonate with that. I also think so and.
1:01:10
It is one of those reoccurring themes that comes up time and
1:01:13
time again on this podcast. And I don't think we can say it
1:01:17
enough. What are some?
1:01:19
Signs in your eyes that things. Aren't going as they should and
1:01:24
maybe you need to scrap it and how do you have that kind of
1:01:28
self talk of hold on just a little bit longer or let's scrap
1:01:33
it up but it's it's a like a phone.
1:01:36
Like one thing I found surprising about myself is like
1:01:38
I'd get too attached to projects right that go.
1:01:40
I'm surprised that I do that because like, I know that I want
1:01:44
to be wanted different things, but.
1:01:47
Yeah, I don't. I don't really have an answer
1:01:50
for that, but let me let me talk about something that's
1:01:52
tangentially related. So one thing I didn't understand
1:01:56
was about metrics, right. So when I'm talking about
1:01:59
metrics, I'm not talking about your precision recall scores.
1:02:03
I'm not talking much accuracy. What I mean is like product
1:02:05
metrics right? Which can be very different to.
1:02:10
To what? Machine learning engineers think
1:02:13
O. My keyword keyword model for
1:02:16
example. So in that case one thing that
1:02:20
we yeah the accuracy that we the metric that product key was
1:02:25
about the pick rate like out of the out of the keywords and
1:02:29
suggests how many of them actually picked.
1:02:31
So I think that's recall in all terms now, like in wait time
1:02:36
biases. I did my position recall, it's
1:02:39
very different to what the X file recall was, right.
1:02:42
So it's in English, it's. Course in non-english it wasn't
1:02:46
close right So. Yeah, so I think I should have
1:02:51
paid a lot more attention to testing my model as fast as I
1:02:56
could and then coming back and iterating, right?
1:02:58
So now I've moved on different projects, so I can't quite
1:03:01
improve their model at this moment, which is a shame.
1:03:05
But the point is I should have been.
1:03:08
I should have been faster. I should have been using the
1:03:12
hugging face libraries rather than do do my own thing.
1:03:16
So yeah, that's focus on metrics from across the product metrics.
1:03:21
I think that's a good way of figuring out is your model going
1:03:25
the right way? It should be.
1:03:27
That's so good that is. It it just hits home that idea
1:03:30
of product and how important it is to have that product
1:03:34
mentality and recognising what's important in it.
1:03:38
It's a bit. Of a It's like human.
1:03:40
Nature aid having a hard time letting go of your creations and
1:03:45
then be. What it's like?
1:03:50
At least inside myself. I also find myself not wanting
1:03:56
to reach for off the shelf, off the shelf solutions and trying
1:04:01
to think, oh, I could do this better and so almost reinventing
1:04:05
the wheel. There is value in being able to
1:04:08
tinker through things, changing things and I do generally think
1:04:13
that we will make you a better engineer in the long term, but
1:04:16
the business also needs you to be a different person in the
1:04:18
short term, right. So it's a question of balancing
1:04:21
that and that's balancing things is probably the biggest thing
1:04:24
that I struggle with, with with today like yes, you've got your
1:04:28
product versus reading papers problem as well, but I'm like
1:04:33
I'm a parent now so I've got, I've got two kids like I can't
1:04:36
like I cannot keep up with the explosion of papers that are
1:04:40
coming up so. Yeah.
1:04:44
So that's that's a bit of a that's a that's a real struggle
1:04:46
that I don't have an answer for. I feel you, man.
1:04:50
I also have. Two now and just trying to keep
1:04:54
up with everything is a job in itself.
1:04:58
Yeah, like. And my children are very
1:04:59
demanding of my time. They're they're getting older.
1:05:01
They're the more time that they demand of me.
1:05:04
So yeah, it's just it's a bit hard to do right by everyone so.
1:05:12
But again, I think, I think if you can get your product
1:05:14
thinking right, you can, you can kind of get away from not
1:05:18
needing to keep up to date with absolutely everything.
1:05:22
Standing What's important? I don't.
1:05:26
I I. Don't again I don't know what's
1:05:27
important because these days that the only one thing that
1:05:30
seems to be important is just the open AI API and everything
1:05:33
else is just blah whatever if there.
1:05:36
Yeah. I would love to see how other
1:05:38
parents MO engineering parents deal with machine learning
1:05:41
thesis that that's something perspective that I'd love to
1:05:44
hear about because this is a very very demanding job being a
1:05:48
MI engineer. So excellent.
1:05:51
Well Sachin, this. Has been a brilliant
1:05:53
conversation, man. I appreciate you doing this.
1:05:55
I know it's late. Where you are in the world, I'm
1:05:57
going to let you go. We'll just, we'll end it here.
1:06:01
Thank you so much for having me. Really appreciate it.
1:06:08
I'm Emmanuel Mason, Machine Learning Engineer at Stripe and
1:06:12
author of Building Machine Learning Powered Applications.
1:06:15
And if you don't want your machine learning models to
1:06:17
explode, well, you should subscribe to this podcast.