Latest / Agentic Conversations (formally mlops.community) / Navigating Machine Learning Careers: Insights from Meta to Consulting // Ilya Reznik // #286

Transcript

Navigating Machine Learning Careers: Insights from Meta to Consulting // Ilya Reznik // #286 · Agentic Conversations (formally mlops.community)

0:01
So yeah, and Ilya Resnick, I am the instructed machines janitor
0:07
and the CEO at the same time. And I actually don't drink
0:10
coffee. Welcome back to the one and only
0:14
Envelopes Community podcast, folks.
0:16
Today I feel honoured because we get to talk with Ilya and
0:22
there's not many times that you can chat with a staff level
0:27
machine learning engineer from a company like Facebook or Meta,
0:32
as it's called these days. Today is that day.
0:35
Ilya recently resigned, I guess you could say, or quit the way
0:40
that he likes to say it. He was more blunt about it and
0:45
he did some consulting afterwards and now he's spending
0:48
his time helping others recognise the best way to become
0:54
a staff machine learning engineer.
0:57
So we talked a little bit about that.
0:59
We talked about his recent trip to Europe's and everything in
1:04
between. For those that are tuning in on
1:08
the podcast level, I've got a nice little song that I will
1:12
share with you. This is one of my favourite
1:17
bands D Pulse and they did a little remix of one of my
1:23
favourite songs, The Velocity of Love.
2:19
We were talking about why fine tuning is not all it's cracked
2:23
up to be and you said that. The TLDR is don't do it.
2:30
Yeah, fine tuning is a great technique and there's a lot you
2:35
can get out of it, especially like, you know, in in the LLM
2:39
world, if you're trying to get a particular shape of an output
2:44
and it, it may be worth it, right, like a four year really
2:47
after his Jason. Like it may be worth fine tuning
2:49
the model to that, though modern models like understand Jason a
2:53
little bit better. And, but, but the problem is
2:58
like every time you do it, it's a lot of effort And the first,
3:01
the first model to come off the press is usually worse than the
3:06
original 1. And the whole idea of fine
3:10
tuning is you're taking this huge LLM or whatever other model
3:14
that can perform really well on many different tasks and you're
3:19
trying to get it to perform better on your task at the
3:21
expense of performing worse on all the other tasks.
3:24
And that works sometimes. But if you if you can prompt
3:30
engineer your way out of it like, you should do that first.
3:34
Yeah, the the other piece on it, which is hilarious to me, is
3:38
that you fine tune it and it's a bit of a crapshoot because you
3:42
don't know if what comes out the other side is going to be better
3:45
or worse until it comes out. So you spend all this money and
3:50
time and energy on fine tuning just to recognise that it's
3:55
actually not that good. Yeah, yeah, you got to be very
3:58
careful with the data you're fine tune on as well.
4:00
And especially like you can fine tune a large model on a few 1000
4:05
examples, but you got to be pretty careful about what those
4:09
a few 1000 examples are. People usually put what they
4:12
have rather than like, think through like, OK, like how
4:15
imbalanced are my data? Like what's what's happening
4:19
here? So yeah, fine tuning is, is a
4:22
great technique. I liken it to training and
4:25
embedding in the 1st place. You know how that's really more
4:28
magic than anything else? Because I've trained so many
4:31
embeddings where you just pulling your hair out, you can
4:34
see it's all gone now. And I used to look like you
4:39
before I started training in bedding models.
4:41
And. And then like at the end, you're
4:44
like, oh, my objective function is wrong.
4:46
I shouldn't be training for this.
4:47
I should be training for something else.
4:48
And you switch that and everything.
4:50
It just works magically the first time, you know, the first
4:52
time after months and months of work.
4:54
Yeah, So. Yeah, and you learned that the
4:57
hard way. It's not like you can read A
5:00
blog post and be like, oh, this is exactly what's wrong with
5:02
mine. You have to just kind of trial
5:05
and error it. Yeah, well, and in fact, there
5:07
is no blog post, right? Like in fact, all of these
5:09
problems are one offs. And just because it worked this
5:12
time doesn't mean it's going to work on the next one.
5:14
And so there's there are best practises.
5:17
So you start there, but at the end there's some voodoo.
5:21
You got to make sure nobody's poking your doll with a needle.
5:26
But I mean I've heard a lot of this fine tune for form and one
5:33
thing that I can't really get my creativity around is what are
5:38
the other reasons you would fine tune for form if it's not Jason
5:42
output? Yeah, well, I mean, there's
5:44
other structured output, right? Like HTML or Markdown or
5:48
whatever, but, but also like sometimes it's not just form,
5:52
sometimes it's if you're in an area where the language is
5:56
really particular. So like expert language is very
5:59
different. I worked with a medical company
6:01
and we, we had to do that. And like that, that's another
6:06
place, right? Because the distribution of
6:08
tokens is really different when basically every word that you
6:11
say is Latin with an occasional English in the middle.
6:14
And so, so those kinds of scenarios where like your output
6:19
is quite different, people try to do that for like knowledge.
6:27
So if there's a very particular set of facts, but the problem is
6:31
LMS don't have knowledge, They don't store facts, they store
6:35
probabilities. And, and so like in that case, I
6:39
don't think it's useful at all to fine tune.
6:40
I I I haven't found it useful yet.
6:43
Yeah. That's.
6:45
The where rags will get you a lot further and.
6:50
Even RAG is no panacea. Like what RAG does is it puts
6:54
the fact a lot closer in context, but it doesn't mean
6:58
that it depends on how strong the prior was for the LLM,
7:02
right? And and like if the prior was
7:04
really strong, it still won't pick it up.
7:05
All right. So what people call
7:07
hallucinations or not hallucinations, they're totally
7:11
like they're an artefact of the way the model works.
7:15
And and so the term hallucination assumes that like,
7:17
oh, you're just like out in left field.
7:19
You're not like this is a reasonable prediction based on
7:22
the distribution of training data.
7:23
And so I think, I think LLMS are extremely useful.
7:28
I think they will be with us for a long time.
7:30
I do not think that they're going to be the only technique
7:32
going forward and. What do you mean by that?
7:36
I think there will be so like on the non ML side right?
7:40
Like you have databases, but nobody is like, oh, you know,
7:43
like I don't know. Mongo DB solves this problem
7:47
100%. No, like Mongo is like part of
7:49
the solution that's there that we rely on and it's useful.
7:54
Well, it depends on who you ask. Some of my friends are like
7:57
don't ever use Mongo, but but you know, you need to build a
8:01
solution around it. And LLM is similar, like you
8:04
need to build some knowledge retrieval around it.
8:06
Is it rag? I don't know, maybe it's graph.
8:10
You know, you, and there's a lot of talk, I, I know you want to
8:15
talk about this. So I don't know if I should be
8:17
moving there yet, but like, I was just at new Europe's last
8:20
week, right? And there's a lot of talk about
8:23
how we ran out of data and what we ought to do is reinforcement
8:28
learning. And people are like, wait a
8:29
minute, RLHF and all the reinforcement experts are like,
8:33
ha, ha, ha. That's not reinforcement
8:34
learning, not at all. Reinforcement learning is one of
8:37
those cool things. It's always like the next big
8:39
thing, but never has been the big thing, you know?
8:42
Yeah, like agents. So.
8:45
So I think there will be some reinforcement learning in that
8:47
system as well. There will be some rag in that
8:50
system as well. There will be some other
8:53
techniques around the LLM that make it actually useful.
8:58
It's interesting to think about like you're talking about, OK,
9:01
databases are there, but then you have different flavours of
9:03
databases, whether it's that your mongos or your cockroach or
9:07
Oracle or whatever. And and you as the architect of
9:11
the system get to decide what your trade-offs are and what
9:14
you're really trying to optimise for.
9:16
And so you're going to have maybe it's an LLM is the centre
9:22
of that or maybe it's not. And maybe it's a a graph rag.
9:26
Maybe it is a fine-tuned LLM or a lot of fine-tuned small
9:31
language models or something like that that we will see.
9:36
And it's cool to think about that as you get the flexibility
9:41
to choose. One thing that I do feel like
9:44
though is that you have some standard patterns that are
9:49
coming up. And if you look at it like
9:52
trails in the woods or trails to the top of the mountain, you
9:54
definitely see there are standard design patterns that
9:58
folks are are kind of taking and one of them is not the fine
10:03
tuning route. And so I think that that's
10:05
pretty clear as to all right, if we're going to fine tune, we
10:09
better have a really good reason and we better really need to do
10:13
it because everything else up until this point has failed us.
10:16
So it's like fine tuning is our last option.
10:19
Yeah, fine tuning is expensive and it's, it gives even when
10:25
you're super successful, it gives you a limited benefit.
10:28
And so like sometimes that's what you need.
10:30
I, I absolutely agree with you. It's all trade-offs, right?
10:32
Like sometimes that last like 2-3 percent is exactly what you
10:36
needed, but it's not, you know, if the model is really bad at
10:41
math, fine tuning is not magically going to make it
10:43
better. But if the model is like, oh,
10:47
you know, you keep giving me really informal things when I
10:51
ask you for a formal letter or, you know, what I really care
10:54
about is Latex output. And what you keep giving me is
10:59
markdown. Like, yeah, that's, that's a
11:01
great place for it. And you could probably get that
11:04
kind of improvement from fine tuning, but but it's not end all
11:09
to be all about the best practises.
11:11
I do want to caution that you're in shifting sand a little bit.
11:16
We're all in shifting sand a little bit.
11:18
So like best practises that emerge today, in two years we
11:21
might be like, Can you believe we used to do that?
11:26
That's such a great point. I mean, even just with RAG, how
11:28
now people are like, yeah, what we did a year ago is called
11:31
naive rag because we were naive back then.
11:33
Now there's this advanced RAG or you got to do a gentic rag.
11:38
And even the what is it? The React agent architecture
11:43
that's now not even seen as really useful because there's
11:48
much better ways to do it. So it's true that this is going
11:52
to be shifting very quickly and it's almost like month by month
11:56
you're seeing new and better ways to do things.
11:59
And maybe they seem new. You get a lot of energy and
12:02
attention around them and it feels like, wow, this is much
12:05
better. And then after the community has
12:08
played with it a bit, they recognise that it's not actually
12:11
that valuable. So you are, yeah, shifting sand
12:16
is a great way of of putting that because you are not super
12:20
clear on what is signal and what is noise.
12:23
Yeah, yeah. And, you know, it's evolving.
12:27
There's a lot of money going into this.
12:29
Obviously, there's a lot of benefit to getting this right.
12:35
It's not the kind of thing where you can learn everything now and
12:38
then spend the next 20 year career on what they've learned.
12:40
It's the kind of thing where you learn stuff now and then you
12:42
learn stuff for the next 20 years.
12:44
Yeah. Totally.
12:47
Well, Speaking of which, I mean, that's what the conference like
12:50
you were at last week. NURB is why it's so cool, right?
12:54
Because it brings all of the newest stuff out into the open.
12:59
What were some of your favourite takeaways besides not being able
13:03
to order an Uber like you mentioned before we hit record?
13:06
I actually use basses most of all, more than Uber is there
13:10
because Vancouver is a very good public transportation, say.
13:14
I don't know if people from there would say that, but where
13:17
I stayed it was definitely really useful.
13:19
But yeah, I mean, it's it's New Ribs is terrific.
13:23
New Ribs is the flagship conference in ML has been for
13:27
decades. The first New Ribs came out the
13:30
year that I was born. So it's been around for a long
13:33
time. And it started out as a very
13:36
academic conference between neurobiologists and machine
13:39
learning people when they were like, hey, we can learn stuff
13:42
from each other. And the, the main takeaway from
13:45
this really is that I didn't see that many truly academic papers
13:51
and posters and presentations like their their majority
13:55
applied right now. And, and there's a huge push
13:59
into like, OK, like we have transformer models.
14:02
How do we use them? Like, how do we actually make
14:05
them useful, which is not usually the focus of Nureps.
14:08
The focus of Nureps is usually let's look for a new
14:10
architecture, let's go off. And so there's definitely
14:14
convergence on Transformers and and you see this from different
14:18
industry. Like when I was coming up in
14:20
this, it was like, if you did computer vision, you didn't do
14:23
language. Like I was the weird one out
14:25
where I was like, no, I can do either.
14:27
But but now it's like everybody's working on the same
14:30
model. SO1 invited speaker put it
14:33
really well. They were like, you know, we
14:35
used to have a lot of models and then we kind of converge toward
14:38
the transformer architecture, but now we got to diverge back
14:41
into the applications and how does this apply to everything?
14:45
So that's probably my main takeaway is that there's a lot
14:49
of thinking that's happening about the system that we talked
14:52
about. Like what does that have to have
14:54
and how does that work? You know, like, and we're way
14:58
more advanced than we used to be a few years back.
15:00
We're talking about like left to right LLMS versus right to left
15:04
LLMS and like they do really well in different kinds of
15:07
things. Right, like the left to right in
15:09
English there, you know, there are languages that are read the
15:13
other way. So yes, but it but in English,
15:15
the left to right does really well at language.
15:18
The right to left actually does math a lot better.
15:20
And we're talking about not every token is as important as
15:25
others. So some tokens are fuzzy and
15:27
maybe you should be treating them as such.
15:30
We're talking about reinforcement learning.
15:32
There's a lot of excitement about using reinforcement
15:34
learning as we're running out of data or it feels like we're
15:38
running out of data. I'm not even sure I buy that
15:40
argument that we're running out of data.
15:42
There's, you know, Ilya Suskover talked about that.
15:45
And I'm like, his argument was that we only have one Internet.
15:50
That was like, dude, we're uploading a new Internet like
15:52
every two weeks. Like this podcast did not exist,
15:55
right? And so like, now it does.
15:58
There are rumblings on the side and the hallway track.
16:01
This didn't make it into the main conference, but there are
16:03
some rumblings about curriculum learning, which is a thing we
16:08
kind of gave up on entirely. And.
16:10
And what is that? Yeah.
16:12
So. So first of all, the way that we
16:15
learned today, it's like if you know, when you were little, your
16:18
mother came to you and said, here's 80% of everything I know
16:22
shuffled randomly. Now I'm going to test you on the
16:25
other 20%, right? Like that's just not what
16:27
happens with kids. What we do is we tell them,
16:30
here's a sound, here's a word, here's a sentence and they
16:34
progress through that in a supervised manner.
16:37
Like a teacher tells you, this is the first thing you need to
16:39
learn. This is a second thing.
16:41
It's a but that's not what we do with models.
16:45
What we do with models is like shovel all the soup and like you
16:48
go figure it out. And, and, and so curriculum
16:52
learning is about how do we create that step by step where
16:56
like a model can learn something basic 1st and then start
17:00
building on top of that. And it involves a lot of data
17:03
curation. It's very like time expensive,
17:07
but it might be worth it. It might get us better outcomes.
17:11
And so like we're starting to talk about that again.
17:14
That hasn't been a conversation for the last like 20 years,
17:17
honestly, but it's coming up again.
17:21
And just because it's more structured, it will be more
17:24
efficient. Is that the idea?
17:26
Well, the idea is that it respects the concepts in the
17:31
real world. It respects that like, you know,
17:36
addition and subtraction are prerequisite to multiplication.
17:40
And so the idea is that you can be quite a bit more data
17:45
efficient and you can also like start really understanding
17:48
concepts and symbols that represent the real world.
17:53
And then you can start avoiding things like hallucinations
17:56
because you're not reliant entirely on statistics.
17:58
Those statistics came step by step, and you know that the
18:02
early steps are fundamental. It's problematic though, because
18:06
it's a lot harder. But but also, you know, NURB was
18:11
always about being inspired by biology, and this is very much
18:14
inspired by biology. So yeah, it's an interesting
18:19
concept. I don't know if it'll ever gain
18:20
traction. Lots of things we talked about
18:22
in NURB never really leave, but but that was one that struck me.
18:27
The other thing for me with my background, so when I was at
18:30
Twitter, I owned model evaluation, offline model
18:34
evaluation for all of Twitter. And the the interesting thing is
18:38
there's been a lot of talks on the eval.
18:40
So how bad they are and? Even at nureps, there's still
18:45
talks. Even at nureps.
18:47
I feel like. The only talks I saw this year
18:49
were rag talks and eval talks. And it was kind of, if you boil
18:54
them down, it was like rag is difficult and you got to watch
18:58
out for where it fails. And here's all the way this ways
19:00
that it fails and maybe here's some tricks you can do.
19:02
And then eval talks were like eval suck.
19:05
We don't have any good. Like these leaderboards are
19:09
steering you astray. It's all marketing.
19:11
And don't listen to anybody who says that they have a soda
19:13
model. Yeah, yeah.
19:15
So there was a talk that I went away with like mind blown and it
19:18
was about something else. But this is the line that like
19:20
blew my mind. And the speaker was like, yeah,
19:23
like we have this benchmark and we got 95% on it, you know, or
19:26
90 something. I don't know.
19:28
I'm high number and the model can't do anything real in the
19:30
real world. And I'm like, how?
19:32
Like what is the disconnect that you can get 95% on your
19:36
benchmark and still not be able to perform any useful work?
19:40
And like, clearly your benchmark is not.
19:42
What do you think it is? Yeah.
19:45
It's not valuable. And and then like you add, so
19:48
this is like evolves are a mess early on.
19:51
And we talked about like LLM as a judge a few years ago, like I
19:56
was fortunate enough to work with awesome people at Arise and
20:00
on like explaining this and putting that concept together.
20:04
But. But that's not even the panacea.
20:07
And and then you go into the the agentic world and and then you
20:14
go into like multi agent world. And I don't even like how do you
20:18
evaluate a multi agent system? Like the amount of complexity
20:22
and and you know, like some of this is understandable, right?
20:25
I think earlier in his talk talks about how as the
20:29
intelligence grows, it's understandable that you will
20:32
have a harder and harder time evaluating it, which is cool.
20:37
But my job is to make sure I can evaluate it.
20:40
And so I don't know how to do that.
20:43
Honestly. There's, there are a lot of
20:45
ideas, there are a lot of concepts, but I don't know that
20:48
we've converged yet. There was a talk by Dee Scully,
20:51
the CEO of Coggle. Love that guy.
20:54
I love. And for anybody that doesn't
20:56
know, he also wrote the most incredible paper when it comes
21:01
to ML OPS in like 2016, which is the high end credit card debt of
21:07
machine learning, which everyone's probably seen it if
21:11
you've ever seen a talk on ML OPS, because it has that diagram
21:14
of all these boxes of things you need to do to put machine
21:19
learning into production. And the model box is just a
21:22
small box out of all the other stuff that you got to think
21:26
about when you're productionizing machine
21:29
learning. But anyway, I digress.
21:30
Keep going. No, that's that's terrific.
21:32
Yeah, yeah, DD is amazing and I, I think the world of him, but
21:38
he, his talk basically made me a paranoic, right?
21:41
Like I was already a paranoic but but he's like at Cagle,
21:45
their main problem is that people find signals that aren't
21:49
actual signals. They're like they leaked into
21:51
the data somehow. Somebody put all the positive
21:55
examples in one folder and all the negative examples in another
21:58
folder. And your model is not predicting
22:00
anything real, it's just predicting what folder it was
22:02
in, right? Which is easy because you have
22:05
the path. And his point was that
22:09
researchers should look at it that same way too.
22:11
Not that like you're intentionally going to hack your
22:14
metrics, but like you're unintentionally going to hack
22:16
yourself. And and so he talked about like,
22:20
OK, so like, what can we even do?
22:21
The problem is when your data is Internet scale, everything is
22:25
already in there, right? Like everything that ever hit
22:27
the Internet is leaked. So you're, you know, you can't
22:30
hold out that that data anymore. And so he talked about new
22:37
benchmarks that are based around some sort of a separation,
22:43
either a physical separation, because it's not on the Internet
22:46
yet. So you talked about for like
22:48
some math benchmark, having mathematicians, like top
22:51
mathematicians in the world, go into the woods, sit in the room
22:54
by themselves, come up with problems that haven't been ever
22:58
with us, and then come back and evaluate models on those.
23:03
And then he talked about like time separation, right?
23:06
There's another one. So if you're trying to predict,
23:08
I don't know, the stock market or something, you can't fully
23:11
evaluate your model until six months later.
23:15
And, you know, unfortunately that's not how it works in our
23:18
stock market predictions that all of those models are like
23:21
they're going to be irrelevant tomorrow.
23:22
So we need to retrain. But, but his point is that we
23:27
need to be a little bit more patient and spend a little bit
23:29
more time generating data for benchmarks because even like,
23:34
you know, we do our best, but all the benchmarks for LLMS,
23:38
like all the data for them is leaked, like all the data for
23:40
them is on the Internet. And so this is how you get to a
23:44
scenario where you're 95% accurate on your benchmark.
23:48
And worthless. And and worthless.
23:49
Yeah. That's.
23:51
Why I appreciate these leaderboards or these benchmarks
23:56
that are like Pro LLM. They that's from the process
23:59
team. They put it together and all of
24:01
the actual data that is created for these benchmarks is all
24:06
closed source. So theoretically the models
24:10
haven't seen them. I can imagine that is only good
24:14
for the first couple go rounds and then the models have seen
24:17
them. So the process team, I think is
24:20
constantly updating that data and it's constantly like
24:25
creating new ones. And so they have different
24:28
types, you know, they've got, I know they have one which is from
24:31
Stack Overflow and that is public data.
24:35
So I can imagine that most of the models have been trained on
24:39
that in some way, shape or form, which is funny because a lot of
24:44
these models still do really bad on it.
24:46
But take that what you will, or take what you will with that.
24:51
I don't. I don't really know how to
24:52
understand it. Well, and, and just to be clear,
24:55
I know lots of folks who train large language models that we
24:58
use every day. And I think all of them are like
25:02
really decent and they try really hard to avoid data that
25:06
they know they're going to be tested on.
25:08
But it's like it's not an easy thing to miss because you can't
25:12
look at every data point that goes into this.
25:15
And when you're just like, just give me the Internet, like, how
25:19
do you want to train this? I don't understand.
25:22
Then there's a lot of stuff that leaks in.
25:24
And so like, I don't think it's an intentional, you know,
25:28
hackery. I I think it just happens by
25:30
virtue of the way that we train these.
25:33
You're going to love this. I saw some kind of a nonprofit
25:37
that was their whole thing was let's make the Internet mediocre
25:43
again so that these this AI is not better than us at doing
25:48
things and we don't, it's get like unreasonable expectations
25:52
put on us because AI can help us do it.
25:55
And So what they've been doing is going around and buying up
26:00
domains in the Common Crawl. And instead of doing like
26:02
nefarious things like I've heard of other people trying to do
26:05
where they will put different poisonous data on these Common
26:09
Crawl websites, they're just putting a bunch of really
26:13
mediocre content that isn't good at all on all of this Common
26:18
Crawl. So it's flooding the OR what
26:22
they're trying to do is flood the LLM training data.
26:24
But at the end of the day, you have to.
26:27
But you have to have so many of these and put so much content
26:32
out there to even make like a little drop in the ocean, right?
26:35
I think the Internet does a pretty good job of making itself
26:37
mediocre without extra help. Without those guys.
26:41
That's a that's a great point. Yeah.
26:44
I mean, so the, The funny thing there is then I know you've
26:49
spent a lot of time training models and thinking about that.
26:53
How do you go about like that data issue and the data quality
26:58
issue? I mean, yeah, it's hard and and
27:02
I'm not sure that I have a silver bullet, right?
27:04
Like often times a lot of the things that I've changed are pre
27:08
LLM. Like I, I still, I worked in
27:10
this industry 10 years before LLMS ever hit.
27:13
So, so most of my experience is still pre LLM And, and even like
27:18
since the LLMS came out, you know, that was training Meta's
27:22
ad prediction model and that's not, not trained on the same
27:26
data. You can be a lot more careful
27:28
there. But you kind of, I don't know.
27:33
I don't know that they're great ways to do this other than you
27:36
do it. And then you look at your
27:37
evaluations and you trust them, which we just talked about how
27:40
you can't. And they guide you into the
27:44
segments where you're maybe underperforming.
27:46
And then you look at what's what's happening with those
27:48
segments and then you train again, right?
27:50
Like the, the beauty and the downside kind of of working in
27:55
ML is that you can always try again and you should always go
28:00
in with an idea of like, I'm, I'm probably not going to think
28:04
through every single contingency.
28:05
I'm probably not going to think through every single data issue.
28:08
When you gain enough experience, you've been burned by enough
28:11
things to where you start checking them.
28:14
But, but like, even for me, like 10 years in, like, I don't think
28:18
that checklist is all that, you know, inclusive of everything
28:21
that could happen. And, and so the, it's an
28:25
iterative process and it's a, it's a process that you learn
28:29
from and you do a little bit better next time.
28:31
Hopefully the person next to you has burned, has been burned on a
28:34
couple of different things. It really does help to work with
28:37
different people. And you know, you gain a lot of
28:40
knowledge that you don't pay for with bringing down production,
28:46
which I have done, I've I've done that, but.
28:50
The battle scars. The battle scars and you know,
28:52
like that, that's good knowledge.
28:54
It's even better knowledge when somebody next to you is like,
28:56
hey, I've brought down production this way before,
28:58
let's not do it again. And and so there's a lot of
29:04
iteration back when I started in the industry, we used to do a
29:07
lot of feature engineering, but the data sets were a lot
29:09
smaller. And so like on a small data set,
29:12
you can start understanding it really well.
29:14
But like, how do you even go about understanding language
29:18
data, right? Like you're, you're going to
29:20
have an embedding and you're going to visualise it.
29:22
But guess what, like how are you going to train that model?
29:24
You're going to train an embedding.
29:25
And, and so like when your tool for understanding the data and
29:30
the tool that you're training are one in the same essentially
29:34
like you just try it. You just try it and see what
29:38
comes out of it. And I know that that that's
29:40
pretty wasteful of the compute. I know that there are
29:42
environmental consequences. I don't tend to think of them
29:45
when I train. I probably should.
29:48
But you know, you're trying to be compute efficient just
29:52
because it's expensive, not because it's environmentally
29:55
friendly. But but Even so, you kind of go
29:58
into it with an expectation that you're not going to be right the
30:01
first time you train it, you're you're going to need to do some
30:03
work on it. And ideally, you know, you know
30:07
all of this, but you start with a small model, you start with
30:11
over fitting on like a small data set to see like even like
30:15
directionally, are you right? And you do early stopping to
30:18
make sure that like you're not training longer than you
30:21
absolutely have to. But even with all those hacks
30:25
like the way that we do machine learning today is just a lot of
30:29
data and a lot of compute. And you do some thinking, but
30:34
the thinking is fairly limited compared to amount of data in
30:37
compute. Yeah.
30:43
Well, we should talk about a lot of this stuff you're doing
30:46
around ML careers and for people that are in the AI world.
30:52
I think that you're doing a huge service to people because of the
30:57
experience that you've had. I know that when we first
30:59
talked, probably back in 2020, you were still at Twitter.
31:03
Since then, you've gone on to work at Meta and then the health
31:06
tech company. But the interesting thing when
31:10
you were at Twitter, I think you just got the job and you were
31:12
like, yeah, I'm thinking about trying to take that next step
31:17
into becoming a staff machine learning engineer.
31:20
And for me, I thought that was fascinating just because it was
31:25
like, oh, OK, and So what do you need to do there?
31:27
And you're like, well, I think I need to give more talks and be
31:31
out in the community more. Turns out that wasn't true.
31:35
That or was it directionally correct?
31:37
Because first of all, you didn't give any talks.
31:39
I kept bugging you. Did you talk in the community?
31:42
And then you're like, yeah, yeah, later, later, later.
31:44
And you ended up getting staff engineer anyway.
31:46
So what happened? Well, now I can give talks as a
31:49
staff engineer. No, I, I think, I think slightly
31:53
different companies do it differently, right.
31:54
And, and so Twitter was particularly quite, they put a
32:03
lot of emphasis on like open sourcing stuff and, and
32:07
publishing within Twitter. Unfortunately, you know that I'm
32:11
sorry, it might be my fault Twitter died while I was there.
32:14
I I think it has something to do with change in ownership, but.
32:18
No, but you were there when Elon took over, no.
32:20
So I left a little bit before I was there for the entire drama
32:24
and I left like a month before because it was pretty clear
32:27
where it was going. So.
32:30
So yeah, Twitter was a terrific place to work, honestly.
32:33
And I don't think it is today, but but it was different
32:39
companies priced different things, right.
32:42
And so there's there's more than one way to staff.
32:45
And at staff, you really start seeing the archetypes and
32:51
different people get staffed for different reasons.
32:53
Some people get to staff because they're really broad.
32:57
Like you're always going to have to be T shaped rather than
33:00
broad. You have to be broad, but like
33:01
really deep in something. But some people like me get the
33:06
staff because we can understand a lot of different things across
33:11
the ML model life cycle. And other people get there
33:15
because they're an ultra specialist on text embeddings
33:20
after it rains and, you know, in the.
33:25
Venn diagram. It's like there's a lot of
33:27
different pieces that they're just in the centre of and
33:30
they're like, wow. Yeah, and, and, and staff
33:34
engineers are leaders. And so like a lot of, a lot of
33:38
what you need to demonstrate is that you can produce at a high
33:43
level over time. And so people get frustrated.
33:46
They're like, oh, I've done this for a year.
33:47
Like when am I going to get staff?
33:48
And I'm like, you know, your time horizon on staff is like
33:51
three years for anything you plan.
33:53
And so just because you could perform at that level for a
33:55
year, like I'm not quite convinced yet sometimes.
33:58
Wow. So, so some of it was just
34:01
timing right for me that I, I still needed to wait.
34:03
And honestly, like, I didn't get staff on Twitter.
34:05
I got staff when I went over to Meta.
34:09
And part of it was, you know, my interview performance and part
34:12
of it was all the experiences that I've had before.
34:16
But I know that like Google, for example, does prize talks and
34:19
being visible in the community in some way for their staff and
34:25
especially senior staff. Everybody I know who's senior
34:28
staff at Google has done at least some conferences said at
34:33
least something that's worthwhile.
34:35
And so, yeah, I mean, I, I think there's the the problem with
34:40
this is that there's more than one path.
34:42
And in ML in particular, I think there are a lot of folks who do
34:47
a really good job like in ML OPS community talking about ML OPS
34:52
there, there's a tonne of good material for software engineers
34:55
on how to like advance through your career, but there's not a
34:59
tonne of material for ML ES. And what material there is
35:03
usually comes from really well meaning folks, but like folks
35:07
who haven't seen the higher ends of the ladder.
35:10
And, and it changes by the time you're here.
35:15
But one of the things I'm working on is next year, I, I
35:20
want to invite a lot of folks who are more senior than me and
35:22
have a podcast with them about specifically about their career.
35:25
How do they, how do they get to where they are?
35:28
And kind of ask them really targeted questions like I'm
35:32
trying to get a former director for Meta that I worked with to
35:38
to come and talk about like, what's it like to be a director?
35:41
Like what, what are the challenges that happened there
35:44
that you don't see until you get to that level, Right.
35:47
And yeah. And, and so like a lot of folks
35:51
who are a lot smarter about it than me and to try to understand
35:56
that area a lot better. In the meantime, I do have a
35:59
small YouTube channel which right now is about helping you
36:02
get the next ML job. I really wanted it to be
36:07
broader. And so I did like a video on
36:08
mentorship and it tanked. Like nobody watched that.
36:12
Like to this day it has like 200 watches and like my other videos
36:15
have like 14,000, right? And that, and what I understood
36:19
was that nobody is talking about ML career path at this level.
36:24
And so people don't think about it until they have to look for a
36:28
job and then they have to look for a job and they're like,
36:30
where do I go? And so, and so I started the
36:35
channel with a little bit more content, with content a little
36:39
bit more heavily leaned toward like, how do you interview?
36:41
How do you get this next job? And it's kind of in my mind
36:47
understood that at some point it'll be a bait and switch where
36:51
I'm going to say, OK, now that you've got the job, here's how
36:54
you actually do it and here's how you get promoted.
36:56
And here's how you make sure you don't ruin the world while while
37:01
you're doing this. So, so that's kind of the
37:04
direction of that, but hopefully it'll be gradual enough to where
37:07
people won't unsubscribe. Yeah, no, they'll go with you on
37:12
the journey because they probably are in the job and
37:14
going uh oh. I only got to find a new YouTube
37:17
to watch videos of who will tell me how to do this.
37:20
I might as well just yeah. Just just keep watching me.
37:23
You don't, you don't need a new YouTube.
37:24
But but there are things about our profession that are
37:27
different, for example, like the shifting Sans that we talked
37:29
about, right? Like this is so my master's
37:31
thesis, it was about computer vision and it was in 20.
37:37
So like I started it in 2011. I finished it in 2013 and it was
37:43
like hard cascades, right? Because nobody has heard of ACNN
37:47
by then. And by the time like I got it
37:49
out in 2013, it was like, dude, like you should have done it
37:52
with CNNS. And I was like, I know I'm, I'm
37:54
not, I'm not going to do another two years though.
37:56
Like I'm done but, but I know I should have done it with CNNS.
38:01
And so it's the only field I know of where like you're
38:04
graduate thesis can be irrelevant by the time you're
38:07
done right hand. And, and like the new kid
38:11
graduating tomorrow probably knows more about transformer
38:15
models than I do having ten years of experience in the
38:17
field, because I have to maintain the models that are
38:20
there today. And they had like all the time
38:24
in the world to go learn about the latest and greatest.
38:27
And so how do you how do you even have a career in that,
38:31
right? Like, how do you stay multiple
38:33
decades doing that kind of thing where Transformers weren't a
38:38
thing before 2017? Like nobody, nobody really
38:42
understood that this was useful. Like a couple of people did, but
38:45
they were crazy. And and it wasn't until you
38:49
could bring a lot of scale to this that you even see this.
38:52
Because I, I saw early papers on attention mechanism and I was
38:55
like, yeah, like, it's interesting.
38:57
Like we have LSTMS, like I'm I'm not sure how this is better than
39:01
an LSTMI literally said that. So please don't take, you know,
39:05
future advice from me, but I I also sold 500 bitcoins at $1.00
39:11
each. Yeah, So.
39:14
Well, I'm glad it still worked out for you.
39:17
Yeah, you've had a pretty successful career.
39:20
You didn't need those. Oh, I.
39:23
Did I did. Stupid taxes, they call it.
39:27
You know, we all got to do that kind of stuff and you're just
39:32
continuing to kill it. So I love seeing that.
39:37
And I love like what you're talking about is very few people
39:42
in the world actually talk about.
39:44
And I think you, you mentioned it to me last time we chatted,
39:47
probably where you were, you were saying the folks that are
39:51
in these positions, they're so busy doing their jobs, they
39:56
usually can't talk about what it means to be in these positions
39:59
and how you get to these positions.
40:02
Like you said, everybody's journey is different into them.
40:05
And then the other thing is 9 times out of 10, when your staff
40:09
MLE, you're at a large company and you've probably got some or
40:14
a lot of Ndas that you sign. So it's not like you can just go
40:18
out there and preach from the rooftops.
40:21
Yeah, yeah. And I think, I don't know if
40:24
it's intentional gatekeeping, but there is a little bit of
40:26
gatekeeping happening as well, which is weird to me, right?
40:29
Because like anybody who came up in this industry, when I came up
40:33
in this industry, part of the fun was that everybody makes
40:36
everything open source and you can go in and you can like, oh,
40:39
by torch. Like I can understand that.
40:40
And Meta still operates similarly.
40:42
Like that's why Llama is there. And it's like, OK, like we still
40:47
can get benefit. We still can get ours, but like
40:50
let's let's make sure that people can use these kinds of
40:52
techniques and like models and whatever.
40:55
But a lot of other companies have started being a lot more
40:57
closed off and you know, notably open AII like.
41:04
The emphasis on the open part there.
41:07
So I think both parts of their name are a lie in fact.
41:13
But you know, closed ML is what they should be closed.
41:20
That should be their real name. The, the other piece that I
41:23
think I wanted to talk about is how, what should you know as a
41:31
staff engineer, like a staff MLE, what are things that you
41:35
deal with and how is that different than the you mentioned
41:39
how big of a gap it is from what is it like L5 to L6 is a really
41:45
hard gap to jump. And first of all, it's probably
41:49
worth just clarifying staff titles are 9 times out of 10 at
41:56
large companies, right? And like tech forward companies,
41:59
I don't see a lot of staff titles at either companies with
42:04
200 people or like enterprises, I don't really see like a bank,
42:10
I don't think I see many staff titles there.
42:12
So maybe you can demystify that part for me too.
42:16
I think in a bank, if you make that much money, you have to be
42:18
AVP. I think they're like legal
42:20
reasons why you have to be AVP. So interesting.
42:24
So that's why they have banks have very weird titles and you
42:29
know, there's title inflation going on to everywhere.
42:32
So like sometimes you look at a small company and you're like,
42:35
oh, that's a principal engineer and they would like convert to
42:37
like a thing type company and they'll be like senior, maybe,
42:41
maybe mid level. And one of my peers at Meta was
42:47
AVP, like an actual VP with like an org at a bank before.
42:53
And I'm like, we're at the same level.
42:55
Like this seems wrong, but but, but that's how it works.
42:59
So, so like when I talk about staff, I talk about specifically
43:02
like tech companies, right? Like and, and specifically
43:06
bigger tech companies, not necessarily just faying like
43:10
Uber, you know, Microsoft, like all of those companies as well.
43:13
But yeah, tech first companies. Yeah, tech first companies.
43:16
I think that's a good way to put it.
43:18
And and I think again, it depends a lot on the archetype.
43:22
My archetype is a technical lead.
43:25
And so I, so for me, what it was is a don't do what I did.
43:33
For me, what it was is by day, you're a technical lead, You're
43:38
basically helping everybody on your team to succeed.
43:42
And at one point I had almost 30 people that I was a tech lead
43:47
for. And at, at those levels that
43:50
that's closer to like a senior staff level.
43:52
But at meta, you're never at the level that you're at, at you're
43:57
usually performing at the next level.
43:59
So everybody is 1 up from what they tell you.
44:03
But but so by day you're doing that and you're making sure that
44:08
there's right scope on the team, that the projects are going in
44:12
the right direction. And it's weird because you
44:15
usually have more information. I've dealt with a lot of like
44:18
really sensitive information, like the way that we're
44:21
approaching a lot of new regulations on privacy and ads.
44:25
Like I had more context on that than a lot of people on my team.
44:29
And so like I had to guide them without like doing things that
44:32
are, it would be illegal that like telling them what they're
44:36
actually working towards sometimes.
44:39
And, and, and so, and so that takes that's a full time job.
44:44
But in addition to that, at Meta you're expected to contribute
44:48
still. And so I had projects that were
44:52
quite, quite a bit more challenging, but maybe a little
44:56
bit less time sensitive. That's the thing that I learned
45:00
at Adobe, because at Adobe I was working on a really big project
45:04
and I was a technical lead on it, coordinated like 80 people
45:07
around it. And, and I being young and
45:11
naive, I was like, I'm going to take the biggest part of this
45:14
and I'm going to implement that. And I became a bottleneck so
45:17
quickly because I, you know, I needed to manage all the pieces
45:21
and I needed to write the biggest piece.
45:23
And so like staff engineers, once you get burned by this a
45:27
couple of times, you start taking on pieces that are a
45:29
little bit on the side or if somebody gets really stuck, you,
45:34
you're there to unstuck them. Often times like 0 to one
45:37
projects, you're like, I'm not sure how this is going to work.
45:40
So like, let me take the first stab at it.
45:42
And then when I'm pretty sure that we're on the right
45:44
direction, I'll hand it off. There are lots of hand offs like
45:47
I admeta you on board onto a new project every couple of months
45:53
because you know you need to move it forward.
45:55
Usually like you come in as a me because something is wrong or
46:00
like nobody knows what they're doing here yet.
46:03
And so you go in and you move it.
46:05
And once it starts moving, the the idea is that your time is
46:09
better spend on another project that's not moving right now.
46:12
So you can hand it off to an E5 and say, I can tell you exactly
46:15
what needs to happen here in the next couple of months so you can
46:19
go get started, but I need to go off to a new project.
46:23
So there's, there's some amount of stress here, you know, and my
46:28
advice for folks, honestly, often times people look at the
46:31
levels and it's like numerical. So it's easy to measure your
46:34
progress. It's like, oh, I'm a six.
46:35
I want to be A7, right? But I'm like, do you do you
46:38
really? Because at bigger tech companies
46:42
that have gotten their compensation straight, which not
46:46
everybody does, but but the companies that have like
46:49
understood what the compensation is, you can make more money as a
46:52
good E5 than a crappy E 6 because wait, how's that work?
46:57
You're going to get better refreshers, you're going to get
46:59
more bonus. And and so like in the long
47:02
term, a good E5 at Meta is going to make more than a crappy E6.
47:06
So if you're going to, if you're going to go to an E6 to be a
47:08
good E6, then yes, but like understand that, that that's a
47:12
significant more, significantly more work and that's
47:16
significantly more responsibility.
47:19
And so when, when people are just like, you know, like I need
47:22
the money or whatever, I'm like, so like go ace your job, like
47:26
you, you will get rewarded for this.
47:29
You will get a better rating, which will equate to more
47:32
refreshers, which will equate to bigger bonus, you know, But
47:37
yeah, don't do it just for the numbers.
47:38
Do it because like whatever the responsibility is of the next
47:41
level is what you actually want to do.
47:43
And for me, it was like for me, you know, I quit my job in tech.
47:47
I'm currently unemployed, sorry, self-employed and and what I'm
47:52
doing now with my time full time as I'm guiding ML engineers
47:56
through their careers. Like I do a lot of one-on-one
47:59
still. And I'm trying to scale that
48:02
through the YouTube channel, the podcast, whatever.
48:04
But that's the kind of work I would do if like nobody was
48:07
paying me, as evidenced by the fact that I'm currently doing it
48:11
with very few people paying me for it.
48:13
So to me, like I had to get there.
48:15
That's, that was the conversation we had in 2020.
48:17
I was like, no, this is you don't understand.
48:18
This is what I want to do, right?
48:21
If that's not what you want to do.
48:22
I, I met a guy at new reps really smart guy has been in the
48:26
industry since the 90s, knows the insides and outsides, works
48:30
on Gemma at Google and he's an E5 and I'm like, do you want to
48:37
move up? He's like, why?
48:39
He's like, I got everything I want.
48:40
Like what, what else? Like what am I going to get as
48:43
an E6 or L6 at Google that I don't get as an L5?
48:47
And I'm like, you're absolutely like, that is a very smart way
48:50
to look at your career because up to E5 you do have to move up.
48:55
So E5 is a senior engineer at these companies and and so like
48:59
senior engineer, somebody who can work independently and you
49:03
can give them a project and say, here's the first couple of
49:05
things, but you go figure out the rest.
49:07
And we're trying to work with colleagues that are like that
49:10
because otherwise we're going to get burnt out and die.
49:14
So up to E5 it is at Meta and Google and all the companies, it
49:19
is really up or out, right. Like if you can't get promoted
49:22
from an E4 to an E5 in a certain amount of time, you're probably
49:26
not going to stay in that company for very long.
49:29
I think you get like 5 halves or something like that.
49:31
So 2 1/2 years you got to, you got to figure out how to be an
49:34
E5. But E52 and E6 is completely
49:36
optional. E62 and E7 is completely
49:39
optional. We had two E eights in the, in
49:43
the meta ads org, which is a huge org that generates all the
49:47
money. We did not have an E9.
49:49
And, and, and by the way, the way that you interview for an E9
49:53
is Mark Zuckerberg picks up the phone and calls you and says,
49:56
Hey, will you come work for us? If you say yes, you've cleared
49:59
the interview, you know, so, so that's how you interview for an
50:03
E9. So like, just just understand
50:05
that like it gets exponentially harder with every level.
50:08
E5 is the expectation. Everything beyond that is like,
50:11
what do you want to do with your career?
50:14
What can you expect from E6 to E7?
50:18
Because you mentioned how you're mentoring all these folks or
50:22
you're, you're leading bigger projects and your help helping
50:26
unstick projects and that's like your sweet spot.
50:30
I didn't understand if that was your specific archetype or if
50:34
that's everyone is expected. To yeah, that's my specific
50:37
archetype. There was another E7 at Meta
50:41
that I worked with quite closely who basically just owned all of
50:44
the revenue. And so whenever there was any
50:48
issue, any like outage or whatever and RJ would come in
50:52
and fix it or like coordinate A-Team to fix it in real time.
50:57
And and so like, that's the amount of responsibility, right,
51:00
right. Like all of revenue, like no
51:04
biggie. Just just a couple of billion
51:06
dollars between friends and we had E sevens who like were
51:13
really deep on what we call signal loss, which is like,
51:17
there's a lot of regulation there.
51:19
There are a lot of like things coming out of Apple, for
51:22
example, right, where they're like, OK, like you can't use our
51:25
data for ads anymore or whatever.
51:28
And so when something like that happens, it's probably an E8
51:30
that starts that. That's like, OK, this is going
51:33
to impact the entire company. Let's understand what this is
51:36
going to be and kind of start setting up initiatives for the
51:39
next 5 years. And then E eights delegate to E
51:42
Sevens. Like, OK, this is a particular
51:44
stream of work we need to get done.
51:46
You go find the teams that you need for this and you go like
51:49
coordinate them however you want.
51:51
But it's basically less and less defined the higher up you go.
51:55
What you do where like an E9 just kind of walks around and
51:59
says, oh, you know what? I bet I can get us 1% more
52:01
revenue tomorrow if I do that. And I talked to the guy that I
52:06
really want to have on my podcast, but hopefully will
52:09
come, but I talked to him about the difference between like an
52:13
E6 and an E9. And then and I was like, you
52:16
know, as an E6, like I got my teams to increase meta revenue
52:20
by 1%. In fact, that was Q2 of like
52:24
20-3 and or Q1 of 23 where our work basically showed up on the
52:31
earnings report and we exceeded by 1%.
52:34
And like my team has increased the revenue by 1%.
52:37
And so I was like, huh, that's interesting, that's us.
52:40
But and his response was like, and E9 does that by themselves.
52:44
So like you do that with a team of people and E9 does that by
52:47
themselves. And I was like, oh OK, but that
52:50
is different so. Yeah, this is all fascinating to
52:57
me, especially because I lived and played in start up life for
53:01
my whole career. And the large enterprise is so
53:06
foreign, especially these tech forward enterprises that are
53:10
very sought after jobs. And there's a lot of people that
53:14
want to be in these jobs because of their, I think mainly because
53:18
of their earning potential. Yeah, the money is incredible.
53:20
Yeah, I don't think. Anybody make as much money as I
53:22
made as in these six at the meta?
53:25
Nobody's like my life. Calling is to optimise the
53:29
privacy ad spins or whatever the hell you were doing when.
53:32
You were there, you could see that, right?
53:33
Like I lasted a little bit over a year, an average 10 years,
53:37
like 2 years. You know, Admeta is a little bit
53:41
higher, but Admeta ads it's a little bit lower.
53:44
So, but, but like it's, it's a lot of churn and burn.
53:49
There's, there's a lot of like, don't get me wrong, like I, I'm
53:53
glad that I had that experience and I think it was a good step
53:57
in my career and I, I'm glad that I did it, But it's, there
54:03
are some people who stay there for a long time, but that's not
54:07
the, the majority opinion. And, and a lot of us just like
54:12
turn through one of those and then go to another big tech
54:15
company and then come back. And then, you know, but, but
54:19
yeah, it's, it's really hard to be in place.
54:22
But I, I imagine in startups it's pretty hard too.
54:24
Like I've, I've worked in startups a little bit and it's
54:28
like a, it's like a different company, right?
54:29
Like when it grows, if it fails, then you have a problem of like,
54:33
I don't want to work for a continuously failing company.
54:36
And if it succeeds, then it becomes a very different
54:38
company. It goes from like 20 people to
54:41
600 people, and suddenly everything is different.
54:44
Yeah, you have HR? Yeah, Yeah.
54:47
You have to navigate politics and you have to navigate the
54:51
company. Whereas before you could just
54:53
say I want to do this like you were saying with the E9 and you
54:57
pretty much have a carte blanche to do it.
54:59
Yeah, Yeah. A lot of distinguished engineers
55:01
actually do come from like early startup engineers, like the
55:07
founding engineer or the CTO of a startup.
55:10
Lots of them often make those like distinguished positions at
55:14
Adobe. About the only way that I know
55:16
to be distinguished was to be acquired.
55:18
And like you were the CTO of the company that was acquired and,
55:22
but they're like they're vanishing a few of them.
55:25
I think at Adobe, we had 20 when I was there, you know, like it's
55:29
under 100 at those levels. But you know, I don't know if
55:34
Guido is even, well, he probably is distinguished, but you know,
55:38
if you write Python like I will you slide?
55:41
I, I think that's worth it. But, but it's like the, the guy
55:45
at Amazon is like the guy who developed Java or something, but
55:48
like you really have to do huge things in order to get to those
55:53
positions. Like you don't apply for them.
55:55
There's no, there's no job opening on Meta website that
56:00
says, you know, come be a distinguished engineer.
56:02
And also you don't need a resume like by then.
56:06
You have to give talks by then. There's only one way forward
56:10
does that. No, you're getting corded to
56:13
give talks. It's one of those type deals.
56:15
Also if you aren't in the E8 or E9 and want to come work at
56:19
meta, you basically just have to call them up like it's not.
56:24
You don't have to wait for a position to be open.
56:26
They will always hire Yeats and nines.
56:29
Anybody who can perform at that level.
56:31
But. Well, sweet man, we're going to
56:33
put all your details on your YouTube channel and all the
56:37
stuff that you're doing into the description so that anyone who
56:41
is trying to go on this journey, they can follow along with you
56:45
and learn a tonne from you. The other thing that I want to
56:47
mention is that you have been super kind in helping us put
56:52
together this asset of what it takes to go from senior to
56:58
staff. And so we're doing that in the
57:01
community as something that, like you said, there's not a lot
57:05
of information out there. So I'm glad that we can work
57:08
together on that and hopefully have something see the light of
57:11
day. It's still early days so you
57:13
never know if it's actually going to happen, but I've got.
57:16
We'll make it happen. We'll make it happen.
57:18
That's what I love and I also look forward to when you have
57:23
the podcast with the ex director that you worked with.
57:27
Maybe we'll just play it on on this stream too, so that the
57:31
folks who are looking for Part 2 and you dangled the carrot, we
57:35
will have it for them too. Excellent.
57:38
Yeah, I'll, I'll definitely, I'll definitely keep keep people
57:41
informed. And I am in the ML Alps
57:43
community Slack too. So if anybody needs to reach to
57:46
me directly, you can. I sometimes take a couple of
57:49
days to reply. Sorry about that, but what
57:52
happens? When you're unworked, what did
57:54
you call it? What was the frame of the the
57:57
change? Of frame point.
57:58
I guess that was that was what I was supposed to say.
58:01
Right, I love that perspective too.
58:03
That is a great one. Yeah, but I, but I mean, it, it
58:07
was, it was a, it was a very intentional move.
58:10
It's not like when I was in nerubs, like there were a bunch
58:13
of companies who were like, Oh yeah, with your experience, we
58:15
really want you. But I'm like, I want to spend
58:18
some time, maybe half a year, maybe a year really focused on
58:24
trying to help people. And maybe that's what I do for
58:26
the rest of time, Maybe not, Maybe I do come back.
58:29
But I do want to see how I can scale the the things that I know
58:34
work right. Like I've, I've worked with
58:37
people one-on-one for a long time.
58:38
I've worked with my teams for a long time and it was really, I
58:42
was helping principal engineer go through the interview process
58:47
and he, he was the first person who like scheduled some insane
58:52
number of sessions with me. Usually people do like 3 or 4
58:55
and then they basically like move on.
58:57
And he's like, no, no, no, I need to pick your brain for a
58:59
lot of things. And then when we were done, he
59:01
was like, wait, why don't you scale this?
59:02
And I'm like, I don't know how he's like, we'll go start like a
59:05
channel or something. And I was like, OK, that's the
59:09
other thing you learn as staff. When a principal tells you to do
59:11
something, you just like, you don't second guess them too
59:13
much. You understand how much smarter
59:15
they are than you. You just go do it.
59:18
So, so that's where that came from is like, people have asked
59:23
me to scale this and I'm trying to, you know, and it's still
59:27
still early days. You'll see if you go check out
59:29
my YouTube channel that they're some bells that are a little bit
59:32
loud and I don't know the last bit of audio editing.
59:36
Demetrios is helping me. So I'll I'll get there.
59:38
But but yeah, definitely excited to be helping so many people.
59:46
And like my, my biggest video right now has like 14,000 views
59:51
and a lot of them are from like unique viewers.
59:53
So let's say 10,000. I don't remember the exact
59:56
number. I'm like, when have I ever
59:57
talked to 10,000 people? Like, yeah, you're making an
1:00:00
impact. Yeah, 10,000 people.
1:00:03
It's a concert. It's a small concert.
1:00:06
I'm not Taylor Swift yet, but it's a small concert.
1:00:10
But, but it's a concert, you know?
1:00:12
And so it really is a great opportunity in our time to be
1:00:18
able to help so many people in such an efficient manner.
1:00:23
So.