Latest / Agentic Conversations (formally mlops.community) / Pioneering AI Models for Regional Languages // Aleksa Gordić // #203

Transcript

Pioneering AI Models for Regional Languages // Aleksa Gordić // #203 · Agentic Conversations (formally mlops.community)

0:00
I'm, I'm Alexa. I used to work that deep mind as
0:03
a research engineer. I'm now building a non-english
0:05
large language models and building a company around that
0:09
and I don't like to record my coffee.
0:11
I'm basically a tea person because I get too hyperactive
0:14
and anxious if I drink a lot of coffee.
0:16
So that's that. All right everyone, Welcome back
0:18
to the ML OPS Community Podcast. I am your host, Dimitrios.
0:22
And today, what a conversation my man brought.
0:25
The heat. I am absolutely jazzed, gassed
0:29
up if you will, after this conversation because at the end
0:32
we talked all about why he is deciding to start something
0:38
right now. Why he decided to leave
0:41
Microsoft and Deep Mind and turn away offers that give him the
0:46
opportunity to make millions of dollars from these big tech
0:50
companies. Since he is a bit of a figure
0:52
out there in the ML research world.
0:56
And he says, you know what, No, I want to do my own thing and I
0:59
think I have got the ability and the know how to make that
1:05
happen. O, It made me remember a quote
1:10
that I just heard, and it inspired me also.
1:14
So I'm going to just say it and I'm going to paraphrase it right
1:17
now. So don't expect perfect accuracy
1:20
on this quote, but it made me think about something that I
1:24
heard, which almost, in one way kind of shatters my soul a
1:28
little bit. On the other way, it is
1:30
incredible and it makes me amped.
1:32
And that is, you're never going to be 17 again.
1:38
You're never going to be 25 again.
1:41
You're never going to be as young as you are right now, so
1:46
it just makes me hear that and think, damn, I gotta take
1:49
advantage of it. Let's go, let's go do it.
1:52
This conversation at the end, it jazzed me up in that way to I am
1:56
stoked. We also got into so many other
1:59
things, one of which being how he's getting all kinds of free
2:02
GPUs. There's a GPU scarcity out there
2:06
right now, and he's getting free GPU's.
2:09
What has he got going on? Also the kind of synchronicities
2:13
and happenstances that have happened since he started
2:19
putting out content on social networks.
2:22
And last but not least, we go into languages and what he's
2:25
working on right now with Hugo GPT.
2:27
Large language model that he's been training and why he feels
2:32
that it is important. Hope you enjoy it just as much
2:37
as I did and let me know if you're jazzed after the ending
2:41
as I was. Am really?
2:52
GPUs. Have you ever thought about
2:56
making a song with those as your bass notes?
2:59
You can just pitch them down. I actually did, I commented.
3:03
I think I tweeted out yesterday that like you imagine
3:06
visualising a training of 10,000 GPUs and visualising the
3:10
gradient flows and then like making a music out of it.
3:13
Like I'm fairly sure there's going to be a tonne of creative
3:16
expressions like in visualising these big systems, these God
3:20
like artefacts. Oh my God, so much.
3:23
And it's cool. It's a little inspiration to
3:26
keep us going, but I get the feeling that when I hear you
3:33
hear GPUs, I'm like, hmm, I wonder if we could just sample
3:36
that and then pitch it down and we use them as some base notes
3:40
and there's our synth that we've got.
3:42
And so I may try and come up with a little song from your GPU
3:48
buzzing, but give us a story. Why do you, why do you have
3:51
these GPU's here and what have you been doing on them?
3:54
Yes. So I've been training a tonne of
3:56
models. I think the first thing that I
3:59
needed for was Whisper, basically the model from Open AI
4:03
to transcribe some of the some of the audio.
4:06
Because I was, I've been building these applications like
4:08
Huberman transcripts. I don't know if you saw that one
4:11
basically the whole podcast from from Huberman and then
4:14
transcribed everything and created like these websites
4:17
where you can just search look through the summaries and stuff
4:20
like so that those the idea but I don't know like also just
4:23
aesthetically I think having GPUs is a thing.
4:26
Like, I like the really as an artefact enough itself, not just
4:30
for its utility, but like, I've been definitely training models
4:34
for various projects I've been doing on these two, yeah.
4:37
All right, so now is the time where we're going to ask the
4:40
listener to place a bet. Does your computer setup consist
4:49
of neon lighting or not? We're just going to ask the
4:52
listener and then we'll have you answer in 321.
4:58
What is it? Does it have some neon lighting
5:00
in there? Is it even the rig if he doesn't
5:03
have neon lighting? I mean by Neil lighting you you
5:05
probably mean RGB. I do have some and it's like
5:09
very configurable, but like I'm on Ubuntu currently and the
5:12
software sucks for configuring everything, but on Windows I can
5:15
literally tweak every single every single library and there
5:18
is a tonne like 4 Rams. I have 120 gigabytes of RAM.
5:22
Like each of them has like RGB lighting, then they have like 6
5:25
fans. All of those who have RGB
5:27
lighting I have like a water cooling for the CPU.
5:29
Those who have lighting as well the the motherboard itself has
5:32
lighting and then two GPUs have some lighting.
5:34
So it's kind of fairly like, yeah.
5:38
If we can get, yeah, maybe take a picture of it and then we'll
5:42
overlay it on the YouTube video because that would be cool to
5:46
look at. And what are the GPUs that
5:48
you've got? I don't think you mentioned it.
5:50
You mentioned to me before we hit record, but.
5:52
Yeah, yeah, I have two RTX 3090. I got one from NVIDIA actually
5:56
as a present well back while I was still in London.
6:00
Yeah, yeah, that's the the, the perk of having some social media
6:03
like presence. So I got the very nice Gu from
6:07
them and then the second one I bought actually here in Serbia
6:09
because they needed more VRAM and it was super cheap like he
6:13
was actually I think around €750 or something.
6:15
So, like he was. It was a fairly it was a seal,
6:18
yeah. Yeah, yeah.
6:20
You. Yeah, so why did you need more
6:24
VRAM? That's a question you don't ask
6:26
a person. In deep learning, you always
6:29
need more GPUs. You always need more VRAM.
6:32
You. Can never have enough.
6:33
Yeah, like first of all I want you to parallelize like whisper
6:37
execution. So So I was running like
6:38
initially just a single model. Then I transitioned to using
6:42
something that's a bit more optimal in terms of inference.
6:44
And so I could stuck like 4 whisper models, a lot of ones
6:48
that's like I don't remember how many billions of parameters it
6:51
has, but it's the largest one before the V3 version.
6:55
And then the second one just added 2X the number of whispers.
6:59
So it was basically to just improve the throughput of of
7:01
like transcriptions. That's how it started there.
7:04
And so this was for these podcast transcriptions.
7:09
How long did it take you to do the whole backlog of Huberman's
7:13
podcast? Because they're like 3 hour long
7:15
podcast. Yeah, actually only for the.
7:18
The thing is like when I left deep mind, we can speak about
7:22
that like I build this MVP of an application that's kind of like
7:26
a Chrome extension that you install when you're watching
7:28
your YouTube videos. And so on the side you can kind
7:30
of communicate with the video. And so for that project I had to
7:33
transcribe I think 40s channels and some of them are much bigger
7:37
than Huberman. Like you might be surprised but
7:39
like I for example the the what's the name of the guy?
7:43
Jocko willing right the Jocko podcast.
7:46
Yeah like tonne of content like I think he has the most content
7:51
out there and it took a while to transcribe all of that but yeah
7:55
and also like stop like Emit open courseware has 8000 plus
7:58
videos. So it was.
8:00
It did require a lot of GPUs. Ohh man.
8:04
So basically start to finish for the IS you can expect what with
8:09
your rig doubling the throughput.
8:12
You've got let's say 500 or 1000 hours.
8:18
You're looking at how long? It was much faster than real
8:23
time, but I would, I would yeah maybe for an hour of content,
8:27
maybe you need a couple of minutes or less, like it's
8:31
fairly fast. Now at this point I'm using this
8:33
library called C Translate which basically hyper optimised memory
8:37
and like speed. It was made by like a single
8:40
developer as these things usually go.
8:42
And that guy ended up in Apple recently so he was a single
8:46
developer building that that was like the fastest team France
8:49
engine out there. It was crazy.
8:52
What else did you do to optimise?
8:55
Umm, multiprocessing Mostly so like just running stuff in
9:00
parallel. Combined with that like not a
9:02
lot. Honestly there was a shitton of
9:04
work to be done on the side of justice like dealing with vector
9:09
databases, doing some postprocessing, putting
9:12
everything in the in the databases and then setting up
9:14
the full the full infrastructure how to serve like the.
9:19
I guess the when the user requests something from YouTube
9:23
and and get and get the served direct response like that whole
9:26
system is going to where the complication lies in I guess
9:29
much more than on the on this side.
9:32
So I got to ask, I imagine you played around with a few
9:35
different vector databases and you had your fun with them.
9:39
Do you have any advice or suggestions for us when we're
9:43
going out there and messing with vector databases?
9:46
Well, I tried to think only two because as it turns out, I knew
9:51
the CEO of V8. He was my friend at this point
9:54
and like I literally just started accusing them.
9:57
They are also open source like the the other one I think I
10:00
tried was Pine Cone. Like it started popular.
10:03
I think if you're on Twitter, in the Twitter space, like people
10:05
mention it a lot, like open source.
10:09
I had friends there already so like they even like I wanna make
10:12
give them a shout out because they they gave me to run
10:15
everything on free. I haven't been paying anything
10:17
anything for V8, so like for me it was like a no brainer.
10:20
Starting to see a pattern here, man.
10:22
Yeah, yeah. Yeah, well, when you know people
10:26
and that's not the whole story by the way, we can go there a
10:29
bit later, but like I got 16A1 hundreds for free as well.
10:32
So like that that's the, that's the, that's.
10:35
The and more coming. Yeah.
10:38
Yeah. No, tell me.
10:39
Yeah, because that sounds a lot bigger than the rig you've got
10:41
right now. Yes.
10:43
And that's where I'm running everything that's serious like
10:45
here locally I'm just running some smaller jobs.
10:47
Currently I'm actually running in the background somewhere in
10:50
the cloud. I think it's like in Dallas, TX
10:52
or something like. So I got them from together.
10:56
AI. I also want to give a shout out
10:58
to them because they've been, they give me amazing support and
11:02
like I mean giving me 1600s to train an open source project
11:05
that was that was amazing. They basically support these
11:07
open source projects and that's kind of trend trust industries
11:10
well, so this this should not come as a surprise if you're
11:12
familiar with some of the work from from U3I folks like
11:17
Stability AI for example has been sponsoring a tonne of
11:19
projects especially if you're reading papers you always see in
11:23
the appendix or like in the acknowledgements section
11:26
sponsored by Stability I. And so like you ask yourself why
11:29
are they doing that? And then the answer is they have
11:31
a shit tonne of computers that they cannot utilise themselves
11:34
and it's a good PR to just like rent it out and like have have
11:37
like it's almost like a you're using that as a way to attract
11:42
good research talent, so potentially hiring good PR like.
11:46
So there are benefits for them. It's not like they're just
11:48
giving out stuff for free, right?
11:50
Yeah, Yeah, 100%. And so you took advantage of it.
11:55
Were there a lot of hoops that you had to jump through?
11:59
Umm, not really. I already knew some people and
12:02
like they they actually gave talks on my Discord, which is
12:05
now becoming like a big thing. Like I literally had amazing
12:09
line up of speakers over the last months.
12:11
Like I don't know, like you probably know, you probably
12:13
don't know Jeeves better. He's the main author of the LE3
12:16
so like the the the recent text image model from opening I/O he
12:20
was there like 3 dog he who is the guy behind flesh attention
12:23
that basically me Transformers go from like context laying felt
12:26
like like 4228 key like just improving the memory and the
12:31
throughput and then like I don't know like Niels Reimers he's the
12:34
director of machine learning and cohere and that's a tonne of
12:37
other interesting people like from elsewhere like so through
12:41
those connections like I I managed to to find out about
12:44
stuff I guess that's it. Yeah, and So what are you asking
12:48
these people when they come through your discord, and what
12:50
are you learning from them? I mean from you learn something
12:55
from from everybody, right? Like especially if they are
12:59
talking about something where they are like an expert in that
13:01
research, you can definitely pick up those types of sites and
13:06
then depending on their position, you can then later
13:10
like use that type of a connection to like you help
13:12
them, you know how it works. I mean it's that type of a
13:14
thing, but it's a good way to for me to learn and just get her
13:20
interesting people and I think it's a no brainer for me that
13:23
kind of replaced my YouTube effort.
13:24
It's like back over the previous years I was more like a guy that
13:28
opens up a paper and then like hour and a half of long of going
13:32
to math formula is going through code That was like a very, very
13:35
niche channel. And that Channel opened up a
13:37
tonne of serendipitous, interesting like opportunities
13:40
for me just because various people were finding out about
13:43
the stuff I'm doing and getting curious.
13:45
So that was also very good effort from my.
13:47
Side yes, we should probably mention that's how I found you.
13:50
I saw that you had a whole section on ML OPS, and so hi, I
13:57
was like, this is awesome. This is great because it's
13:59
basically zero to Hero for ML OPS in like an hour or three
14:04
hours. I can't remember.
14:05
You did a few videos, right? There was like part one, Part 2,
14:08
yeah. I did make a Jack Zero to Hero
14:11
that was a very good and full series.
14:13
The thing with them, all OPS is I started and I kind of ended
14:17
before even starting the series. I didn't really, I didn't really
14:20
like like wrap it up the way I should have because I didn't
14:23
have the time back then. I was literally at the point
14:26
where a tunnel what's going on in the background and I was I
14:29
was living deep mind there was like back I think January
14:32
February period or something. So like unfortunately that that
14:36
series is not complete. You probably saw the first one,
14:38
which is a nice flashy video and then like.
14:41
Yeah, yeah, that's 100%. When I saw it, I thought there
14:44
was a second one too, but I guess I was.
14:46
Expecting more, but yeah. Maybe you said in the first one.
14:50
Alright, this is like the end of the first video.
14:52
In the next video I'm going to talk about this.
14:54
So I just assume that you've made the next video.
14:57
Turns out you didn't. You fooled us all exposed.
15:01
Yeah, exactly. Well, it's probably also worth
15:05
mentioning that you worked at Microsoft.
15:08
You worked at Deep Mind. You were in London for a bit and
15:10
you left Deep Mind. You were making YouTube videos
15:14
and now you're doing your own thing.
15:17
You mentioned how you are getting a bunch of free stuff
15:22
because you are a little bit of an influencer on LinkedIn.
15:24
So if anybody and and Twitter and all these fun social
15:27
networks. So if anybody doesn't follow you
15:29
that is listening, I encourage them to go out and give you a
15:32
follow because what you share is highly valuable.
15:36
And you also mentioned that you run into serendipitous moments
15:42
because of putting yourself out there, whether it's on YouTube
15:47
or on LinkedIn or Twitter or whatever it may be, or your
15:51
Discord community. What are some of these
15:53
serendipitous moments that you've had?
15:56
I really don't know where to start.
15:57
Like, there's been so many interesting things that happened
16:01
as a consequence of me being out there.
16:02
And like, and I guess you also know that like maybe a recent
16:08
thing that happened was that an amazing person that's like ex
16:11
meta person without a tonne of multilingual experience reached
16:14
out to me and that person is now my my cofounder.
16:17
I'm still not gonna release, like, I'm not going to share
16:19
much more than that right now because like everything is still
16:21
in the early days. But like stuff like that or or
16:25
or I don't know Gary Schmidt tech CEO of Google starts
16:28
following your Twitter and like that all those things start
16:30
opening up small doors and then I get the sheet on the offers to
16:34
to work wherever at NVIDIA recently even got an offer from
16:37
opening Eye. Like stuff like that happens
16:38
because you're out there, you're and you're also doing the work
16:41
because there is a lot of influencers nowadays like the
16:44
Thread Boys and not only them which are just talking and they
16:47
are basically just news reporters in the ISPs and for
16:51
some reason they think they're engineers or what not but none
16:55
of them is building. And so I actually figured out
16:58
that even if you have a much smaller audience like even even
17:00
if you have like 2000 or 3000 Twitter, but you're doing cool
17:04
shit in the open source space, you're probably going to get
17:06
much more amazing offers than somebody who has 10s of
17:10
thousands. But it's just like talking
17:11
because then people like Andreessen Horowitz reach out to
17:15
you actually talk with some of the guys who who who came to my
17:18
to my discord. And and he told me that like
17:21
Gary Tan literally DM team the the the white committer like
17:24
president because of some of his posts and he's like even like
17:27
he's not even an influencer and then also like the Amjad like
17:31
the the the CEO DM team to hire him directly.
17:35
So stuff like that happens so to other people as well, to me as
17:38
well. And like, I think that's what's
17:40
amazing because when you're building and being out there,
17:44
that combination is kind of killer combination because most
17:47
people are doing either one or the other right and then you
17:49
miss the benefits. Yes, that is 100% what my
17:54
feeling is to that if you actually have something to say
17:58
because of experience as opposed to just breaking news or.
18:03
Just. In, yeah, or for her been 24
18:08
hours since GBD 4 has been released and we already see it's
18:12
changing the world. Those kind of threads, I
18:14
understand they don't add a lot of value, But the threads where
18:19
you're breaking down some of the math or some of the papers or
18:23
things that are really in depth when it comes to the level of
18:29
sophistication and you're doing it yourself in the background.
18:34
So that's how you have the experience and the know how to
18:36
break it down. I think that gives you a lot
18:38
more credibility and gives. You gives you more integrity and
18:41
like just consistency between what you do and what you say.
18:45
Although I'm not going to bash all of those people.
18:47
Some of them do bring value, but to a completely different
18:50
audience and they know they're going to monetize on the current
18:53
hype in a I so, like, they just want to.
18:55
Which is again understandable. I'm not, gonna, like, say a bad
18:58
thing about somebody because they want to boost up their
19:00
Twitter or LinkedIn followers even.
19:01
But like, I cringe a lot though. Like because when.
19:05
You read my Twitter that way. You cringed a lot when you see
19:08
my. No, I don't.
19:10
Like the like, almost feels like ChatGPT generating some of those
19:14
messages. And that's that's the part where
19:15
I'm like, OK, like, Oh my And then, like, I don't know,
19:33
like that's an interesting thought, no?
19:35
No. No emojis on your tweets
19:37
anymore. No.
19:40
I'm getting, I'm getting allergic reaction to a lot of
19:44
emojis. Like a couple of them is fine,
19:45
but like I'm getting allergic reactions when I see too many.
19:47
But again, that's just my style. I don't want to bash and it's
19:50
funny you say. That too, because I was
19:52
listening to another podcast today and they were saying how
19:56
when you are using like as humans, it's really easy for us
20:01
to sniff out if it was made by Chad, GBT or just any of the GPT
20:06
and. What happens is it almost
20:10
backfires on you because if I'm reading your post and I can tell
20:14
that it's obviously created by one of these models, I lose
20:20
trust in you. And I think, yeah, this is
20:24
great, but like you said, this is the average of humanity.
20:29
If I have a good prompt then I can also spit this out from GT,
20:35
yeah? There's a couple of Contra
20:37
arguments there though. Like first of all people forget
20:40
so you will see somebody and like.
20:43
Yeah, a. Month later, you'll see that
20:45
same person posting something you'll forget.
20:47
That's less. You follow right away.
20:50
Yeah. And and and.
20:51
And and then the second thing or block whatever, I usually don't
20:54
do that or or the second thing is like you have to keep in mind
20:58
that by the very nature of that being average, which is actually
21:02
not, that's a bit of a misrepresentation of that
21:05
output. But that means they're targeting
21:07
the biggest audience And so just because you don't like it nice,
21:11
that means just you're you're not the audience for that
21:13
person. So like there is that there is
21:15
that angle to to it as well and and the thing that they kind of
21:18
hinted is even though I said it's average, it's not actually
21:21
average because even though the the wake of these models like
21:25
trained right like statistically on on the like a whole web
21:28
corpus oftentimes the authors of the paper deliberately inject
21:33
much more scientific like data from archive mathematics all of
21:37
that. And then you additionally have
21:39
instruction tuning which is again very high quality curated
21:42
by much more educated humans. And then so they are definitely
21:46
not average in terms of whatever like intellectually and and then
21:50
there is the, the RHF, the alignment part, it doesn't have
21:54
to be LF buttock. Yeah.
21:55
And so we'll do is actually bring it up much, much more
21:58
above. And we saw recent results, I
22:00
think it was the AGI eval benchmark where GPT 4 basically
22:05
outperforms average human on set tests which are already for our
22:10
population which is selected right.
22:12
It's not that's not the average sample people who do the set.
22:15
So like it's definitely way above the average the output of
22:18
especially GPT 4. But anyway I'm digressing a bit.
22:21
Yeah. I think it's becoming very clear
22:23
that when it comes to exams, a large language model, and
22:28
they're one step away from being able to do any kind of exam
22:33
better than a human if they're not already beating it, right?
22:36
Exactly. I think of my SAT scores and
22:38
wow, they were so bad and I struggled so much in high school
22:43
trying to get those up. And so I'm like the below
22:48
average on that. Well, yeah, I mean it's hard to
22:53
compete with these models. First of all, you probably know
22:56
that like the evaluation of these models should not be
22:58
trusted because oftentimes those set score like tests were
23:02
already in the in the training data and when you train the LLM
23:05
one on something, it's already seen something it can basically
23:08
memorise and just spits out the answer.
23:11
So it's not necessarily the model reasoning of being smarter
23:14
than an average human. So we have to be careful there
23:16
as well. Like that's probably one of the
23:17
hottest topics right now in the in the research community I'm
23:22
among one of the most important things to to kind of tackle is
23:26
better evals. Yeah.
23:27
We've done and evaluation survey with the Milos community and we
23:33
open sourced all the data for everyone to see.
23:35
So anyone who took the survey can see all of the the data of
23:40
the other people that took the survey.
23:41
Or if even if you didn't take the survey, you can see it.
23:44
And what is fascinating is just how nobody really knows what the
23:50
hell is going on with evaluations right now.
23:53
I think that's the biggest takeaway.
23:54
The TLDR is that we're just kind of throwing spaghetti at the
23:58
wall. Yeah.
23:59
And like the the the less paper I saw did not help.
24:03
I think it came from together. I actually where they showed.
24:07
If you take the benchmark, if you take the data from the test
24:10
set so something that you want to evaluate the model on, you
24:13
just reformulate depth depth data using.
24:16
I think they use maybe even GPT 4 O like you just tell the model
24:19
here, rephrase this, and then such reformulated text, you
24:24
cannot now match it to the original data.
24:26
Like if you use something like fuzzy duplication or something,
24:29
that text looks like it's original and then if you train
24:32
on that data, the model gets 100% on the original test.
24:36
And so even though like, so that means basically you cannot even
24:39
catch. There is no way you can catch
24:41
whether you see something in the training data, because it can be
24:44
just slightly reformulated and all of our hard coded heuristics
24:47
cannot catch that. And then you can kind of claim
24:49
leader high scores on an eval even though that's not actually
24:53
the case. And so ultimately what they see
24:56
happening here is, is the only way we can evolve these models
24:59
properly is having crowdsourced ways of arenas where you're
25:03
having multiple models competing and you're greeting the outputs
25:06
and and just having a same with humans.
25:10
You have to kind of build up the trust with the human being.
25:13
It's going to be the same with models like we have to build
25:15
like a societal trust into the model by by, by the very nature
25:19
of the model being out there and interacting with humans.
25:21
That's how I think about the world right now, honestly.
25:24
Yeah, I've seen some companies also putting out new evaluation
25:29
test sets that basically they curate and they have these high
25:34
quality evaluation test sets in something like the financial
25:37
domain or the tax domain. And so since they curated, they
25:44
can kind of assure that these models have not seen this type
25:52
of evaluation test sets before because it's it's been behind
25:56
closed doors and then, yeah, and then you bring your model in and
25:59
you can test against it. I I don't know if that's the
26:03
answer either because it feels like OK that is good for the
26:06
first time. But then what happens once it
26:08
the model seen it, and then maybe you just, yeah, have it,
26:11
it gets out. And then once it's out, you got
26:14
to continue to make new test sets, right?
26:16
And and and. I think that's going to be the
26:18
the kind of the situation and like I'm glad you mentioned that
26:22
because I recently read the Bloomberg PT paper.
26:25
And so Bloomberg basically has obviously a tonne of financial
26:28
data and they trained on that data.
26:30
And so like the thing with their data is it's chronological in
26:32
nature, right. When you have such data set then
26:35
it's easy to decline future samples and Justice like use
26:39
that to evaluate the model. So that's I think much more
26:41
robust because we know we cannot really predict the models using
26:44
using like looms or whatever because there is a tonne of,
26:48
yeah, it's an open system. Exactly, exactly.
26:52
Yo, dude, tell me about the stuff that you're working on
26:54
right now. Because I know that you have
26:56
Hugo GT, which is, and I know you're a huge proponent of the
27:03
languages and making sure that underrepresented languages get
27:07
their shine when it comes to these large language models,
27:09
which I think is awesome. But give me the breakdown.
27:12
Like, how have you been spending your time besides inviting
27:15
famous people on a discord and getting reached out to by
27:18
Andreessen Horowitz and the like?
27:21
Is that because it nicely connects to the previous topic
27:25
of Bloomberg PT and all of that? In a way I'm gonna get there.
27:29
But what? So first of all was Hugo debt.
27:33
It's LLM for HPS languages and those are Serbia and Croatia and
27:38
Bosnia and Montenegro. So that's where I'm from.
27:39
I'm from from Serbia originally and I'm quite involved with
27:42
actually that's the capital of Serbia.
27:44
And so I'm training a first level, the first ever like large
27:48
language model for these languages.
27:50
The reason being is everybody's hyper obsessed with English.
27:53
Like you just see all of the improvements, evals everything
27:56
in English and it's going great. But like there is 196 or
28:02
something countries out there which are maybe well some of
28:06
them are, are using English as the first language, but you know
28:09
what I mean, Like there is a tonne of languages which are
28:11
underserved. There is actually I think over
28:13
8000 or 7000 languages in the world, although realistically
28:18
speaking maybe only 500 ish have like any decent amount of
28:22
information and like some amount of million plus people speaking
28:27
the language. Plus they have like not it's not
28:30
only a verbal language, they also have a written, they have a
28:32
scripture, they can actually log the things they're saying,
28:35
right. And so I'm definitely bullish on
28:37
that because me coming from Serbia and this summer actually
28:42
how this kind of whole thing was triggered, maybe I should start
28:46
there is I started, I I tried to fine tune a lemma model, I think
28:51
it was Lemma 1 began on Serbian. And I realised I cannot even
28:55
translate properly into Serbian because I went to DPL, which are
28:59
the main commercial player basically next to Google
29:02
Translate. And they don't even support any
29:04
of the regional languages here. And they're kind of big player.
29:06
They are already like, I don't know, 300 million new funds or
29:09
something. Yeah.
29:10
They're, they raised a huge round and I also think that
29:15
they're based in Germany or Europe.
29:18
Somewhere in your drinking, Yeah, during.
29:20
Cold. That's like western Germany.
29:21
Not not trying this fun or something.
29:23
Yeah, they're close to me because I'm right outside of
29:26
Frankfurt. And so, yeah, yeah, 100%.
29:28
So anyway, I didn't mean to derail you, but I also will say
29:33
that I'm 100% on board with this because my wife is Basque and
29:39
you may or may not know that the Basque language, because it's,
29:44
yeah, northern Spain and it is nothing like Spanish.
29:48
It's not like Catalan where you can kind of decipher the words.
29:52
If you speak Spanish, it is in a league of its own and there's a
29:58
couple, maybe a million people. I don't know how many people are
30:03
in the Basque Country itself. I get the feeling that it's
30:06
definitely over 1,000,000, maybe 2 million.
30:10
And they have a rich history because there's been a tonne of
30:15
writing that goes back throughout time in Basque.
30:20
But it is a very, very small portion of the world and a very,
30:25
very, very small portion of the population.
30:29
And I see it with my daughter, because my wife, my wife bask,
30:35
right? And so we speak Spanish and we
30:39
speak English to my daughter and we live in Germany and we had to
30:44
make the choice at some point whether or not we wanted to
30:48
speak bask with her. I obviously don't speak Basque,
30:51
but it was at this certain point where it was like, ohh, do we
30:56
speak Basque with her? Do we know it's not that
30:59
utilised around the world? So she's going to be much better
31:03
off learning Spanish. So we opted to speak Spanish to
31:08
her. Now I feel a little bit bad
31:11
because we basically cut that the propagation of that language
31:15
within our family line and so anyway, that's just a little bit
31:19
of musings on the topic. That's a great story.
31:23
This agrees story on. Actually I actually Googled
31:25
where you were speaking like I I see there is 2.1 million of
31:28
There we go. That's the population.
31:30
At least according to Google, that that's still like a
31:33
respectable number. There are much smaller like
31:39
countries and languages. Alright, let's take a quick.
31:41
Break from the episode and talk to you for a second about our
31:45
sponsors of this show, One DB and what they've done with their
31:49
newest LM course. You can elevate your machine
31:52
learning skills with weights and biases.
31:54
Free course on training and fine tuning large language models,
31:58
Elms as they're called out in the wild.
32:01
You can learn from industry experts like Jonathan Frankel
32:05
from good old Mosaic, ML, friend of the Pod, We were Yang and
32:10
Mark Serafin and while you're doing it, learning from them,
32:15
you can deep dive into the LM fundamentals, advanced training
32:21
strategies like Laura and prompt tuning, all while getting your
32:27
hands dirty and having some hands on applications.
32:30
This engaging course offers 37 lessons and over 4 hours of
32:34
videos and a certificate upon completion.
32:37
Enrolled now to master LLM's and advanced techniques.
32:42
Check out the link in the description to start your
32:44
journey and let's get back into the pod and so.
32:47
It is a big problem, yeah. And by the way, like I think
32:52
they are just like a completely different language family.
32:54
They're they're not in the in the exact ones language group.
32:57
So that's that's the reason they're completely alien
33:00
languages. It's a similar thing in Hungary
33:03
they are just north from us and like all of the countries here
33:07
in this region are basically Slavic.
33:08
So they come from the Balto Slavic language group and they
33:12
are like anger or finish. So it's like basically Hungary
33:16
and Finland which are completely like separated by like Germany,
33:20
Austria, sugar like have they have the, the, the common, the
33:24
common like language that they're sharing.
33:26
So that's kind of interesting dynamics.
33:28
But anyway to the default totally.
33:30
Different, totally different and along the lines of that and
33:34
sorry to cut you off again, but no worries because language
33:37
fascinates me. Before I was in tech, I was
33:40
teaching English and I got really into languages.
33:43
I also learned a few languages along the way, and when it comes
33:49
to these different languages, especially with Basque, they
33:53
have cave drawings that date back to before the Roman Empire.
33:58
So it's like there is not a clear definition of which
34:04
language Basque comes from, which I find absolutely
34:08
incredible because some people say, Oh yeah, it's like the
34:11
first language on the European continent and then others say,
34:14
well you know, slow your roll. I don't think it's that much.
34:17
But it is really cool to think about because there isn't a
34:21
language that is similar to it. You know, you can't say ohh, it
34:24
comes from this family tree or ohh, it's a romantic language or
34:28
a Slavic language. It doesn't really have those
34:30
roots. So it kind of puzzles the
34:33
linguists in that regard. Yep.
34:36
Actually have another example, Albanian, which are also
34:39
neighbours here on the in the Balkan region.
34:41
Nobody knows where the language stems from and it seems they are
34:44
the only like language in that language family basically.
34:47
So it's kinda, yeah, unclear. It's so wild how that happens,
34:51
Yeah, I mentioned. The deep L and the.
34:53
The reason was like they don't support Serbian and so I went to
34:55
Google Translate. They translated some of the data
34:58
into Serbian. But it turns out then I realised
35:00
there is 0 ecosystem for Serbian language.
35:02
There are no evals there are no lumps that can pick up like the
35:06
the the last LMLM was like 750 million parameters GPT 2 and
35:13
that's like enamel world that's like ancient history like that
35:17
came out in 2018 or something and and it's it's super
35:21
solvable. Yeah, you can run.
35:23
Literally 7B models right now on your laptop.
35:25
So that that's that's the state of affairs and whereas we are
35:28
running trillion parameter models in the servers and and
35:31
like and. So I realised, OK, there is a
35:33
gap here and also I'm selfish. I want to have something for my
35:37
language. I want to be able to communicate
35:39
with these models. Also you might native language.
35:40
And so I started building various things.
35:43
The first was the the open source replication of the No
35:47
Language Left Behind project by Meta.
35:49
For context. That's a project where they
35:51
basically created a machine translation system for 200
35:54
languages. 202 is the precise number and so for any single
35:58
combination direction, So that means over 40,000 directions is
36:02
theoretically supported. I say theoretically because not
36:05
all of those directions are obviously going to be high
36:07
quality. And then they ended up like they
36:11
said they are open sourcing the models, but they did not and
36:14
people tend to like. It's.
36:16
It's a. Bit and not just men, Well meant
36:18
is actually probably one of the best companies out there in that
36:20
regard. But like people are abusing the
36:23
word open source to mean we released something.
36:26
Releasing is not the same as open sourcing I feel.
36:30
So what they've done is the they released the non commercial
36:33
checkpoints, so you cannot use them actually to do anything.
36:36
So you can just look at them and play a bit with them and that's
36:39
it. That's where the fun stop stops.
36:40
And so my idea was to hear let's build something that can
36:43
actually be used for commercial use cases.
36:45
And that was kind of the whole multilingual journey for me
36:48
started that eventually ended up with me building the huge GT.
36:52
And how many parameters and how did you go about cleaning all
36:58
this data? Did you get other people?
37:00
Are there other researchers in this region that you could rely
37:04
on, or was it you being a one man band?
37:07
Again, Discord here helps. I have, I built the community
37:09
around around this project as well.
37:11
So some people are helping there with data.
37:13
We have like this language Champions Dock where people can
37:16
basically put their name next to some of the 202 languages that
37:19
we want to support. And so it's kind of progressing.
37:22
I did train the HPS model, so that's for the Serbian, Croatia
37:26
and Bosnia languages. I'm currently almost done with
37:29
the Slavic data processing. So I have like maybe 30
37:31
languages, so Russian and Polish, Czech language.
37:36
And so that's kind of he's stalled a bit because I'm now
37:39
focusing mostly on Yugo, GPT, but I'm going to wrap up the
37:43
BLOB project as well. I'm going to release those
37:45
checkpoints and make them commercially available.
37:48
Wait, so explain to me what the difference between the No
37:51
Language Left Behind and the Yugar GBT is?
37:55
Aha. O the first one, the NLRB is
37:59
basically machine translation system, and the second one, the
38:02
Yugo GPT is a large language model which means it's much more
38:05
flexible. It's not used only for
38:07
translation. It can be used for whatever
38:08
summaries for like document retrieval for just like
38:11
generation of text if you want to do creative writing or
38:14
whatnot as you know what elements do in general.
38:17
And like you go it is kind of a meme name because like you
38:22
Slavia was like a thing and we all we all still still yearn for
38:26
that we have this nostalgic sentiment towards that all
38:29
country. And so like it was it was like
38:31
kind of play even though I only included four languages which is
38:34
not the complete because slavery.
38:36
So it might be a controversial mean like I kind of thought that
38:39
the name is kind of cool and it's also semi pun because semi
38:43
pun if that's even a word because there was this very
38:45
famous car called Hugo in like again in the.
38:49
Yeah. Which was one of the worst cars
38:50
in history. But again it's one of those it's
38:53
literally one of the worst cars in history.
38:54
If you Google it right now you're gonna find documentaries
38:57
why people think it's the worst car in history.
38:59
But Despite that people kind of love it And it was like you
39:02
could see it all around the streets when we were growing up
39:05
and so like and yeah. So it's I.
39:09
Definitely have heard of that car, and I know it's become one
39:11
of those. Like there's a cult following
39:13
around it because it's so bad. It's one of those things where,
39:16
wow, this is so bad that we have to love it exactly.
39:20
And there are these memes like you.
39:22
You don't even have to wear like seat belts because, like, men
39:25
drive like Hugos. And like you, you need to die
39:27
like real men without the seat belt in that car.
39:29
Like the car will survive. By the way, you're just gonna
39:31
die. Ohh, that's killer.
39:35
And so now that project's finished and what we're what are
39:37
the parameters on that and how did you go about getting all
39:40
that data so? Do you go GPT eventually running
39:44
as we speak, it's humming on the 16 and one hundreds I mentioned
39:47
before right now. So it's gonna take I think at
39:49
least three or four more days to get through the full epoch,
39:52
which means it won't pass through the full data set.
39:55
And after that I might go multi epoch just to get the lower loss
40:00
i.e. better quality. And that's going to be the base
40:02
model. So it's still work in progress.
40:05
Actually I'm I'm planning on doing the instruction tuning
40:08
afterwards as well, and I'll probably not do alignment at
40:12
this. This stage because like
40:14
instruction tuning does the alignment in a way, it's just
40:17
not as good as when you add URL component on top of it, but it's
40:20
going to be good enough. And do you see yourself just
40:24
having this? Are you going to host an API and
40:27
let people play with it or you just going to put it on hugging
40:29
face and let people go about using it?
40:33
What do you see this like the future?
40:36
This one is going to be Apache two, it's going to be open
40:38
source model. I'm going to release this one,
40:40
but it's not going to be nowhere near as near good as I as I can
40:43
make it. So there is a tonne of ideas I
40:46
have like on the backlog that I have to implement like better
40:49
tokenizer because I'm currently using Llama tokenizer which
40:52
makes Serbian very verbose and so it's much harder to training
40:55
and inference when you're using these suboptimal tokenizers.
40:59
Tonne more ideas of how I can make it better, so like my D is
41:02
to basically our open source is Checkpoint and then help
41:06
businesses in the region actually run this and like make
41:10
this like a business around it. So that's kind of what I'm
41:13
thinking about, this multilingual space.
41:15
Yeah, it feels like there's a lot of opportunity right now for
41:21
the languages that are not as supported for anything, whether
41:25
it's voice or text, like large language models or text to
41:30
speech or speech to text. All of that feels very
41:35
unsupported. You have Whisper and it does
41:37
have a bit of support, but I imagine when you really start
41:42
digging in, it's just like what you found with the LM's.
41:45
It's OK. They're not that good.
41:47
And like you said, Speaking of one of the telltale giveaways
41:51
when you're using large language models is all of a sudden your
41:56
LinkedIn post or whatever you're doing get that you want.
42:00
The output is very verbose. And so if you're talking about
42:03
this tokenizer making it more verbose, it's like interesting
42:07
that that is something that you're you're focusing on.
42:10
But I went a little bit what what a little bit off the rails
42:14
there. But I do think that there is a
42:19
lot to be said for these languages that are not supported
42:23
as commonly 100. Percent, and this is just a
42:27
quick remark on the verbosity, it's more about the tokenizer.
42:32
So basically what the LM sees, he doesn't see the text, it sees
42:36
these tokens. And so the only thing that's
42:38
more verbose is going to be for the same sentence in English and
42:42
in Serbian, the the the number of tokens that the model sees is
42:46
going to be much bigger. So that's the verbose part.
42:48
But when the model outputs the text that you actually as a
42:50
human being read, that's going to be but like as verbose as
42:54
English. So it's gonna be more verbose in
42:56
that sense. So like I am maybe lower quality
42:58
because of that, just like a yeah.
43:01
A comment on on that part. Awesome.
43:04
Yeah, really cool. And so you are doing all this
43:12
focusing a lot on the different pieces of languages, large
43:18
language models. But I know that you also have
43:21
done some cool stuff with vision and image models and but have
43:27
you done something with a diffusion model maybe?
43:29
What? What?
43:30
Yeah. I I.
43:31
Did I did play with that as well?
43:33
There was maybe last year. I also did recently.
43:35
I made this Jarvis for for, for images.
43:38
You're probably referring to that one.
43:39
Yeah. And so yeah, there I basically
43:43
made a local app that you can run on your system and like all
43:47
the models are open source. I'm using 0 APIs running
43:51
everything locally. And like, you can do very cool
43:53
stuff, like you're literally communicating with voice in real
43:56
time and then you say, yeah, looking for some image and the
44:00
model basically goes through your directory and finds that
44:02
image. Or like images that have that
44:05
type of a feature. For example, I want burgers.
44:07
And then you see, hey, I want you to pick, I want you to
44:10
select maybe 6th image or whatnot.
44:13
And then the mullet selects the 60 image and then you can
44:16
manipulate the image. Then I say something like can
44:18
you segment the person from the image?
44:20
Can you segment the clouds? And then I say, can you repeat
44:23
something instead of that. And I'm doing all of that
44:25
locally through voice and manipulating the images and
44:28
searching the images and all that.
44:30
So it's kind of wild that we can do that in 2023, honestly,
44:34
because only a couple of years ago it was science fiction,
44:37
yeah. Yeah, exactly.
44:39
There is something that I want to pick your brain about when it
44:42
comes to images and image generation and basically the
44:46
generative AI with these image generation models.
44:52
One thing that I had a buddy on here, Michael Eric, and he was
44:57
talking about how he's trying to build the business off the back
45:00
of an image generation. And what he feels like is that
45:06
it had its moment and that moment may have been probably
45:10
like end of last year, beginning of this year.
45:13
And a lot of people came into the space, there was a lot of
45:15
energy and a lot of action around it.
45:17
And then you had the Chad GBT moment that blew up and people
45:22
shifted their attention and they went and started building with
45:26
that. And so it felt like there was a
45:30
lot of building blocks and a lot of things that were left open
45:35
and have not been properly optimise for.
45:40
If you're trying to build a business with these image
45:43
generation tools or or do anything in that space for that
45:48
matter, did you feel, do you feel that way?
45:52
Have you, I mean, maybe you are a perfect example of that
45:55
because you played with Jarvis for for images and then you said
45:59
OK, cool, this is great. Now I'm going to move on to Hugo
46:02
GT, yeah. So for for me this was more of a
46:06
fun project while I was reading for Compute.
46:08
That's actually the story. So, So like there is definitely
46:12
a tonne of interesting opportunities all across the
46:14
spectrum across all of the modalities.
46:17
Any combination of those modalities.
46:19
When they say modality, I mean text, image, video, audio or
46:22
whatnot. And so there there is a tonne of
46:25
opportunities, but I think like vision, especially text to image
46:30
and even text to video right now like might be one of the most
46:33
saturated markets and also in house.
46:37
So what do you mean well by that?
46:40
They. Mean like images and image, if
46:42
you manage to generate an image and like anybody in the world
46:46
can can parse that and enjoy that image.
46:48
Whereas with languages there's going to be much more
46:50
separation. So winners in the English space
46:52
doesn't mean that those are going to be the winners in the
46:55
like I don't know like in Vietnam or winners in like Kenya
46:59
or whatever. So like the the there is I think
47:02
much more market opportunities for these because it's
47:05
segregated language. Language is kind of he has, it's
47:09
a bigger space and also more open-ended And I think it's
47:11
going to be I've done more improvements that that have to
47:15
be done and will be kind of implemented over the next years.
47:19
Whereas images are already kind of amazing.
47:21
Like I'm not saying it's going to be a tonne of flexibility and
47:25
steerability that you'll get from these tools.
47:28
In the sense if you saw the like the the, the the dragon or
47:31
basically put the dot on the face of somebody and you can
47:34
basically just move the dot and turn the the person to you.
47:38
So exactly. There is a tonne of those things
47:41
that can be done and improved in in the in image with images.
47:44
So, aside from working on some of these languages, these
47:48
underrepresented languages, what else are you excited about in
47:53
this space? And what are some hard
47:55
challenges that you think are worth tackling these days?
48:01
I mean, I'm excited about I'm starting with the Balkan region
48:04
just because I can easily validate the outputs from these
48:07
models because I speak all of these languages, they are
48:09
basically dialects. So I can call that's again a
48:11
local meme like everybody's hyperbolic about here because
48:14
everybody speaks 5 languages. As soon as you're board they're
48:17
pretty much the same language And and so I don't know after
48:20
the Balkan region I see a tonne of opportunities different
48:23
markets where where I'm thinking of of tackling those those
48:26
markets as as it when it comes to technical challenges there is
48:30
a tonne because obviously there is not enough tokens in most of
48:34
these languages to train like a large language model is gonna be
48:38
7 billion or more parameters. Just if you take a look at the
48:41
Chinchilla scaling laws which was a paper that came out
48:44
actually from deep mind like last year maybe March and that
48:49
that law basically tells you, I mean law quote unquote it's
48:52
about hard physical law but it tells you how many tokens you
48:56
need to have per parameter and that number turns out to be
48:58
maybe 20 tokens per parameter. So you can then count easily how
49:02
many tokens you need in a language for a given size of the
49:05
model. And that's kind of a lower bond
49:07
because we we are now people are now pushing into this data
49:10
saturated regime. So like, I don't know, like
49:12
Lemma one was maybe 140 tokens per parameter, so it's like 7X
49:16
they're watching Chula propose. So it's like there's a TLDR for
49:19
saying you need a sheet of data that most of these languages
49:22
don't have. And so you have to be creative
49:24
about how do you take something that's a powerful reasoning
49:27
engine, maybe in English, and do the so-called language
49:30
adaptation by taking a smaller subset of tokens and teaching
49:34
the that model how to speak that language while preserving the
49:38
original reasoning capabilities, mathematics, knowledge about the
49:41
facts about the world and just teaching it how to see that same
49:45
thing in a different language. That's kind of the main
49:47
challenge. And maybe a quick plug here.
49:50
I'm definitely looking for a technical cofounder, someone who
49:53
who like, who knows how to build these systems and who has
49:57
ideally like a like a publishing, like a track record
49:59
already in this piece building large scale systems.
50:03
So, so yeah, yeah. Doing what?
50:05
Like hosting systems or building them training.
50:10
Training your systems. So like somebody who knows, ask
50:14
marine Elms at scale, who knows how to deal with all the issues
50:17
that come up with scale and ideally some experience with
50:21
multilingual space. Understanding the difficulties,
50:24
you end up with collecting data in different languages.
50:27
All that's kind of bonus type of an experience.
50:30
Well, yeah. And it feels like for me, the
50:34
pieces where you start to get to languages that are not with the
50:41
alphabets that we're used to, that is a fascinating problem
50:45
too. I imagine that just adds a nice
50:47
little spice to the mix, even if there are the tokens.
50:51
I understand that there's probably a tonne of tokens out
50:54
there in Japanese per se, but it is.
50:58
It's got to be a bit different because of the characters.
51:03
Exactly. And that is the case, it depends
51:05
how you train your base model. O for example, Lama deliberately
51:10
like used mostly English or code whatnot, but like mostly English
51:14
when it comes to languages. And so when you attempt to
51:16
tokenize, some of these may be Easter, like Eastern Asia
51:20
language, like trying some of the Chinese languages like
51:22
Mandarin Chinese or whatnot. Like, are you going to end up
51:25
either having these so-called ank tokens, the unknown tokens,
51:28
or just like a tonne of verbosity And like it's not
51:32
going to end up learning well, so you have to train the
51:35
tokenizer from scratch. And some of the models, like
51:38
there's been a tonne of work in this space as well, like balloon
51:40
being one of the prominent examples where this was from.
51:43
The big science project where they built this 176 billion
51:47
parameter model for a tonne of languages and its underlying
51:50
tokenizer had like a vocabulary of 250,000 tokens.
51:55
And so it covered both Chinese scripture as well as Japanese
51:58
kanji. And like Korean and like, I also
52:02
like Western languages quote, unquote and so like, yeah.
52:08
Wait, so you are obviously showing that you know how to
52:15
build these large language models you're you're training?
52:18
You go GBT right now. Why do you feel like you need a
52:22
cofounder that has these experiences also or has this
52:26
skill? Because like obviously when
52:30
starting a company you you have like I'm a very generalised
52:33
person like so I know a lot of stuff that's not necessarily
52:36
just building models and ultimately I would likely end up
52:39
not having the bandwidth to completely train these models.
52:41
That's one thing. Plus I don't have a publishing
52:44
record in this piece. So there there is definitely
52:46
100% better people than I am in this particular niche of just
52:50
like building these systems because that's that's something
52:52
that they've maybe done a PhD in and then they've been doing only
52:55
that for years. Whereas like I started my
52:57
journey kind of started with phonics, then they piloted into
53:00
software engineering at Microsoft and machine learning.
53:03
Then I was like oh run machine learning, computer vision and
53:05
then LPL like everything. And then I kind of converged
53:10
over the last couple of years towards the more towards an LP
53:13
space and maybe like multi multi model.
53:15
But you get what I'm saying, like I I've been all around the
53:17
place and while I've been exploring and doing this breadth
53:20
first search somebody has been doing only this.
53:23
So like, I definitely looking for for such a person who like
53:25
has both the entrepreneurial capabilities and aspirations and
53:29
also he knows how to build these systems at scale and that field
53:32
that's like the only thing they've been doing over the last
53:34
years or whatnot. That hyperfocused.
53:36
I have a lot of respect for people that can stay
53:40
hyperfocused because I'm not that person.
53:42
I, as you can see, I got a guitar in the background.
53:44
I've also got a piano around. I like talking to people on
53:48
podcasts. I like learning about so many
53:50
different things, and my hat is tipped to the people that say,
53:55
you know what? I really like doing this one
53:58
thing, and I'm going to go as deep as possible on this one
54:01
thing because they become obsessed in a way, Yeah.
54:04
But I I think you might have a wrong assumption there.
54:07
Like some people are just it's natural for them not to be
54:10
brought. So for me for example, I also
54:12
like I played guitar for three years in a school and then I was
54:15
doing electric guitar for a couple of years.
54:16
And like I also speak, I speak 5 languages with varying degrees
54:20
of like proficiency and so like I've been doing the thermal
54:23
stuff and also like callisthenics training 6 martial
54:26
arts, I've been around the. Place I've been all over the.
54:29
Place as well. Like that was my childhood.
54:31
But like I kind of grew up coming to like faculty and being
54:34
OK. I've done a lot of cool stuff.
54:37
But like while I was doing all of this, somebody has been
54:40
programming and doing competitive programming since
54:43
they were 7th grade. And like I'm missing out.
54:46
Like I'm falling behind and then I get to catch up and then
54:49
slowly I realise The thing is like you have to focus because
54:55
even like because like even if you're just doing languages, if
55:00
you want to be like a polyglot who speaks plenty of languages,
55:02
if you don't use two of those languages for like 3 years for
55:05
whatever reason, it's already weaker.
55:06
It's not going to disappear. But then you don't feel as good
55:09
because like things are constantly just like being eaten
55:12
by the by the by the entropy. And so I feel that just being
55:16
hyper focused on something now in my part in this point of my
55:19
life is very important because otherwise I feel I can never
55:22
get, I can never become the best at something unless I
55:27
artificially focus myself. Because naturally as I said
55:30
previously, like I'm really if you just gave me the freedom to
55:34
do whatever I want and I was not obsessed about making something
55:37
and succeeding as one thing, I will be doing everything.
55:40
I literally mean that like I will be.
55:42
I can imagine myself in parallel life being a travel Blogger,
55:46
just eating food, going to hotels, like exploring the
55:49
world, learning languages completely different from what
55:51
I'm currently like this hyper focused tech guy who is just
55:55
like reading papers, building stuff or sitting at the computer
55:58
for like 16 hours a day. So he was a hard decision for me
56:03
in particular as well, well. Talk to me about your decision
56:07
to leave Deep Mind and especially you're going to be
56:13
looking for a cofounder. I you mentioned before that you
56:17
already have one other cofounder and you're looking for another
56:20
cofounder that is a bit more hyper focused on this.
56:23
Yeah, that person presumably is getting those offers that you
56:29
were talking about from Open AI and Google and Meta and all that
56:33
fun stuff. Why would someone want to pass
56:38
up on the big payday of working for one of these big tech
56:43
companies and join a startup? That's a good question and I get
56:50
the TLDR will be I am also refusing some of those offers.
56:53
I'm also getting offers from from all of those companies and
56:56
like. So there's something that's
56:57
almost like. What what I want to hear about
56:59
is like, what is your rationale? Because I know that you had to
57:03
have strong conviction in yourself and what your next step
57:06
is to say, OK, cool. I've done the Microsoft thing,
57:10
I've done the deep mind thing and now I want to do my thing.
57:16
Yeah, well, I I think the answer, however you want to put
57:21
it, well, first of all, I'm very entrepreneurial.
57:22
So for me this was a natural path Before I even started
57:25
working at Microsoft. I knew I'm going to leave after
57:27
two years. It turned out to be a bit longer
57:29
because like Pandemic struck and so things kind of got out of out
57:33
of schedule and similar thing for deep mind.
57:36
And so for me, it was natural that I knew I wanted to have
57:39
this high agency role where I'm just building this stuff I want
57:42
and I'm leveraging all of my skills, which were not always
57:45
used inside of corporations. For example, Like thanks to some
57:50
of the blogs and stuff I I put out there for landing a job at
57:54
Deep Mind. I literally had over 20 new
57:57
joiners at Deep Mind like reaching out and thanking me for
58:00
like helping them land the job and like I get zero financial
58:03
gain for for debt. Like they they didn't do give me
58:07
any type of award because that's not part of my role as a cog in
58:10
the system. And so I'm thinking, man, if I
58:13
was in a syrup and if I like, literally acquired 20 smart
58:17
people, like I would be, I would be like, I don't know, like I
58:21
would be like, so much richer already.
58:23
So like travel. Point period.
58:26
Mean retirement. Really.
58:28
Also financially also like the expected value of me pursuing a
58:32
startup problem right now is so much higher than me getting
58:35
whatever at any of these companies even if the
58:38
composition is like million or I know this first camp like for
58:41
open eyes some of them literally got 10 million packages.
58:43
But even for those types of like if you if you if you know you're
58:47
like outlier in terms of determination in terms of like
58:51
skills whatnot, like your expected value is much more than
58:54
1,000,000. Like I if I'm just shooting for
58:57
one million I can go and enjoy the comfortable life with
59:00
corporation and climbing up my letters and doing the low agency
59:02
jobs because I honestly think those of all of those are are
59:05
low agency like jobs. You cannot do whatever you want.
59:08
You're still an employee under some manager.
59:10
You still have to restrict yourself to do something.
59:12
When you build your own thing then you can.
59:14
If you are the founder, you have much more freedom.
59:17
Maybe at one point investors and board members start like
59:21
constraining you somewhat, but still it's much higher agency
59:24
than being like a L4L5 engineer at Google or Meta or whatnot.
59:30
Facts that is you. Simply it's simple.
59:34
I honestly it's simple like I like if you're optimising for
59:37
wealth the riskiest thing you can do right now is going to
59:40
save route of being in back tech.
59:42
Like worst thing that can happen in two years time I raised 50
59:46
million. I like built A-Team from scratch
59:49
like I oversaw an organisation. I built it up, I failed and like
59:53
I'm going to get fired as an L7 like at Met or NVIDIA like you
59:57
know what I mean? And I don't think that's going
59:59
to be the outcome because I don't, I don't plan on.
1:00:01
On coming to big, big corporations.
1:00:03
But like that that's kind of the worst thing that can happen
1:00:05
financially for me. And so I don't, I don't see from
1:00:08
that standpoint, I really don't see any any downsides.
1:00:10
Like I only see me missing out now when I'm 29 and not pursuing
1:00:14
something. That's crazy and long sighted,
1:00:18
yeah. Dude, I love it.
1:00:20
You inspired me, man. I'm like, fuck, let's go get it,
1:00:23
baby. Let's do it.
1:00:25
Exactly. No, not that I have any offer
1:00:29
from Meta or Google hanging around in my inbox.
1:00:32
But even if I did, I'd say no after listening to that.
1:00:36
I mean the the The thing is there is just so so many
1:00:40
misconceptions about the salaries in big tech.
1:00:43
Like you hear somebody having like a somebody's in Bay Area
1:00:46
and they have like 700,000 compensation.
1:00:50
Nice. OK.
1:00:50
And then subtract the taxes and then you end up with like half
1:00:54
of that or something and then you're like OK, 350, OK.
1:00:56
And then you have to pay for like a monthly rent of like if
1:00:59
you have kids, you're going to be paying 4000 or something and
1:01:02
then you have kids you have to pay for medical or whatnot.
1:01:06
You you end up with saving me he, I don't know if they save
1:01:09
200K or 100K of all year, that's great And so and so like it's
1:01:14
not that great when actually subtract the costs and if you
1:01:16
have children, like you're actually not that well off And
1:01:19
so I don't know it sounds much better when you hear 700,000 but
1:01:24
like I don't know it's still kind of low league when it comes
1:01:27
to potential outcomes financially.
1:01:31
Yeah, after you do all the math, if you are forced to be in San
1:01:36
Francisco, of course, if you have the luck and you say,
1:01:39
alright, I have that pay, but I'm in somewhere like Thailand,
1:01:44
then you're living large. But I don't know many researcher
1:01:49
jobs that are fully remote, 100% remote where you can hang out in
1:01:53
Thailand and be training the models across the pond.
1:01:57
And also if you end up in that type of a deal, you're almost
1:02:01
certainly going to get lower lower wage almost certainly
1:02:05
unless you negotiate something a special deal.
1:02:07
For example I have a friend here from Serbia who literally has
1:02:10
San Francisco salary in Belgrade because he was part of the
1:02:13
founding he he helped found the local office.
1:02:16
And so those people who like felt open up a branch of a
1:02:19
company, some other country can end up having the the the five
1:02:22
seller in the sense of geographical, in the sense they
1:02:25
are receiving Bay Area celery in some export of the world.
1:02:29
So that's kind of cool deal. But again it just depends what
1:02:31
are you aiming for because it's still just a salary you're still
1:02:35
no matter what you do you can do some crazy shit.
1:02:38
You can optimise like I don't know, you can save Google 5%
1:02:41
storage like across the whole planet which is like amazing
1:02:45
contribution and you'll maybe get promoted 2 levels whatever
1:02:48
like who cares like you you got you got 2 levels for that.
1:02:51
If you if you had your own business you would now be like a
1:02:54
multi deck, a millionaire or like whatever like so.
1:02:56
So I don't know it's dampening your your outcomes when you're
1:03:00
in a corporation. Dude, I love it.
1:03:03
I think this is a perfect place to end it because I'm gonna take
1:03:05
this inspiration and I'm gonna go and do some shit.
1:03:09
I don't know what I'm planning on doing for the rest of the
1:03:11
day, but whatever I'm going to be doing, I'm going to be doing
1:03:13
it with vigour, that is for sure.
1:03:15
I'm. Gonna be playing guitar like
1:03:17
Hendrix by the end of the evening, like.
1:03:20
There ain't no more fucking around, baby.
1:03:22
I am doing this. This is I I just got amped.
1:03:25
Let's go. All right.
1:03:26
Well, this has been an absolute pleasure.
1:03:30
I'm super excited for what you are doing and for anyone I said
1:03:35
it before, but for anyone that wants to follow along on your
1:03:37
journey, follow you on Twitter and LinkedIn, it's great to have
1:03:41
you pop up on my feed. And also I would also say join
1:03:45
your Discord if they want to hear some of the great speakers
1:03:50
that you get on there. And yeah, man, thanks.
1:03:54
Thanks. A.
1:03:54
Lot Demetrius. I really, really enjoyed this
1:03:57
one. Oops.
1:04:00
You have to immerse yourself in the ML OPS content.
1:04:05
The best way to do it is to subscribe to the E-mail OPS
1:04:09
Community Podcast, so good luck and keep learning.