Latest / Agentic Conversations (formally mlops.community) / Building the Future of AI in Software Development // Varun Mohan // #195

Transcript

Building the Future of AI in Software Development // Varun Mohan // #195 · Agentic Conversations (formally mlops.community)

0:00
Hey, I'm Varun, CEO and Co founder of EXA function that
0:03
builds the product codium, which we'll be talking about.
0:06
And I like my coffee, as as a latte, I'm pretty basic.
0:11
I like just a lot of milk in my coffee.
0:15
Welcome back to the ML OPS Community Podcast.
0:17
I am your host Dimitri OS and today was a wonderful
0:22
conversation with Varun, all about how he pivoted his company
0:26
from doing virtualized GPUs into a code pilot that is killing it
0:34
that he's got over. I think he said 400,000 users
0:38
when we talked to him and we really went deep into what some
0:42
of the issues were when it comes to scaling, why he chose to
0:47
pivot the company and how they built out their their whole deep
0:53
learning step and they really optimised from everything from
0:59
the training to the inference. They built their own models.
1:02
So it's not running on GPT 4 or GPT 3 or any of that.
1:07
They made sure that the main product is free.
1:13
They made sure that they are able to go and bring this tool
1:18
into customer's own infrastructure and it only
1:22
requires one GPU, which for me was mind blowing.
1:26
But then also he talked about how when he was building the
1:32
tool that was the GPU VM, basically the virtualization GPU
1:40
before they pivoted, how he had to take this long hard, cold
1:45
look in the mirror and say making progress, are we solving
1:52
a pain that is painful enough for us to scale to 100 million
1:58
in revenue. They had great revenue and it's
2:00
got to be hard to have incredible revenue and say you
2:05
know what, we're going to change the whole direction of the
2:08
company and it's obviously worked out for them quite well.
2:12
I'm very happy that he was so transparent about how things
2:17
were and how that conversation was with himself, basically.
2:22
And then how he looks at things moving forward and what he wants
2:27
to focus on and how he evaluates if his product is actually
2:32
useful, how he evaluates, if the large language model that they
2:36
built is actually useful, how they're monitoring the use of
2:40
the product. All of this and more in the next
2:46
50 minutes, so hopefully you enjoy.
2:49
And if you can do one thing and one thing only, you would mean
2:52
the world to me to share this with one friend that you think
2:56
will find it useful. And without further ado, let's
2:59
get into the conversation. But before we do, let's hear a
3:03
word from our sponsors. All right, listeners, I have the
3:06
pleasure of announcing right now with you that the Emelops
3:10
community and Quantum Black, the AI branch of McKinsey and
3:15
Company, have partnered up. And for this special
3:17
partnership, I brought on Nair Khan.
3:20
Your official title is partner, but your unofficial title is
3:24
Professional Data Nerd Nayer. What exactly is Quantum Black?
3:28
That was a good. Question really, really great to
3:30
to be here. Let me let me start briefly and
3:33
tell you our origins. We started off in Formula One.
3:36
We realised early on that we could use data in AI in very
3:39
smart ways, off the track, have big impact on track times.
3:42
And as I know we took that mentality of using data in AI in
3:46
very smart ways to help companies and industries on
3:49
their toughest AI use cases really using AI to drive impact.
3:52
And we've done this across a whole myriad of different
3:54
industries. I've mentioned this formula,
3:56
Island Sports, but there's also natural resources, health and
3:59
life sciences, consumer and so on and so forth.
4:02
We're about 2000 colleagues globally in 40 different
4:05
countries and we have a whole host of different types of
4:08
skills and technologies, data science, data engineering as you
4:11
know, but also DevOps, software engineers, product design, SRE
4:15
cloud. So we bring and fuse all these
4:17
different types of talent. I had the pleasure of going and
4:21
meeting you in person about a year and a half ago and I went
4:25
to the Quantum Black office in London and I loved it there
4:28
because you set up a nice little Lego welcoming Matt for me.
4:32
That said Emma Lobs and we also had some whiteboarding sessions
4:36
and you blew my mind with some of the projects that you are
4:38
working on internally. Can you tell us a little bit
4:41
about what makes Quantum Black special and that vibe that I
4:44
felt when I was there? Though I think you know we all
4:46
part of McKinsey and company, but one of the things that we've
4:48
maintained as QB is trying to keep our culture and there's two
4:52
different things. It's are always like learning
4:54
type of mindset and being very curious about the problem but
4:57
also curious about the solution. There's a lot of great
5:00
technology and solutions out there, but there's also a lot of
5:02
white space that hasn't been sold and I think our teams
5:06
really get down and they're really happy to try to drive
5:08
through solutions with that. The 2nd is this fun aspect you
5:12
mentioned Lego and Lego is part of our DNA and really doing
5:15
things in new and creative ways. I think this is really not only
5:18
with our the organisation we work with is very energising,
5:21
but also our people, they really, really dig this way of
5:24
working. Incredible.
5:25
So if anyone wants to go deeper down the rabbit hole and learn
5:29
more about what you all are doing at Quantum Black, we're
5:31
going to leave a link in the description.
5:33
Quantum Black is always on the lookout for incredible talent,
5:36
so if you think you fit the bill, we'll also leave a link
5:40
for you to contact them. Thank you, Nair, for coming and
5:43
joining us. Let's get back to the show,
5:51
dude. I know.
5:52
We're here to talk all about what you've been up to and just
5:56
really get into this idea of a coding pilot and the work you're
6:04
doing with Kodium. And I would love to start though
6:10
I guess before any of that and hear about how you got into tech
6:16
and what gave you the bug. Yeah, I guess this goes back.
6:20
This might not be that cool, but this goes back to maybe high
6:23
school. I did a lot of these programming
6:26
competitions. This might be a common story
6:28
with a lot of people. Actually me and my co-founder
6:32
used to be a programming competitions in in.
6:34
In high school he did much more, much more than just the
6:39
competitions. But because of that we we did
6:41
programming competitions together.
6:42
I was really excited about that went to MIT.
6:45
Actually my co-founder, I wear the same year at MIT also.
6:48
So we did competitions against each other in the Bay Area and
6:51
then after that worked a lot together at MIT.
6:55
After that, I joined a company called Neuro.
6:57
They build autonomous goods delivery systems.
7:02
I worked there for around four years, became a sort of a senior
7:05
manager at the company overseeing largescale Dee
7:09
learning workloads, and my cofounder was at Oculus building
7:13
deep learning compilers at the time.
7:15
O afterwards we both came away with a sense of deep learning
7:19
was going to revolutionise many industries and we wanted to make
7:23
technology to make it easier to scale and run these deep
7:26
learning workloads for these for these different verticals we
7:31
started. That's that's where we started
7:33
the company EXA function over 2, 1/2 years ago and what we
7:38
started and I think for your audience, this would be maybe a
7:41
common story. We started with a, we think
7:43
scaling these, scaling GPU workloads is hard, right?
7:48
GPUs are not very, very easy to run, run diverse workloads on
7:53
for a variety of reasons. GPUs have a limited amount of
7:56
memory models use up a lot of memory in GPUs, So it's hard to
8:00
run a lot of models at the same time.
8:02
So we built, in some sense, GPU virtualization technology.
8:06
So take workloads that would run out GPUs and instead they would
8:10
run on CPUs and transparently under the hood.
8:12
It would take these anything you could run on GPUs, including
8:16
CUDA kernels like the lowest level stuff in order to offload
8:19
them to a remote GPU. And if a machine dies, we would
8:22
actually transparently make it run on a different machine
8:25
without the user even know it by reconstructing the state of what
8:28
was on one machine or another machine.
8:30
And so exit function started out doing this and we sold to a
8:33
handful of companies and we sold to many companies, but but most
8:37
of the large workloads were from a handful of companies and we
8:41
ended up managing upwards of 10,000 GPUs for a handful of
8:45
companies. And one interesting stat I'd
8:47
like to throw around is at our peak, we were managing over 20%
8:52
of GCP's GPU inference capacity in a single region.
8:55
And we were optimising these workloads.
8:57
And you can imagine 2 1/2 years ago that was very important
9:00
because there was a scarcity of GPUSGPUS were expensive and
9:04
there were a lot of these large autonomous vehicle work
9:07
workloads that just required a lot of GPUs.
9:10
And specifically the workloads they were running were the
9:12
simulation workloads. Maybe just to go into that like
9:16
a little bit, the fundamental reason in these off workloads
9:19
where the reason why simulation is so big is it is quite
9:22
expensive to collect Milos in the real world and it is much
9:26
cheaper and much more safe to collect it offline.
9:29
So, but that does require running your entire deep
9:32
learning stack offline. So we were, we were targeting
9:36
these large, large workloads and for us I guess interestingly you
9:39
know if you were to think about ML OPS, we were targeting the
9:42
inference space. But the thing that we felt and
9:45
this is maybe a little bit of a hot take was middle of last year
9:48
as we started to basically work on generative AI workloads.
9:53
We used to have companies tell us, hey, can you optimise these
9:55
Transformer workloads For us? We provided them with solutions
9:59
and we actually even built our own in-house deep learning
10:02
compiler that competed against Tensor RT and we actually sold
10:05
it to large autonomous vehicle companies.
10:07
And even right now there are VA VS running around with with Exa
10:11
Functions compiler technology running.
10:14
But we realised that Transformers were going to
10:17
Transformers and generative AI was going to fundamentally
10:19
change the ML landscape. We were not going to see nearly
10:22
as many conventional ML engineers and maybe if I can go
10:26
into that a little bit more, instead of having many people in
10:30
your company tuning these bird models, for instance for
10:33
sentiment classifier, you are going to take these large
10:36
generative models and instead you would just ask them, is this
10:38
a positive or negative sentence. So we fundamentally felt that
10:42
there was going to be a shift in the set of people that were
10:45
going to consume MO. It would be normal engineers
10:48
rather than rather than really, really specialised ML engineers
10:53
is what we felt was going to happen and so.
10:55
You basically you saw that boom coming with the ability to use
11:02
these large language models instead of having to get really
11:05
deep in the weeds and be very knowledgeable about ML.
11:10
That's exactly right. And I think we, we were
11:12
targeting these workloads where people had custom bespoke models
11:16
and we were able to Multiplex them across many GPUs.
11:19
But we saw the writing on the wall that it's possible in like
11:22
a year no one is going to be training a resonant model or a
11:25
tuned resonant model. What was inside going to happen
11:28
is there would be a good enough single shot image classifier
11:31
model, which to be honest proved to be the case, but there would
11:36
be industries where people would need to where the precision
11:39
recall matters a tonne like autonomous vehicles.
11:41
It is not good enough to have 90% precision recall because if
11:45
you miss out on the 10% you could potentially kill someone.
11:49
But we felt the vast majority of workloads would not look like
11:51
that. So what we decided was we felt
11:54
that the workload we were serving was slowly on the, you
11:57
know, going to decline. And we thought generative AI was
12:00
going to revolutionise the way we do work, period.
12:05
And I think we came up with a big shift of, hey, why don't we
12:08
use our technology to scale and run, run these, run these models
12:13
to basically build our own generative AI application, which
12:16
is probably a very strange sort of transition to do.
12:21
And we were all early adopters of Copilot at the company.
12:24
So what we decided was, hey, why don't we think that there's much
12:28
more to programming than just what Copilot has done?
12:31
Why don't we start and try to build what Copilot has today?
12:35
And we believe that there's significantly more.
12:37
But we added A twist that we forced on ourselves, which is
12:40
the product had to be entirely free, which forced us to make
12:43
the infrastructure scalable, which forced us to optimise the
12:47
infrastructure, which forced us to train our own models.
12:49
Podium is not built on top of which is I can go into more,
12:53
it's not built on top of opening Eye, It's not built on top of
12:56
Entropic. And that has its own set of
12:58
advantages that we've been able to make as a company.
13:01
And it's built on top of a lot of the infrastructure we've
13:04
built 2 1/2 years ago. So this didn't happen in a
13:06
vacuum. So middle of last year, we sort
13:09
of started working on Podium and the spelling CODEIUM, we started
13:14
working on on Codium. And one quick question that
13:18
people ask ask is why did you guys just build OPS for LMS,
13:23
right? I guess that's that's a that's a
13:26
common question people people have asked us instead of why,
13:29
Why did you guys pick Codium? And I think the reason why was
13:32
we felt that it was it was generative AI was having its
13:36
Internet era. It was similar to what was going
13:39
on in the in the late 1990s. And one thing that I believed
13:44
is, and this is maybe a stupid take for me, was the biggest
13:47
companies that I remember from, you know, the late 1990s are the
13:50
Googles and the Amazons. It's not the people that build
13:53
web server infrastructure and it's not because web server
13:56
infrastructure isn't big. It's just that the space was so
14:01
new that the companies with the largest workloads are the
14:04
largest applications. Ended up building the craziest
14:07
things for themselves rather than outsourcing the technology
14:10
because it was so nice. It was unclear what you wanted
14:13
to build for someone, if that, if that makes sense.
14:16
Yeah, totally. It's like you are creating the
14:20
infrastructure for yourself because you have this lofty goal
14:22
of an end product that you're going to give to your customers.
14:26
Ye, that's exactly right. So well dude, I gotta ask
14:31
because it feels like we are still in this GPU scarce world
14:37
and the idea of exit function and the capabilities that it
14:40
had. I mean I know so many people
14:43
that are banging their heads against the wall today because
14:46
they get those ohm errors, right?
14:48
Umm. Errors.
14:49
Yeah, they're out of memory, right?
14:52
Yeah, yeah, yeah, exactly. So you.
14:55
But you basically you took a hard right or a hard left and
14:58
said OK, instead of helping people with these these errors
15:03
and figuring out how to offload some of this stuff and helping
15:07
them with the lower level CUDA workloads or just lower level
15:13
CUDA pains that they're going through, you are going to
15:19
leverage what you've created there and then help them with
15:23
the coding part, which is very much an end user, almost like
15:27
you were being. There's like a consumer at the
15:30
end, yeah. Yeah, exactly.
15:32
As opposed to, I'm an infrastructure layer for some
15:35
really advanced ML/AI company. Yeah.
15:41
If I can go into that a little bit, it's that we believe that
15:44
Transformers and generative AI was going to make up a large
15:47
fraction of GPU workloads in the future.
15:50
But we felt by only providing value at the inference layer, we
15:54
were not, we were not really helping the end company in the
15:58
way that I wanted to help them in a differentiated way.
16:01
So let me maybe go into that a little bit and, you know, feel
16:05
free to stop me. In the.
16:06
Past if there were many different model types, yeah,
16:09
there were many different things to optimise on.
16:11
But now the set of workloads that were operating, they were
16:14
sort of narrowing down and homogenising.
16:16
Yeah, there were. There were differences in
16:18
different transformer architectures for sure.
16:20
Like you can have maybe sparse attention later as you can have
16:23
multi query attention versus group query attention and there
16:26
are different nuances but for the most part everything started
16:29
to 0 in on a certain type of architecture.
16:32
So what I felt was over time the infra layer maybe will not
16:36
commoditize but at the very least the differentiation was
16:39
reducing. In addition to that, what I also
16:43
felt was we were solving a very little amount of the pain of the
16:46
company. And I I'm saying that from the
16:48
perspective of many generator the I companies or application
16:52
companies reached out to us run at the time a bit over a year
16:55
ago. The best model that was out
16:56
there was was GPTJ which is actually like still a really
17:01
good model, but it's not going to compete with any of the GPT 3
17:04
models. And that's fundamentally because
17:07
for a variety of reasons a like let's say someone like a Jasper
17:11
reached out to us, did it have proper data for content
17:14
marketing, They probably did they clean the data properly,
17:18
Did they actually fine tune the model properly?
17:19
Like did they figure out how to train the model properly?
17:22
Did they figure out how to evaluate the model properly?
17:24
First of all, evaluation in and of itself.
17:26
I could talk for an hour about evaluation and how we do
17:28
evaluations of the company. It's very domain specific.
17:31
It has what we do for code evaluation.
17:33
It looks nothing at all like what other companies do by the
17:37
way. Fascinating.
17:38
And then did they, did they actually run the model
17:40
efficiently? Yeah, that's that's there's a
17:42
little bit of that component. And the 5th part, can they AB
17:45
test different models in the wild properly?
17:49
Can they effectively build AB testing?
17:51
And then on top of that, can they take user, user data,
17:55
whether or not it's exact data but preferred data to then
17:58
further improve the model. And as you can notice, each of
18:01
these pieces is are different. And yeah, like you can build a
18:04
point solution for each one of these, but that's not going to
18:07
solve the real pain. Like imagine me as an
18:09
application application company buying six or seven point
18:12
solutions to run my general VI app.
18:15
I just felt like that was not, that was, like, going to be
18:18
weird, Yeah. That's not the future.
18:21
Yep. So I I 100% see that vision.
18:26
It's like there, especially the way that you put it, where the
18:32
pain was a small pain point and there were so many other pains
18:38
in that journey that you weren't touching.
18:40
So you were almost dependent on the fact that they needed to get
18:45
2-3 four steps right before they even started talking to you and
18:49
were advanced enough to need your product.
18:53
That's actually a great way to put it, especially given the
18:55
fact that these companies beforehand were running on top
18:58
of an opening eye model. And in a lot of the cases what I
19:01
told them and this is because like yeah, serving a customer is
19:05
great, but being honest with them is, is what's going to
19:08
create long term value. I just said, if you guys don't
19:11
do the other stuff, it's just better to continue using Open
19:15
AI, yeah. Totally.
19:18
And that's the hard truth right there, because we can all talk
19:22
about fine tuning and how nice it will be when we have our own
19:26
proprietary model. But then there's the hard parts
19:30
of actually making that happen. And what you need to go through
19:33
to make that happen is, is something else.
19:35
And so you have to become very skilled at it.
19:38
I mean, granted, I think there's a lot of momentum going in the
19:42
direction of trying to find ways so that it is not as hard as it
19:46
probably was like a year and a half ago, but it's definitely
19:50
not like click a button easy, right.
19:52
And I don't think we're ever going to get there, because you
19:56
need the data, and the data is the messy part.
19:59
That's exactly right. You need the data you need, you
20:01
need the infrastructure to build out all the other stuff and
20:04
there are still domain specific problems on a per per.
20:08
Kind of generative AI OP basis. The kind of work that we do to
20:11
evaluate our product looks different than what our friends
20:14
at Mid Journey do. You're doing something a little
20:17
bit different. The way they look at human, they
20:19
look at feedback and the way they constantly improve their
20:22
models, it looks different. And this is where it comes back
20:25
to like yeah, maybe there was something common that companies
20:29
could have sold to both Amazon and Google right in the early
20:33
days. But my sense is it was quite
20:35
hard. It made a lot more sense in the
20:38
in we're in the weird way to sell EWS.
20:41
That was your Cloud infra solution in maybe 2006, 2007
20:46
once the early ages of the Internet had been proven out and
20:49
there were some winners. But it doesn't make sense to
20:52
provide technology to companies. In my mind.
20:55
This is maybe a little bit of an odd take before there are well
20:58
defined workloads in the space already that you can tackle.
21:04
I see. So you're basically saying like
21:07
because we're so early, still with ML and AI, it's in your
21:13
eyes you didn't want to jump in and be opinionated.
21:16
Opinionated about something before the cards had fallen,
21:22
yeah. And then I think until then, I I
21:24
believe and not only maybe this is like there's a pragmatic way
21:28
to look at it and also there's a passion way to look at it.
21:31
I believe building applications is where where the value will
21:34
actually lie ultimately. But all that being said, at the
21:38
same time, like I also believe what we're doing to accelerate
21:41
software development with Codium is fascinating and I'm super
21:45
passionate about it and happy to talk about that as well.
21:47
But. But yeah.
21:49
Yeah, I I find that fascinating too, because obviously I play a
21:52
lot in the ML Op sphere. And these days, as you know,
21:56
it's hard to call it LLM OPS. But the idea of, oh, the real
22:03
values that the application layer I'm not necessarily sold
22:07
on because if I look and I have plenty of examples of like you
22:11
still need infrastructure. You still do.
22:14
There's still going to be value created at the infrastructure
22:17
layer and we can look at like the Confluent or Hashi Corp and
22:23
those type like even just GitLab, right?
22:27
Like infrastructure still is there And there are companies,
22:30
gigantic companies being made on the infrastructure layer.
22:33
No. Are there a tonne more
22:35
applications? Yeah, probably.
22:39
The only thing, the only thing I will say about that and I think
22:43
those are fantastic examples I was actually at LinkedIn back
22:46
when the I guess the like as an intern back when like the
22:49
founders of Confluent were still at at LinkedIn.
22:52
So the one thing I will say about that, that's maybe one of
22:54
those pieces of infrastructure that makes a lot of sense
22:57
because that was proven where a bunch of Internet scale
23:00
companies were like I need a public pub subsystem and
23:03
Confluent was like actually you know what this is like the best
23:07
Kafka is the best pub subsystem you can make and it's a
23:10
primitive that every Internet scale company or any company
23:13
with a lot of data needs. And it was proven out over many
23:16
years. Like every company had figured
23:17
out their own crappie pub subsystem and suddenly one
23:20
company was like, we have the best one, guys.
23:22
We figured it out over years of building it out.
23:25
LinkedIn has been using it for years.
23:26
It's replaced everything we had in the past and now here's ready
23:30
to go and consume. But that took many years to hit.
23:34
Like, no, If Kafka had come out in 2000, companies would have
23:38
been like, what is this for? Or maybe Kafka wouldn't have
23:42
looked as polished as it looked like now to be easily consumed
23:46
by every company. That's like the take I have in
23:48
my head, right? There was no Kafka in 2000
23:50
basically. Yeah, it goes back to really
23:53
like this idea of of course, like even in the data
23:57
engineering sphere, which I feel is a bit more mature than the ML
24:01
Op sphere, almost every tool that I see come out is just a
24:09
new abstraction. It's like we're going to create
24:12
a better abstraction layer than the last two or the other tools
24:16
out there. And that's really what they're
24:18
trying to hit on is, hey, how do we create a abstraction layer
24:23
that is comfortable for data engineers?
24:25
And it's almost like we are again so early in the journey
24:30
when it comes to using ML and AI that trying to hit on the right
24:34
abstraction layer is maybe a little bit preemptive.
24:39
Yeah. And maybe if I can add one thing
24:42
to that and this might not be super interesting to the
24:45
audience. I actually worked at Cloudera in
24:47
like 2014 and then and then data breaks at 2017.
24:51
And obviously one is like living in the Hadoop ecosystem right
24:56
back when. And Hadoop solved like an
24:58
interesting problem, right? Companies had a lot of data and
25:00
processing it and then Spark came out and at data bricks, the
25:03
reason was the reason why it won was it was 10 times better.
25:06
And there were so many companies that already had Hadoop.
25:08
And it was like, this is actually the right way to
25:10
process your data. But if something else came out
25:13
that was like, and maybe this goes to an interesting point to
25:16
what you said, which is like every company's trying to make
25:18
something that's like, you know some way ergonomically better.
25:22
If one company came out with a product that was like 10% or 20%
25:26
better than Spark, it would not get the adoption that Spark had
25:29
right. Because Spark was such a big
25:31
improvement over the previous technology and it was already
25:34
being used at so many large companies that it was a no
25:37
brainer for more companies to take it on after that.
25:39
So it was proven out at a lot of companies before IT database was
25:41
able to become the behemoth that it is right now.
25:44
Yeah. So it's almost like saying that
25:46
if you're 10% better than Spark, that's not going to be worth the
25:53
pain of ripping out Spark in your infrastructure because 10%
25:58
is like I could deal with 10% whatever.
26:02
Yep. Especially if it deals with
26:03
having to work with a vendor that's like a much smaller
26:06
player, right? Yeah, 'cause you don't have that
26:08
assurance that they're gonna be around forever, So that's right.
26:11
I like this. I'm gonna change gears, though,
26:13
because I I really want to hear about Codium.
26:16
And I I really like this idea of, like, you built something
26:22
knowing that you were already in this gigantic scale, 'cause you
26:26
were with Exa function, you had scale that you were giving to
26:30
customers and you were helping people with the serving layer.
26:34
And then you were like, OK, well, let's just do that but
26:37
with our own product now, right? Yeah.
26:39
So I'm guessing that, 'cause now give me the, like, Fact Check me
26:44
on this. But you're serving a lot of
26:46
people, like, hundreds of thousands of developers.
26:48
Yeah. So yeah, break that down.
26:51
I let me go into the story. So we start.
26:53
We started building Kodium last year.
26:55
We realised the product needs to be free.
26:57
We would go bankrupt if we used Open AI.
26:59
So we then start saying, hey, we got to like train our own
27:02
models. We have to train our own models
27:05
to make this work. So we actually retrain entire
27:08
models from scratch to run Podium for our developers.
27:12
We train our own models, we evaluate them, we deploy, we
27:15
test them, all this other stuff. And we learn a tonne by the way
27:19
and a lot of the conventional stuff of like prompt
27:21
engineering. Yeah, we had to prompt engineer
27:24
in interesting ways that I can go into that, but a lot of it
27:27
was domain specific. And I can also go into that
27:29
because for our problem, I'll tell you why.
27:31
Training our own models provided us a lot of value.
27:35
When you write code, Kodium's one of its main features is
27:38
autocomplete. And autocomplete is maybe a
27:41
little bit of a devious term because it sounds like, oh, this
27:43
is like providing me like a couple of characters.
27:45
We sometimes generate 2030 lines of code for the end user.
27:49
No way. That's awesome.
27:50
And. The reality is when we when we
27:53
generate code, we need to look at both code before and after
27:56
code in different files too. And this this doesn't look like
28:00
the data set of. This doesn't look like any of
28:03
the tasks that exist right now for these for these generative
28:06
models. Generative models, generally
28:08
speaking, the way it works is like you have some existing text
28:10
and it just keeps adding on to it.
28:12
It doesn't by default. It doesn't know how to apply
28:14
context from other files. It doesn't know how to fill in
28:17
code in between in in between your cursor.
28:20
And so we had to train models explicitly to do these tasks
28:24
well and to tune them to to do well on these tasks.
28:27
So we had to actually go out and train our own models to actually
28:30
do this. Some numbers about our scale.
28:32
We started the year with around 1000 users and now we have
28:35
hundreds of thousands of users using the product on the VS Code
28:38
marketplace. If that's any sign.
28:40
We have close to 400,000 sort of downloads and then sort of five
28:45
star reviews across all of our users.
28:47
And in fact, the thing that gives me the most excitement is
28:49
because we forced ourselves to actually train our models and
28:53
give the product away for free. It's a bunch of people that have
28:55
asked us, I don't really understand how this product is
28:58
free. It is better than Copilot.
29:01
That is like, you can just look at your love.
29:03
Yeah, that's how is this product free?
29:05
And the reality is there's a team on the other end that's
29:07
like, that's like, you know, breaking their back, figuring
29:10
out how to run this product at massive scale.
29:12
And just to put in context, we process 10s of billions of
29:16
tokens of code across all of our users.
29:19
They accept over 100 million lines of code a day across all
29:23
of our users right now. That's like the level of scale
29:26
we're talking about 10s of billions of tokens of code.
29:29
Think about it as billions of lines of code that we process
29:32
across all of our users now, right?
29:34
That's like one, and we're doing that.
29:37
We're doing that in in ways where we've had to pull out all
29:40
the stops. We need to do crazy amounts of
29:42
batching. We take advantage of
29:44
quantization. We take advantage advantage of
29:46
model parallelism. We do any trick under the book
29:49
that we can do, we we sort of do.
29:51
And not only do we do that, which is very interesting and
29:54
I'm happy to talk about chat. Two, we generate more
29:57
suggestions on a per user basis than products like Copilot do
30:00
too. There was a recent article from
30:02
the Wall Street Journal that your users might have read that
30:04
they're burning a lot of money on a per user base.
30:06
Because we've optimised the stock so much and verticalize
30:10
the product, we've been able to. If you actually notice, there
30:13
are places where we generate suggestions that Copilot doesn't
30:15
even generate suggestions. If you're in the middle of a
30:17
line, Copa will never generate a suggestion for you.
30:20
We will generate edits even within a line, which is actually
30:24
a vast majority of the way people write code you can
30:26
imagine. You know, you take a normal
30:27
software developer life cycle. Most of the time they're
30:30
changing and editing existing code.
30:33
They're refactoring existing code rather than adding large
30:36
files of of new code, right? And then on top of that, our
30:40
context. When we apply context throughout
30:42
your code base, we apply context not even through open files.
30:45
We apply context through all adjacent, not even adjacent
30:49
files. But we use embeddings to find
30:51
any file in your codebase to apply context to for for the end
30:56
user, yeah. Hey.
30:58
Everyone. My name is Aparna, founder of
31:00
Verizon. And the best way to stay up to
31:02
date with Emma OPS is by subscribing to this podcast
31:08
that's just for auto complete. And on top of that we have a
31:10
chat. We have a chat system as well
31:12
that is entirely in house that also applies context throughout
31:18
your code base. And I think some of these
31:19
aspects might be similar, but there are aspects that I can go
31:21
into that are different across companies where even right now a
31:24
product like Copilot doesn't use context throughout the code base
31:27
two for either auto complete or chat.
31:29
It just uses the open file write up.
31:31
So we've taken it over the last year.
31:34
We've moved extremely quickly. We're now one of the world's
31:37
largest generative AI workloads, which is actually like another
31:40
crazy stat. I mean we serve so many users
31:43
and our product. The crazy thing is we serve, we
31:47
run a large multi billion parameter model on every
31:50
keystroke that a user types. That's like the scale like of
31:53
what we're talking about. That's basically what auto
31:55
complete is, so. Wait, so yeah, of course.
31:59
There's all kinds of questions that are coming through my head
32:01
right now. Like, is it an API?
32:04
It's, it's a plugin that comes in, you mentioned.
32:08
How does that look under the hood?
32:10
Are you making or am I making an API call every time I type
32:15
something? Are you running a model on my
32:17
machine? Like, what?
32:19
Does that even break it down for me?
32:21
Yeah. I can, I can break that down for
32:23
you. So a couple things that just
32:25
want to say about how Kodium looks.
32:27
So Kodium is like a like AID extension.
32:29
That's the way it presents itself for different users.
32:32
So you can download an extension on VS Code, Intellij, A Vim,
32:37
Eclipse. We support basically every ID,
32:40
and part of the reason why we've been able to support every ID is
32:44
we actually allow the open source to contribute extensions.
32:47
And the open source has actually contributed extensions in Emacs,
32:51
Vim, I believe. Another one is Xcode, and so on
32:55
and so forth. We support many different kinds
32:57
of extensions and underneath the hood, you're totally right.
33:01
On every keystroke we go out, we go out to the cloud and we do
33:05
some processing right to for for a generation, right on every
33:10
keystroke that the user types. We don't actually run these
33:13
models on the user's local machines.
33:14
And just to give you a sense of the reason why is because at
33:18
least conventional Macs, there might be the M2 Pro that people
33:21
run. They they can do like somewhere
33:23
in the 10 to 15 teraflops of compute range.
33:26
Whereas modern H1 hundreds H 100 is just an example of just some
33:30
hardware can do between 1000 and 2000 teraflops of compute.
33:35
So you can just see that there are two words of magnitude
33:38
difference between what consumers have access.
33:39
And these are like top of the line consumers versus what
33:43
exists if you have proper GPUs and GPUs have much more compute,
33:47
right, compared to compared to consumers.
33:49
And most consumers don't run on high quality Macs like most of
33:52
our users are Windows users that run on top of old laptops that
33:57
have probably an order of magnitude or maybe like 1 1/2
34:00
orders of magnitude less compute than even a Mac when it comes
34:03
to. So there's no assumption we make
34:06
on the end user because that would be unfair to them, right?
34:08
They would. We would not be able to provide
34:10
a good enough product to them. So yeah, we've needed to
34:13
orchestrate large scale inference to basically run this
34:16
workload for our users. Yeah.
34:19
So tell me then about the scaling that you were doing.
34:24
Like how did you go from 1000 users like you were saying at
34:28
the beginning of the year and then just like ramp you were up
34:31
and to the right. I'm sure investors love talking
34:33
to you, man. I think, I think the way we, I
34:37
think the way we, so I I'm happy to talk a little bit about about
34:41
some of the issues that we that we had but let me first talk
34:44
about the scaling or individual developers.
34:46
So the way in which we did that was, hey, like we've built
34:50
scalable systems. We've already managed 10,000
34:53
GPUs in the past. It's not that hard for us to go
34:56
about managing a lot of DPS again and scaling our workloads
35:00
for across many users. That was like that was the that
35:03
was like we built the company around doing that already.
35:06
So yeah, we had more users, we needed more GPUs and.
35:11
We. And so were you already, did you
35:14
buy a bunch of GPUs or did you just reserve them from Amazon
35:18
and all that like you had GPUs on reserve and so it was a quick
35:23
and easy shift, yeah. So we, we were able to get
35:26
reservations to basically to basically do this and we could
35:30
plan for it. Hey, we're going to have this
35:32
many, we need this many GPUs to run the workload like this
35:37
obviously like maybe one thing Demetrius that could be
35:40
interesting is we actually need significantly less GPU's then
35:43
then probably a company like than other companies in the
35:46
space. And you know we've needed to,
35:49
you know we don't talk too much about our numbers and exactly
35:52
how we do things internally of of in terms of optimising our
35:57
workloads. But the big thing we provide to
35:59
enterprises and we have a many large Fortune 500 customers now
36:03
that are paying us money, They're not just like I guess on
36:06
the individual plan is we enable companies to self host the
36:09
product. So for a lot of companies you
36:12
brought up Git Lab, right, Demetrius, like for a lot of
36:14
companies, they believe code is their IP.
36:16
They don't want code leaving the company and that doesn't matter
36:19
how. They just don't want it leaking
36:21
outside of the company, right. And for those companies, they
36:24
even self host their code. SEM and GitLab, all those other
36:28
things are self hosted for them. And for those companies, we let
36:31
them actually self host the product.
36:33
And you can just imagine if we had bad infrastructure, if we
36:37
made the company self host the product, they would need to buy
36:39
a lot of hardware and that would not be fun.
36:41
They would just say like hey like you're making me spend a
36:43
lot on hardware, but instead these companies don't need to
36:46
buy a lot of hardware when they self hostium.
36:48
And for these companies that self host kodium, we not only
36:52
give them the ability to self host the product, we give them
36:54
additional benefits of being able to personalise and fine
36:57
tune the product on their private data.
36:59
And we've actually built infrastructure to continuously
37:02
fine tune on the same piece of hardware on their code base.
37:05
And this is another advantage we have because we train our own
37:09
models from scratch that we can enable companies to go and do
37:12
that and we can optimise that workload and not be like, hey,
37:16
no, actually ship your entire code base out to Open AI.
37:18
The company doesn't want to do that.
37:20
We won't make them do that. And so basically you're a, you
37:27
meet with a customer of Fortune 500, they have their GPU
37:30
capacity, whatever that is and you give the go, no go sign on
37:36
if it's going to. Work.
37:38
We asked them to start with. Just give us one GPU, no?
37:41
Way. OK, give us.
37:43
One GPU. And in fact, like we haven't
37:45
posted a lot of case studies, but we had Dell actually come
37:47
out with a public case study on their site about how either
37:51
developers benefited a lot, but they were able to run the
37:53
workload entirely on Prem in an air gap.
37:58
OK. So that of course was one of my
38:01
other questions. Like, OK, you trained these
38:03
gigantic models. GitHub Copilot is in a bit of
38:08
hot water right now because people are like, hey, my open
38:11
source code shouldn't have been trained on that.
38:15
Did you guys get pulled into that or did you sidestep that?
38:18
What's that look like for you? We've.
38:20
Been extremely careful about this and this was another reason
38:23
why we decided to go and train our models.
38:25
We've done two things. First, we don't we may we don't
38:29
train on any non permissively licenced code.
38:32
So that means if your code is GPL, I mean, and I don't know,
38:36
the audience might know this, there's some forms of code where
38:40
if you actually copy that code into your code base, you need to
38:44
make your code base that licence to it.
38:46
And that's GPL, right? Which is problematic for
38:48
Commercial Code. You don't want to end up like
38:50
putting some GPO code in your code base and then you need to
38:52
take your private company's code and make it open and modifiable,
38:56
right? So we took the steps of early on
38:59
don't train our GPO code and we had a big blog post that we
39:02
actually came out with that had hundreds of thousands of views
39:04
on our on our website about how generates GL code regardless of
39:10
all the philtres you turn on, right?
39:12
But we took the point of we will not train on any GPL code
39:15
because we're serious about this.
39:17
We want to be the most secure and compliant way of also
39:20
consuming this and large companies should feel, feel good
39:24
using our product to be, hey, I'm not going to be in any hot
39:27
water down the line and this is something which we are also
39:31
providing end users going forward.
39:33
We also let them attribute the code.
39:35
So even for the permissively licenced code, if they want, for
39:38
some subset of companies we let them attribute the code also.
39:43
So we built the software in such a way where we're not going to
39:46
get into these legal, I guess these legal issues.
39:49
Because basically, if first of all for any permissive, non
39:52
permissible licence code, it's not in the training set and we
39:54
can give people the guarantee, and for permissively licenced
39:57
code we can give them attribution guarantees if they
40:00
really want. So I like to say that I only
40:04
know two types of licences out there.
40:07
One is the kind that Amazon can steal and make their own, and
40:12
the other is the kind that they can't.
40:15
So I'm guessing that you're talking about the kind that
40:18
Amazon can steal. So that is a little bit
40:20
different. So OK, companies like MongoDB
40:23
and maybe Elastic, maybe Elastic didn't do this and that's why
40:26
they're hurting a little bit. MongoDB did was early on they
40:29
said so that Amazon cannot just build our product and then on
40:33
top of that just you know kind of sell it because they have
40:37
much more distribution. We are going to make it so that
40:40
our product has a commercial licence, but if you want to sell
40:43
the product I don't believe you can be anyone but Mongo, the
40:47
Mongo DB Corporation, which is a little bit different.
40:51
The licence we're talking about with GPL, that's a non
40:53
permissibly licenced licence, is is the thing that's licencing
40:58
Linux for instance. Linux is free and open software,
41:03
but if you copy pieces of Linux and you put it in your source
41:06
code, then your source code necessarily needs that licence
41:10
and needs to be free and open software, which is problematic
41:14
for large companies. Imagine telling I'm going to
41:17
name a company. Imagine telling Shopify hey use
41:21
GPO code. Make your entire code base free
41:23
and open now. They're not going to be too
41:27
happy about that. That's not going to fly at all,
41:30
man. Yeah.
41:30
That is wild. So then I I would love to hear
41:35
about like some of the basic challenges that you've had.
41:41
I mean, it seems like scaling because of the optimization, you
41:44
were able to kind of sidestep that pain and all of the work
41:48
that you had done in your past lives.
41:51
But I imagine there's a tonne of other pains that you had.
41:56
I could. I could go on for hours.
41:57
I think the problems were actually very domain specific.
42:00
How do we build the best code product for an individual in an
42:03
enterprise? Is what I think about nonstop
42:06
and we needed to build large scale evaluation systems to
42:10
actually even just validate, Hey, is my model better at
42:13
applying context? Is my model better at completing
42:16
code? And we've built crazy.
42:18
And this is where it gets to the internal systems that we've had
42:22
to build. Because we are a code based
42:24
product, we actually have evaluations that require us to
42:28
take a bunch of open source repositories, find, run, run the
42:33
unit test in the code base, right find, find functions that
42:37
that 'cause these unit tests to pass, delete a bunch of the
42:40
code, and then see if our models with the context of the
42:44
remaining part of the code base make the unit test pass.
42:48
That's the beautiful thing about code.
42:49
You can actually execute it and see if the code is correct.
42:52
Because in other spaces maybe you do need to validate that
42:55
it's word for word the same, But we don't really care if the code
42:57
is word for word the same, we care if the functionality is
43:00
correct, right? So that has allowed us this
43:03
large scale evaluation has allowed us to check a Is our
43:06
model better at applying context?
43:08
Are we getting context from the right data sources?
43:11
Are we actually as our model the base model?
43:13
Is that an improvement on top, On top of the previous base
43:16
model at generating code And there are so many tunables that
43:19
we get access to once again by verticalizing right by us having
43:23
so tight access to the model, so, so much tight access to the
43:26
data. There's so much value we get
43:28
from that and that has become an awesome signal.
43:32
That evaluation, I told you, has become an awesome signal before
43:35
we deploy a new batch of software.
43:37
And then on top of that, when we deploy a new batch of software
43:40
we can in the wild. We we have so many completions
43:43
that we have a day, we have 10s of millions of of sort of
43:47
inferences we run a day. We are able to see, is the model
43:50
getting better in the wild too? Is there a discrepancy between
43:53
our eval and what we have in the wild?
43:55
And then on the enterprise side, where we have very little
43:58
visibility and access to data when we deploy our models is the
44:01
end user with fine tuning on their private code, which is
44:04
something that only we offer, right, and personalization with
44:08
their with their internal SEM, whether it be their code lives
44:11
in GitLab, Bitbucket. Is there a product actually
44:14
getting better? Those are also things we check
44:17
and we check in eval offline as well.
44:20
So there's so many things that we've needed to solve by the way
44:22
that are outside of just scaling the product that are are hard,
44:26
right. And they're unique to us.
44:28
Like if you talk to another generative AI company that's
44:30
outside of the code space, they're not going to tell you
44:32
any of these things that are that exciting.
44:33
All these things are are specific to our company
44:36
basically. Of course, it really makes it
44:39
reminds me of we just did an evaluation survey about a month
44:43
ago and we asked people in the MO OPS community how they're
44:47
doing evaluation and all that, and it's very much like what
44:50
you're saying. Are they what are they
44:52
evaluating is a huge question because as you mentioned,
44:56
evaluating code is one thing, evaluating if a poem works or
45:01
doesn't work is another thing. But code, you have this nice
45:04
little it's almost a hack or a cheat.
45:07
Code that you get to say it's going to pass unit tests or it's
45:12
not going to. It's very black and white.
45:14
And so because of that, it makes it so that you can really build
45:20
out, I imagine, very robust evaluation frameworks.
45:24
Yeah, of course, 100%. Yeah.
45:27
And then and also going back to you mentioned mid journey before
45:30
and all the evaluation that they're doing and how they're
45:34
evaluating what photo is better than another and and they're
45:39
keeping us in discord. Those guys, go ahead, I said.
45:42
I thought you were about to say something.
45:44
I was, I was just about to say that that is exactly right.
45:47
I think for us, I would say our product is significantly kind of
45:50
further ahead in terms of what the users have than what a
45:52
product like like Copilot has. But if I was to talk about one
45:56
thing this is I don't want to spend the time bashing a product
45:59
because it doesn't make sense for us to.
46:01
Yeah. I think Cobalt was a fantastic
46:02
product. That's like the thing that got
46:04
us into into the space. I think the one interesting
46:07
thing that you said that was very telling was the
46:10
requirements for for writing a poem are very different than the
46:14
requirements for generating code, right.
46:16
They look very different. And there was a feature that
46:19
Copilot came out with a long time ago.
46:22
They had this announcement in March about this feature that
46:25
they had called Copilot XI. Still don't know what exactly
46:28
Copilot X is, but they did a marketing campaign for.
46:30
Yeah. It's huge.
46:31
It was like, this is the future of software engineering.
46:34
I don't. Know what is really has really
46:36
changed since then, but maybe maybe I'm missing out.
46:39
Copa had a feature called Copilot for PRS that was going
46:42
to say, hey, we are going to generate APR summary.
46:46
OK yeah. And we were early adopters
46:49
about. We were like, hey, like maybe
46:50
they figured out something very interesting, like let's try it
46:53
out. We are always excited to try new
46:54
technology. And basically you can think
46:57
about the PR process. There are three things you can
46:59
do. You can either write the PR, you
47:00
can review the PR or you can summarise the PR, Right.
47:03
There are only three really things you can do.
47:06
And when they wrote the PR summary, we noticed that the PR
47:10
summary was most of the time like either wrong, it missed out
47:14
or it missed out things in the PR.
47:16
And then at that point I ended up reading the PRS anyway.
47:19
But the PR summary had one really good thing in it, which
47:22
is a poem. And it was the short haiku that
47:25
that that sort of summarised the PR.
47:28
It was like 3 sentences and it was kind of cute.
47:30
But it went to show to me that the bar required to make tasks
47:35
to actually optimise a task isn't just make a user feel
47:39
good, It's like if you make APR summary and it's wrong, you will
47:42
erode user trust in the product. And by the way, that is why
47:47
Hawkish when it comes to like our performance of our product,
47:50
how well our people are using the chop product, How well are
47:53
people using the auto complete product?
47:55
Are we generating more and more lines of code for people?
47:57
Because we look at the metrics with a fine tooth comb.
48:00
We constantly are like, oh, our users really enjoying the
48:04
product. Like can we find out if we ship
48:06
this out, are users going to accept more code and how do we
48:09
give them more value is all I think about.
48:11
It's not the point of like, hey, can I generate them a cute poem
48:14
and and you know, build this hackathon feature.
48:17
There's so many easy ways, easy Twitter demos.
48:19
I'm not saying the Twitter demos are easy, but there are so many
48:21
things we could ship that are like easy eye attention robbers.
48:25
But unless there's recurring usage, we're not going to go and
48:27
do it, basically. So are those the main metrics
48:30
that you're looking at and you're focusing on, like how
48:32
much code was accepted, how, how long are people using this or
48:38
what? Yeah, like.
48:39
How how much time are people spending and getting value from
48:42
our tool, right? And we look at it from the
48:44
holistic software development life cycle.
48:46
There's like you. Chat is now a paradigm people
48:49
do. We people do some work in the
48:51
terminal to debug stuff. People do a lot of work in the
48:54
IDE to read and understand code, and then people do work in the
48:58
SEM, like get Lab to generate PR summaries to to to review code.
49:04
And we're looking for ways in which we as a company can
49:07
actually improve the life cycle for the Endic developer, right?
49:11
How do we actually improve that? And that may mean we're not
49:14
doing the entire thing in 10. We're not at the stage where
49:17
developers can be, are not, are useless.
49:19
Now we cannot go and generate APR all in one shot like no
49:23
company can go and do that. So what we should be doing is
49:25
trying to be the best aid to reduce the amount of work that
49:28
the developer does and make it as easy as possible for them to
49:32
do their underlying task. And that might mean just getting
49:35
rid of the boilerplate tasks, but it may mean, hey, we're not
49:38
at the point where we can generate accurate PR summaries,
49:41
and that's fine. Ultimately, we're only going to
49:43
deliver a product if developers will reliably use the product
49:47
with time. So you mentioned before how
49:52
basically with Exa function you were one point on a six or seven
49:58
point step in putting machine learning AI into production.
50:03
Now you've you've kind of switched and now you are a tool
50:07
that is using all of the different points or at least you
50:11
have to get all these different points correct.
50:14
Are you using any pointed solutions or is everything in
50:18
house like have you gone out onto the market and said what's
50:21
out there for the monitoring tool?
50:22
Let me go see what's out there for a curving tool, etcetera,
50:25
etcetera. So I think, I think maybe one
50:28
thing I will say is like, yeah, we we're not gonna reimplement
50:31
things like monitoring with Prometheus and Grafana.
50:34
These are systems that we we use and for feature flagging.
50:37
We're not gonna go and like build our own in-house feature
50:40
flagging system, right, if we want to like ship changes very
50:42
quickly. But when it comes to the LLM
50:44
space, we also need to self host the product for our end user.
50:47
So it doesn't make sense for us to go out and rely on a SaaS
50:51
solution if we can build a like a tightly integrated
50:55
verticalized product. Because if we do like for
50:57
instance, maybe I'll give you an example, we actually run our own
50:59
large vector databases ourselves and and if we used a system like
51:04
pine cone which I'm sure pine cone is a great system and then
51:07
we want it to go to a company that needs to self host.
51:09
We can't just be like, hey guys make a create a Pine Cone
51:12
contract with us also we have to be like, no, like we're going to
51:17
provide you the solution. And also on top of that, our
51:20
vector stores are at a scale right now where we would be
51:22
spending a tremendous amount of money if we wanted to use Pine
51:25
Cone. So we out of necessity figured
51:27
out how to build the product ourselves, like across each of
51:30
our users. And you know, probably as you
51:33
know, maybe a million vectors or maybe 10s of thousands of
51:36
vectors to give us. So we're talking about you know
51:39
maybe billions of vectors that we need to collect, right, maybe
51:42
10s of billions of vectors we need to collect.
51:46
So I think, I think for us right now we are focused on for the
51:50
things that require the core product to work.
51:52
We have not gone out and got a lot of point solutions like I
51:55
don't know what we what point solution we would buy for prompt
51:58
engineering when we actually fine tune our own models.
52:01
We know our data sources that we need to use.
52:04
We have tight, we have tight integrations of is the product
52:07
getting better or worse with time.
52:09
Obviously we would need to measure these things ourselves.
52:11
We can't outsource that to someone else.
52:12
So it's actually strange we're not really purchasing anything
52:16
externally. So then I mean basically it goes
52:19
back to that point you were making before where you were
52:22
saying how because we are still so early, it doesn't make sense
52:27
to buy something on the market. And it feels like you are just
52:32
like a microcosm of what you were talking about before that
52:35
is this macrocosm of you're not going and getting or AWS and
52:39
Google didn't create or they didn't go onto the market and
52:44
figure out how to create AWS or buy AWS I should say when they
52:50
were scaling and they were building Amazon because it was
52:54
too early. And we are, we are being a
52:56
little pragmatic here like we're not re implementing and self
52:59
hosting like crazy databases for our own our own internal
53:02
product. Although we do need to self host
53:04
them for end users. And this is the same.
53:06
Actually one very interesting story about Amazon was Amazon
53:09
early days ran a lot of their database transactions off of
53:12
Oracle. Oracle build ran crazy high and
53:15
then at some point they were like hey we got to have an in
53:18
house team to build their own database.
53:20
Granted we don't want to get there.
53:22
We genuinely I think database software is the hardest software
53:25
out there. We never want to go and build a
53:27
database like we will for anything where we can buy or use
53:31
something off the shelf. We will use things off the shelf
53:34
if possible, but we have we tend to like open source solutions
53:38
that we can self host if possible too if that makes
53:40
sense. Yeah, I think I heard a podcast
53:43
where they were talking about Amazon was still using Oracle
53:47
until like 2018. I'm sure that was the last that
53:51
was like the last Oracle server got shut down in 2018 and they
53:56
had like a party. But the the thing that I find
53:59
fascinating is this idea of you understand alright and
54:04
Prometheus or if you're trying to use feature flags then you're
54:08
going to go out there and you're going to buy some sass.
54:12
But when it comes to the MLA I life cycle, you're not as
54:16
bullish on going out there and buying something you would
54:21
rather create an in house. And really that just shows to me
54:24
like exactly what you were talking about before, right?
54:27
How right now it feels too early.
54:31
You are so custom and everything that's out on the market doesn't
54:35
really address the needs that you have gone in.
54:37
You've ran so fast in One Direction for the last whatever
54:41
year, year and a half, that there's not tools out there that
54:46
are sufficiently advanced as you would like them to be.
54:49
And so maybe it just needs time to catch up.
54:52
And then in a few years, you're going to be like, OK, cool, we
54:54
don't need to do this anymore because there's enough out there
54:58
that we can just offload it. Yeah.
55:00
And the weird thing is I also feel, you know, putting my
55:03
startup pad on, I don't know if we're like the best customers
55:06
like let's suppose someone helps us with code evaluations,
55:09
they're not going to be able to take that and sell it to mid
55:10
Journey, whereas all that tech character AI, right.
55:14
So I mean you know what I'm saying?
55:16
So it's like, I I think, I think it's people need to figure out
55:21
what the common piece of infrastructure is that is worth
55:24
selling. And if there's a whole, I think
55:26
there's a hundred companies now selling like a Lama 2 APIs,
55:30
right. Potentially, right.
55:32
But but like ultimately, I don't know.
55:35
Like is this a race to everyone getting to the reservation cost
55:38
of a GPU? Like is that what's gonna happen
55:40
in a year? Like everyone's running the same
55:41
model. Why is this not gonna like
55:43
completely? Why is the alpha not gonna
55:45
completely disappear from this product?
55:48
The reason why, the reason why we're not just doing that is we
55:50
have our own models. We actually have our own ways of
55:54
quantizing, potentially quantizing the models, but
55:56
that's very specific to our workloads.
55:58
And we've tuned the training pipeline even for the
56:01
quantization step for our own workload.
56:05
But then so we can't use something completely off the
56:07
shelf. And then if we did, yeah, I'm
56:09
sure there's going to be a lot of providers that are going to
56:11
just do Luma 2, but I don't know why we'd pick one over the other
56:14
right now. Maybe maybe at some point we
56:16
would pick one. I will say I think we probably
56:19
have a better understanding of this than most companies, just
56:23
because we've been running these models longer than some of these
56:25
generative AI companies. Some of these info companies
56:27
have been running infra models for Llama too and stuff, but but
56:31
yeah. Yeah.
56:33
And it it very much is like you are focused very, very sharply
56:39
on one specific problem which is this coding problem as opposed
56:43
to hey, we're we're training this big ass model AKA GB D4 and
56:48
one of the things it does is coding or we have LAMA and even
56:53
though we didn't really do much training data with coding, we're
56:58
going to fine tune it and call it Codelama and it's going to be
57:00
pretty good, you know. So I do see the value in you
57:04
just saying like we want to be the best at this coding and we
57:10
are going to focus 100% on it and optimise every area that we
57:14
can for it. That's right.
57:17
Ultimately for us the only thing that matters is we have we have
57:20
an application that our users love and want to use every day.
57:23
That's like all that matters to us.
57:25
And yeah, like if the reality is we could run some logistic
57:27
regression in the background. That is not a complex model.
57:30
We do that in a heartbeat. Unfortunately, that doesn't cut
57:33
it. Like I don't think if we ran
57:34
logistic regression on the user's machine they would, they
57:36
would have a great product. You actually brought up before
57:39
the interview something about a company called Kite, right?
57:42
You brought up this is a company that did auto complete that went
57:45
bankrupt or maybe not bankrupt but dissolved.
57:49
I think it was a great product. I think the only problem was L
57:52
limbs or would have made these code assistance capable now and
57:56
they existed pre LLMS existing and I think if you didn't make
58:00
the jump you were not going to meet your users where they
58:03
wanted the product and that was what I believe happened.
58:06
And for us, we just want to build the best product that our
58:08
users, our users love and they got to be I I I really, I'm
58:12
excited when our users like how is this thing free.
58:14
That's that's what I'm always excited by.
58:16
Yeah, that's got to be the best compliment.
58:18
That is so cool. I mean, I've heard it like once
58:21
or twice with somebody just being like, wow, these podcasts,
58:24
all the the community, I can't believe you don't charge for
58:27
that, on charge for that. And I'm like how would I charge
58:29
for what we're doing here in the community, you know, like that
58:32
if anything we can get sponsors and we can charge them but like
58:35
to the end user no. And so the, the one thing that
58:40
I, I wonder about because I've been talking with people in the
58:44
LLM space for, I don't know, the last 6-8 months.
58:48
And there is ideas of like new software architectures that are
58:55
coming up right. And I think one thing that you
58:58
mentioned that would be cool to talk about is how you basically
59:02
are future proofing the work that you're doing at Codium.
59:06
And I was just talking to my friend Rohit about this and he
59:11
was saying like you know, a lot of the times we think about
59:14
wanting to be backwards compatible.
59:16
But he's trying to like bring this idea of like, how can we be
59:20
forward compatible so that if you are someone who is thinking
59:25
about bringing AI into your company, you don't have this
59:29
FOMO of like, Oh well, in six months the world's going to
59:32
completely change. And then everything that this
59:35
big project that took me 6 months to implement, the day
59:38
that it goes out, it's already going to be obsolete.
59:41
And so when you think about things like that, like how do
59:44
you make sure to cover your bat? Yeah, I think it's maybe two
59:50
things. So I've in the past, I worked in
59:52
the autonomous vehicle space where we were, at least when I
59:55
was there, we were never exactly there yet.
59:58
Like the tech was always just right around the corner.
1:00:01
And the thing I love to tell people, which is very, very
1:00:04
interesting, is. There are secular trends that
1:00:07
you can never beat. And when I started working at
1:00:10
neuro, like the GPUs that people had access to and that we were
1:00:13
running in, the cars had between 10 and 20 teraflops.
1:00:17
And now consumer grade GPUs have hundreds of teraflops, right?
1:00:19
There's a 50X increase in GPU capacity and a per GPU basis
1:00:23
that has happened over the last 60 years, right, Which is crazy
1:00:26
by the way. That's just nuts.
1:00:28
And these are, these are increases that have enabled
1:00:31
brand new models that were never possible to run six years ago
1:00:34
from now being able to run now. So all that goes to show that is
1:00:38
that this space is very exciting.
1:00:39
There will be crazy things that are capable of being run in four
1:00:43
years that we are not going to be able to anticipate today.
1:00:46
And don't design systems today that require a herculean amount
1:00:50
of engineering to make the ML models of today workable.
1:00:54
So that the end take away is don't build elaborate UIS and
1:00:58
all this other stuff to make crazy to make models today kind
1:01:02
of work well. Because the problem is what's
1:01:04
going to happen is a couple years from now the model will
1:01:06
get better and a lot of the things that you implemented are
1:01:09
not going to be nearly as useful.
1:01:11
They're not going to be as as pragmatic.
1:01:14
So build for technology that works today, that's like the
1:01:17
biggest thing that I always think of and when you build for
1:01:21
technology that works today, make sure that the technology
1:01:23
that you build is a primitive that can later compound.
1:01:26
And by that I mean you know there are many things that we
1:01:29
can sort of do when we when as a startup we can implement
1:01:31
features here, features there and every feature is, is some
1:01:35
work from our engineering team, right.
1:01:37
But if that feature is something that is ever agreed, right, if
1:01:42
we figure out ways in which our model is much better at applying
1:01:46
context and we are much better at fine tuning, as we improve
1:01:50
the base model, as we improve more contact sources, as we
1:01:54
improve all these other things, it's going to be a rising tide
1:01:56
lifts all boats, it will improve all facets of our system.
1:01:59
So that's an awesome kind of feature to invest in as a
1:02:02
company because the technology we're today, it's something that
1:02:05
if I improve the surrounding pieces, it'll continue to
1:02:08
compound on itself, right? But on the other hand if all I'm
1:02:12
doing is I'm adding UIS here and there, those are things that AI
1:02:16
believe the competition can copy very quickly, right?
1:02:19
Which is, which is not good for a start up like us.
1:02:22
But B also that is not something where like as the models get
1:02:25
better that's like gonna, you know, that's gonna improve my
1:02:27
product. I don't know if that that helped
1:02:29
there. Yeah.
1:02:32
My main take away from what you just said is the rate of change
1:02:36
is so fast that all this stuff that you think super advanced
1:02:40
and it gets you like a point or a percent better today, but it
1:02:45
takes you so much effort and it's so hard to do.
1:02:49
In 2-3 years, it's going to be laughable how small of a change
1:02:55
that is compared to just how everything has moved.
1:02:59
You know, 5200% more in the direction that you were trying
1:03:04
to go today, and you didn't need to put that Herculean effort
1:03:08
into it. Yeah, or most of the time I
1:03:11
think what happens is you end up putting in the Herculean effort
1:03:13
and it really still didn't get there, but you just put it out
1:03:16
there anyways. That's like him from Wisdom,
1:03:18
isn't it? You've had that path enough.
1:03:22
Yeah. Yeah, totally.
1:03:24
There's a lot of there's a lot of battle scars here for sure.
1:03:27
Dude, well this has been absolutely fascinating.
1:03:29
I really appreciate you coming on here and talking with me.
1:03:32
I love everything about what you're doing and I would
1:03:35
encourage anyone who is not trying Codium to go and give it
1:03:40
a play. It seems like me and my horrible
1:03:44
coding skills could even become an incredible coder.
1:03:48
I might start start shipping stuff over the weekend now that
1:03:51
I'm going to go have a play with.
1:03:52
Love it. Watch out world.
1:03:55
Here we go. This is.
1:03:56
It's so cool, man. So thank you, Varun, and we'll
1:04:00
end it there, man. Awesome.
1:04:02
Thanks a lot for having me. Hey, this is Mike Gold, also
1:04:09
Cofounder and CEO of Tecton. MOPS Community is the best way
1:04:14
to stay in the loop on the latest MOPS news and best
1:04:17
practises. It's also a great way to connect
1:04:19
with experts and get support from an amazingly helpful
1:04:22
community. Subscribe and stay in the loop.