Latest / Agentic Conversations (formally mlops.community) / AI's Next Frontier // Aditya Naganath // #277

Transcript

AI's Next Frontier // Aditya Naganath // #277 · Agentic Conversations (formally mlops.community)

0:00
Aditya Naganath I am a principal at Kleiner Perkins and I love
0:06
cappuccinos. Folks, welcome back to the ML
0:09
OPS Community Podcast. As always, I am your host
0:12
Dimitrios, and we are in for a wild ride in this conversation.
0:18
As you may have seen occasionally, I like to bring on
0:24
some VCs to hear what they think about the current state of the
0:29
market and the space. This is one of those times I got
0:34
to talk with Aditya about how he sees ML OPS tooling, LLM OPS
0:41
tooling, if you want to call it that, infrastructure and AI
0:45
infrastructure, and what the difference is between those two
0:48
because I needed a little bit of clarification on that and oh so
0:52
much more where he's excited about investing and where he's
0:55
not excited about investing. He made me rethink some of my
1:03
previous assumptions and I hope some of the stuff he said is
1:09
wrong for the sake of My Portfolio companies.
1:12
But you know what? He made some convincing
1:15
arguments, so let me know what you think in the comments or
1:21
give me some feedback. Tell me if you are seeing the
1:26
same thing that he is seeing or if you're seeing something
1:29
totally different. Let's get into it.
1:32
And as always, if you enjoy this episode, share it with one
1:36
friend. So for those of you that are
1:39
listening to the podcast and not watching it on YouTube, we've
1:43
got The Beautiful musical recommendation.
1:47
This one is Faith's Hymn and it is by Beautiful Chorus.
1:55
I'm just going to say, listen to these harmonies.
3:03
We are in a field that is very busy and you more than anyone
3:08
know that because you are investing in this field.
3:11
How does it feel to be an investor in the AI space these
3:15
days? Man, it's got to be wild.
3:18
Thanks for having me Demetrios. It is indeed wild.
3:21
It is crazy out there. And you know, the reason for
3:24
that is we just see so many kind of ideas emerging and at the
3:28
same time so many different start-ups tackling those ideas.
3:32
For any one interesting idea, you have 10 or 12 pretty solid
3:36
teams taking a crack at it. And you know, our job is to is
3:39
to pick once. And the question is how do you
3:41
do that? I hear people talk a lot about,
3:45
well, it's got to be the team. It's got to be the team.
3:47
Do you have any other insights that you tend to lean on or
3:52
biases? Dare I say it?
3:54
Yeah, I think, I think beyond the team, I wanted to see some
4:00
semblance of a novel approach, right.
4:03
And and ultimately the question is, what are you going to do to
4:07
rise above the quote unquote noise floor?
4:10
And and I think, you know, there are people who are just like,
4:13
hey, this is a pretty consensus opportunity and we want to go
4:17
for it. That's not that exciting.
4:18
I want to see someone actually put some thought into look like,
4:21
you know, there are 12 other pretty solid folks going after
4:25
this. But here's why I actually think
4:27
we have a chance and it kind of points to the details and the
4:32
points to say having put in more thought into the actual
4:35
underpinnings of a successful product.
4:38
And I can imagine a lot of times that has to do with go to market
4:43
motions. Go to market for sure, right.
4:46
Like, you know, if you think about at the end of the day,
4:49
what, what what it takes or it's a great team that has a great
4:51
product that has very good go to market, right.
4:54
So in that sense it's not rocket science, but you know, good go
4:57
to market follows a great product.
5:01
And then so to me it's still about what is going to be that,
5:04
that insight that drives a really novel approach that
5:08
yields a great product, which will then yield a great go to
5:10
market. Yeah.
5:12
It feels like one way to bet on a team or have a little bit more
5:19
success rate is if you know that the founders or the team has
5:26
some kind of advantage on that go to market and their motion.
5:31
Whether that's like the founder has a lot of distribution or is
5:34
well followed or has a community or has been very active in
5:39
communities, that type of thing just feels like, oh, there's an
5:42
unfair advantage here. And so you can't count that out.
5:47
For sure, for sure, you know, any kind of distribution
5:50
tailwinds that you can generate helps, but at the same time, you
5:52
don't want to be one of these flash in the pan kind of
5:54
companies that has, you know, steep revenue growth.
5:58
But then realise that, hey, look, that again, the
6:00
foundations were unstable and, and as a result, you know, two
6:04
years out, three years out, that company has has churned a bunch
6:07
of customers or, or that growth is, you know, a shadow of what
6:10
it was, right. So there's no substitute for
6:14
thoughtful product execution. And what about those companies
6:19
like a Lensa, where it's a fad and maybe in the hype, in the
6:26
peak of the fad, you're trying to discern if this is lasting or
6:30
if this is just a flash in the pan.
6:33
And first of all, I'll say consumer is hard, right?
6:36
And if you think about consumer in as a black box, it's all
6:41
about user retention because at the end of the day, you're
6:44
competing for people's time and attention, right?
6:47
And that's gotten so hard with obviously the Facebooks and all
6:52
sorts of other apps of the world.
6:53
So we pay a lot of attention to retention data.
6:58
Yeah, so there is a so consumer is is a beast.
7:03
I like how you put that it is a black box and when we're dealing
7:06
with AI, so you get the double black box and two wrongs don't
7:10
make it right. I can imagine how it's a bit of
7:13
a crapshoot. You're going to the casino with
7:15
some AI apps for consumers. But what I'm really interested
7:22
in talking about with you is the ML OP space, the AI
7:28
infrastructure space, and then the apps space or the
7:34
application layer as some people call it.
7:36
Because I want to know first and foremost, there was this
7:42
Cambridge explosion that happened with machine learning
7:46
platform tooling companies in let's say 2018 to 2021-2022.
7:56
And I would love to know what you need to see now from a
8:03
company that was funded in that stage to make you a true
8:10
believer. Of course, you know a great
8:12
business is one that that has a lot of revenue and is generating
8:16
that revenue efficiently. But but really what that points
8:19
to you is in the ML OPS world in particular, were they able to
8:24
show enough value to their buyer set such that that buyer said
8:28
wanted to buy the solution versus building house?
8:30
And if you kind of think about the ML OPS landscape as a whole,
8:34
again, from the perspective of a venture investor, a lot of those
8:37
companies haven't been great outcomes, right?
8:40
Like, you know, scaly eye is one that obviously is a great
8:43
outcome, but they were far left in terms of the the value chain
8:47
of of an ML OPS or or ML engineer, right?
8:49
Like they handled labelling data that nobody will want to do in
8:53
house, right. But if you think about all of
8:56
these companies that emerge with like, you know, EB testing your
8:58
model, like evaluating your model and you know, being a
9:02
feature engineering platform or what have you.
9:06
The sad reality is that they they couldn't really crack these
9:09
larger enterprise contracts because those enterprises had
9:13
teams that would build those solutions in house, tailored to
9:16
their own needs. Do you feel like there is still
9:21
too much custom white glove that needs to happen with the way
9:28
that the each enterprise has architected their ML platform,
9:34
you can't quite sell this standardised tool into the
9:38
platform. So you haven't seen such a large
9:44
take off like was anticipated in that 20/18/2020 era?
9:49
That's right. I, I, I still believe that.
9:51
And you know, even if you are able to sell something into
9:55
these enterprises, what I found is that you tend to capture a
9:58
small portion of budget because you fit into a narrow sliver of
10:01
that overall work, lower value chain.
10:04
And big challenge for these companies has been to the the,
10:08
the dream rather for these companies has been to actually
10:10
own a lot of that value chain. But again, because of the amount
10:13
of customization that's been required, it's been really hard.
10:18
How do you compare that with the data platform?
10:23
Do you feel like, because if in my eyes, what I've seen is that
10:27
there is a lot more growth there and it does feel like there are
10:30
companies that have taken off. So I'm always questioning, is it
10:35
the maturity of the ML platforms or is it that ML is such a small
10:42
Tam or such a small sliver? And it is a very, as you
10:47
mentioned, unique use case. But almost everyone is going to
10:52
have a data platform, whether you're doing ML or you're doing
10:55
analytics, whatever it is. So there's a there's a larger
10:59
Tam there. Yeah, great question.
11:03
And I do think it's a larger Tam, right?
11:05
And if you look at the companies that have really broken out,
11:07
whether it's data Bricks and Snowflake, etcetera, they've
11:10
really made their money off of compute or storage.
11:14
And that's really anchored to this trend of data only
11:17
exploding in the enterprise, right structure.
11:19
And then you have unstructured data that's 4X's largest
11:22
structure as structured data. And you know, in the case of
11:26
data Bricks, they sold a compute fabric, right?
11:29
Ultimately, Spark was a better way to do computations over
11:33
large swaths of data, essentially through this
11:35
innovation called the RDD, which allowed you to do in memory
11:38
computations and they monetize that, right?
11:40
So they were in the flow of compute.
11:42
And then with Snowflake, it was, you know, hey, look again, we'll
11:45
just give you a way more efficient warehouse on top of
11:49
again, you're growing swaths of data.
11:51
So they anchor themselves that a trend.
11:53
Now you've also had to your .6 pretty successful companies that
11:58
have, you know, existed in that value chain of, you know, data
12:02
operations, right? So DBT being a shining example
12:05
where they've owned, you know, transformation.
12:08
But you know, I would say the jury's out on how big DBT will
12:11
be as a company, but it but it's still a much better outcome
12:13
compared to a lot of these ML OPS companies, right, that have
12:17
also tried to go after, you know, analogous work flows.
12:20
So, yeah, overall a a bigger market because data was only
12:24
exploding and people across the board needed to do interesting
12:28
things with their data. While in the ML OPS world, to
12:31
me, it feels as though, you know, these ML OPS, these ML
12:34
practitioners, they were doing a bunch of custom things with
12:38
their models. But ultimately, it's unclear
12:40
what that, you know, what that secular trend was for a startup
12:45
to go and anchor themselves against beyond, by the way, what
12:48
scale AI did, which of course, you know, they were very smart
12:51
in identifying that every model is going to be powered again by
12:54
surprise, surprise, Le Baldito. Yeah.
12:58
So knowing that and I appreciate these almost like cold buckets
13:06
of water that you're pouring on me right now.
13:11
And I wondered, being equipped with that vision, do you try to
13:20
dig deeper into the new ways that compute and data slash
13:25
storage are coming out? Because I know there's no
13:28
shortage of companies that are trying to reinvent the ways that
13:34
we're using compute or just take down Spark, if we want to be
13:38
blunt. Totally.
13:39
So I, I presume you're, you're, you're leading to this whole AI
13:44
wave, right? And the reality is that they
13:47
have introduced 2 new workloads of note, right?
13:49
LLM training and inference. And what's happened is you can,
13:53
you can think of the infrastructure layers being
13:56
bifurcated across LLM OPS companies, right?
13:59
And these are the companies of the same ilk as these ML OPS
14:02
companies that preceded. And then you have companies that
14:05
are true infrastructure across, you know, compute storage and,
14:08
and we're not seeing emerging loss of networking companies.
14:11
So what I've found is that, you know, the compute layer has been
14:15
quite interesting to invest in. And in fact, you know, we have
14:19
Kleiner Perkins invested in a company called together AI,
14:22
because it turns out that you have on both obviously LM
14:26
training and inference, there's a tonne of demand right from
14:29
these models. And so anyone who can figure out
14:32
how to, you know, serve that compute at a cheaper rate and,
14:36
and with lower latency has a chance of being very valuable.
14:39
They essentially again, monetize through compute.
14:43
Then there's a company called vast Data, which is finding
14:45
themselves in the, in the data centres themselves and
14:47
essentially being the hub for, you know, again, petabytes scale
14:52
of, of training data and, and they have done very well as a
14:55
result. The jury is still out, I would
14:57
say, on the LM OPS companies, right.
15:00
And you have a tonne of companies going after, say,
15:02
evaluations and, you know, we're seeing companies try to be in a
15:05
data curation platforms and we'll see like, you know, I, you
15:10
know, we haven't invested in that layer.
15:11
And and again, I think that the big question is, you know, will
15:15
there be one that is able to break the, you know, build
15:18
versus buy mould and and also, you know, be able to rise above
15:23
all of these other competitors that look pretty identical to to
15:26
each other? Talk to me more about the LLM
15:30
OPS paradigm and how you have sat out up until now.
15:39
You're saying we'll take the wait and see approach because
15:42
probably it a lot of things look familiar from the ML OPS.
15:47
Is it again that you're looking at a small Tam?
15:51
Because when I look at LLM workloads in this, the AI
15:56
infrastructure IA 100% agree with you where there's been the
15:59
Cambrian explosion, it's a lot of similar tools.
16:03
And that's not very inspiring. But I do feel like you have a
16:10
much broader user base because you're not only getting the ML
16:16
practitioners, you're also getting a lot of software
16:18
developers who now are playing with AIAPIS.
16:23
And they can go out and they can put a few API calls together.
16:29
And now they're an AI engineer, which in the ML OPS era, you
16:35
didn't have that. You were looking at a much
16:39
smaller subset of the engineering world.
16:43
Yeah. When I actually think about ML
16:47
OPS, yes, you had your machine learning engineers who were the
16:50
guys behind a lot of the interesting models of that of
16:53
that world. But you also had software
16:55
developers as well who were able to, you know, you know, write
16:59
Python And, and make a call to XG boost and, and get some
17:02
interesting outputs, right. And so in the LM OPS world, what
17:08
I think has happened is you have, you know, people coming up
17:11
the maturity curve, right? And so again, LMS are, are a
17:15
black box. That's what's different.
17:17
They're non deterministic. And so compared to, you know,
17:21
some of the typical machine learning models, there is, you
17:24
know, a question around reliability and, and how do we
17:27
work with the outputs of these things provide which, if you're
17:31
a bull can, you can make the case that you will need
17:34
scaffolding tools to, to work with these things.
17:38
And then the question is, will people want to build those tools
17:42
in house or just buy off the shelf?
17:44
Because again, all they care about is, you know, the use case
17:47
and the problem they're solving. At the same time, you know, the,
17:51
I would say the, the question is, what is the scope to
17:54
differentiate yourself, right? Because let's take again, this,
17:58
this really hot category of LLM evaluations, right?
18:01
A lot of teams have identified this pretty, you know, obvious
18:05
problem that, hey, LLMS are not always reliable.
18:08
Their outputs will will many times drift against your key
18:12
objective and sometimes it's not even knowable why they drifted
18:15
against this key objectives. So can we first start with, you
18:18
know, just giving you observability and then over time
18:21
worker work our way into helping you, you know, fine tune, you
18:24
know, re prompt your model to to be better.
18:27
The question to me then is, you know, what is someone going to
18:30
do that's markedly different from from, you know, the 12 or
18:32
15 other companies going after this problem?
18:36
And then I think the second thing to keep in mind is look
18:39
like a couple of years from now, like people are going to be that
18:41
much more aware of how to work with these things.
18:43
There's going to be that much more knowledge disseminated
18:46
around LMS that engineers will read and, and get smarter about.
18:50
And then so when they do, you could still have a situation
18:54
where people build in house because they have custom needs,
18:57
right? So I think that's my concern
18:59
with that overall market opportunity.
19:02
It, it, it doesn't feel like, it doesn't feel like it's probably
19:06
big yet. And on top of that, it it, it
19:08
feels like it's getting chopped up in many different pieces.
19:13
So now I I have to ask why is together different?
19:18
Because it does feel like together is in that space and
19:22
and we stay together because you guys invested in them, but we
19:25
can also throw out just so that we're not completely biassed.
19:28
I hope you allow me to humour you.
19:30
There's a few other tools that are in the same space, right?
19:34
Like I would consider modal part of that space and base 10 and
19:39
maybe beam dot cloud, maybe lepton and run pod.
19:45
Do you consider them? Yeah, yeah, absolutely run pod
19:49
fireworks, right. So you're absolutely right.
19:51
There's there are a tonne of. Fireworks.
19:53
I forgot about them. Yeah, A.
19:54
Tonne of folks who are who are also operating at that compute
19:58
layer right, which AI would still demarcate as separate
20:02
versus the LLM OPS category? So that's number one.
20:07
Because I, because I think of them as actually core providers
20:10
of, of infrastructure in some sense, right?
20:13
They are ultimately providing compute.
20:16
While to me LM OPS is more about, hey, are you providing
20:19
some tooling that, that kind of works around the LM?
20:24
So it feels one layer lower to me.
20:28
OK, so hold on, let me run this back to you because I, I think
20:32
this is the first time that I've thought about it that way.
20:35
You can use AI and not have AAI infrastructure company like
20:41
these LLM OPS tools that you're talking about, but you can't use
20:44
AI and not have a GPU. That's right.
20:47
That's. Right.
20:48
I see. Yeah.
20:49
All right, All right, It, it makes a lot of sense now why
20:52
those folks are in their own category as opposed to all the
20:56
rest like the evaluation or the orchestration tools.
20:58
That's right. It's, it's just about the layer
21:00
of abstraction that that the, that you're operating at.
21:02
So coming back to your question, right, there's a tonne of demand
21:07
for compute, right? And that's because on one end
21:10
you have people who are training their own models and that takes
21:13
a tonne of compute. And then once these models are
21:15
trained, you obviously have them being slotted into applications
21:19
and then running inferences as a result, which is again, compute
21:22
on a GPU. So the claim that or the work
21:26
that folks like together are doing is, hey, you know, we will
21:29
provide some really, really interesting pieces of software
21:33
that optimises the utilisation of these large scale GPU
21:37
clusters, which then allows you to be, you know, at the kind of
21:42
the apex of this, you know, cost and latency curve or Frontier.
21:47
And if you mentioned certain other companies, right?
21:51
And, and some of these companies, what they're doing is
21:53
they are acquiring GPU's and then kind of passing them
21:56
through to the end user without much of those optimizations
22:00
built in via software. I think that's one thing to
22:03
note, right? Like ultimately, if you again,
22:04
look at the work that companies like Together are doing, they're
22:08
doing deep research right across training and inference.
22:11
You know, for example, Together has a chief scientist named Tree
22:14
Dow and he's the inventor of this thing called Flash
22:16
Attention. And it's basically a way to
22:17
train our lens more efficiently, right?
22:19
So he's commercialising this into Together's platform.
22:22
Then, you know, the inference side, you can do some pretty
22:24
nifty things with speculative decoding and what have you.
22:27
And ultimately what this boils down to is can you offer a
22:30
compute more cost effectively without compromising latency and
22:36
still have pretty good economics.
22:37
I think that's the the key piece to understand here.
22:42
So all right, I, I can see that now let's, let's go one level
22:47
up, maybe 2 levels. If we're coming from the GPU
22:51
infrastructure area, then we go, we bypass the whole LLM OPS area
22:58
and we go up to the application layer and I know that you've had
23:03
some excitement around there and it feels like that's where
23:07
things are happening. We just got done with the Agents
23:10
in Production virtual conference.
23:12
There was a tonne of energy around it, like a tonne.
23:16
It feels like it is the new thing.
23:21
Everyone's learning together. We had some great people that
23:26
were sharing their knowledge, but nobody is an expert in this,
23:30
right? Nobody's been doing it for more
23:32
than a year. And even the team that we did it
23:34
with process, they started their journey with agents, which
23:39
really was in the beginning, it was just a rag.
23:43
And that started two years ago. And it was about a year ago that
23:48
they really started going with the agents.
23:50
But that's one area of what feels like the application layer
23:56
and how people are using that. I know it feels like 1 aspect
24:04
that has been very clearly defined to have business value
24:08
when it comes to agents is the customer support.
24:12
Have you been seeing other areas where it's obvious that's going
24:18
to be an AI application area and we need to bet big on it?
24:23
Absolutely. So the application layer, first
24:26
of all has been the most exciting to us at this moment in
24:29
time as as investors at at Kleiner Perkins and you know, to
24:33
directly answer your question, software development has emerged
24:37
this pretty obvious application of LMS, right?
24:40
And you know, for now, it's this auto complete use case, right?
24:44
Which is how do we essentially provide useful suggestions to
24:48
developers within their IDE that allows them to move a lot
24:51
faster? And, you know, it's a pretty
24:55
beautiful use case if you think about it, especially in the
24:57
context of being a wedge. Because on one end you had first
25:02
of all, get up copilot, educate the market back in 2022 about
25:07
why this is even useful, right? Which which was very, very
25:11
helpful to a bunch of startups that emerged in 23 that haven't
25:15
had to pay that tax as a as a result.
25:17
And they've grown very quickly. But the other things to know are
25:20
one, you know, it's a it, it's a use case where the LM will get
25:25
it right, let's say 30% of the time.
25:28
And when it gets it right, it's a very helpful thing for a
25:31
software developer. They can move a lot faster.
25:33
But if the suggestion is wrong in that 70%, it's not
25:37
devastating. The developer just doesn't, you
25:39
know, tab complete and then they, they try to figure it out
25:42
on their own. And so as a result, you have
25:45
this use case where there's a pre provable productivity gain
25:49
with at the same time, you know, tolerance for error, right?
25:53
So every CIO has now decided that they need to equip their
25:56
software developers with some kind of coding accelerator.
26:00
And that's opened up a huge market, right?
26:01
As a result, I, you know, so we have a company called Codium in
26:05
our portfolio that's actually leading the charge here.
26:08
Yeah, Varun, that's right. Came on here and that was I
26:14
think almost a year ago. And after we stopped recording
26:19
he was like, yeah, man, we used to be in the GPU space, but we
26:23
got out of it and it's all for the better right now.
26:26
We're doing really well. That's right.
26:28
You know he is operating in, in I would argue like the poster
26:32
child market for this first wave of application companies.
26:35
It's one of those markets where you can talk to any enterprise,
26:40
any CIO and that enterprise, no one's going to disagree that
26:43
they need one of these things, right?
26:45
So we're, we're very, very privileged to be in business
26:48
with, with someone like Varun who's building codium.
26:50
And then I would say the other market is enterprise search,
26:53
right? And, and so we are in business
26:55
with a company called Glean that's leading the charge here.
26:58
And that's another interesting case study where, you know,
27:01
glean, first of all, the the team is phenomenal.
27:03
The CEO, Arvind, he was one of the Co founders of a company
27:05
called Rubric. They have one of the most
27:08
technically stacked teams out there.
27:11
When ChatGPT came out in 22, it was obviously, you know, the the
27:16
watershed moment for for AI. And I would say what was
27:20
interesting about it again was that a lot of CIOs were like,
27:24
Hey, I want this on top of my own data.
27:27
So I ended up being like a, you know, again, market education
27:31
phenomena where open EI was pouring all these resources into
27:35
making ChatGPT a consumer product.
27:37
You had these CIOs who obviously were fiddling around with
27:39
ChatGPT saying, Hey, it would be great if we had this on top of
27:42
our our own data. You know, glean had already
27:44
built a bunch of these connectors to applications and
27:48
data sources. And we're very quickly able to
27:50
launch their own assistant that was privacy aware.
27:53
And you know, it, it's been lights out from there.
27:55
So that's been another use case that's emerged as as pretty
27:59
consensus at this point where every enterprise feels like they
28:01
need it. And then we have seen, you know,
28:04
certain other use cases start to emerge and be very interesting,
28:08
right. So we're in business with a
28:09
company called Synthesia. They provide avatars for the
28:12
enterprise and we can we can talk about that.
28:15
We have some companies in stealth that I would say I'm
28:17
very excited about because they're using LLMS in a smart
28:21
way in conjunction with well thought, our product to, you
28:25
know, go after workflows that were previously like
28:29
unaccessible where you could have only owned say 70% of that
28:32
workflow. But now, again, by infusing the
28:35
product with LMS, you're able to go after the entirety of it
28:38
because some of those workflows involved, you know, reasoning
28:41
over unstructured data and now you finally can't, right.
28:44
So that's that's what I'm excited about, at least from
28:47
this first wave. But as I'm sure we'll get into
28:50
you, this is only a stepping stone for what's to come, right?
28:53
You know the the whole dream for this AI movement is can you get
28:57
these things to be with your reason and well enough such that
29:00
they can resemble resemble a human being and as a result be a
29:04
Co worker versus a copilot. Hey folks, quick break because
29:08
I've got something you'll want to hear about.
29:10
My friend Hugo Bound Anderson has a fantastic new podcast
29:15
called High Signal produced by Delfina.
29:18
If you're working with data, this is your new must listen.
29:21
Hugo's already hosted some incredible guests.
29:23
Michael Jordan from Berkeley shared how AI could tackle
29:26
global challenges. Fascinating stuff about scaling
29:29
solutions. Andrew Gelman from Columbia
29:32
broke down his practical tips for better statistical thinking,
29:36
the kind of stuff you can use right away.
29:39
And coming up, you'll hear from tech pioneer Hillary Mason.
29:43
The podcast is all about giving you high signal on how to build
29:47
your career and make smarter decisions, whether you're just
29:50
starting out or leading data-driven teams.
29:53
There are 5 episodes live now, so check it out wherever you get
29:56
your podcast links in the description.
29:59
All right, back to the show. Yeah.
30:01
And I want to get into that Co worker versus Co pilot because I
30:04
think that's a great way of putting the current versus the
30:08
future scenarios. But first, I'll address a few of
30:12
these points on just starting with Codium.
30:15
And when Varun came on here, he was the first person to talk
30:19
about how much easier of a problem evaluation is when it
30:24
comes to code versus when it comes to unstructured data of
30:30
language. And he's like, if it runs, then
30:33
you're kind of good. It may not be the most
30:36
efficient, it may not be the best, but it still runs.
30:40
So if if it compiles and you're happy, then the other piece that
30:45
I always think about when it comes to the coding tools and
30:50
especially the AI coding tools is a segment of the evaluation
30:57
metrics. If I were running it right, I
31:02
would be 100% looking at how much of the code that I
31:07
suggested is still in production after three months or six
31:11
months, because that's how you can know like the quality of
31:15
this spaghetti code or non spaghetti code that you're
31:19
spitting out. Then moving on.
31:22
Let me just remember what else you were talking about.
31:25
It was the next one that, oh, Glean completely took over the
31:29
market. So The funny thing for me is
31:32
that is it's almost like there's Glean or there is you're doing
31:37
it yourself. And I know there's probably a
31:40
tonne of other tools in there, but what has been so amazing to
31:45
me is how Glean took everything and just ran with it.
31:50
And every time you hear about them, they're raising another
31:52
hundred $200 million. And so it's outrageous.
31:57
But the chat with your data is the quintessential hello world
32:01
for AI. So the fact that Glean has had
32:06
so much success when going back to what we were talking about
32:11
earlier, how hard is the problem that you're dealing with?
32:14
You would think that it's not that hard because everybody is
32:17
creating this chat with your data.
32:20
But then you recognise that chat with your data when you have a
32:23
few PDFs versus chat with your enterprise data and across all
32:28
of these different sources that and making sure that the right
32:33
people have the right access to the data.
32:35
And then you are constantly updating this access and
32:39
constantly updating the data that's inside.
32:42
That's where things get a little bit more difficult.
32:45
And so I understand why Glean took the market on that one.
32:50
Yeah, that's a great summary. That's a great summary.
32:53
Well, so the companies that are infusing it.
32:57
I also wonder about how you feel when it comes to the no code,
33:02
low code with AI and LLMS. Because I've seen quite a few
33:08
use cases where you get someone that is on the sales team or
33:12
that is in HR or they're on the marketing team and they're able
33:16
to create immense value for themselves.
33:19
Because they spend a weekend or they spend maybe two weeks
33:24
playing around with one of these low code, no code tools,
33:27
recognising that they can automate, like you said, 70% by
33:33
themselves with their own workflow that they like and
33:39
they're able to just hit the ground running.
33:42
That's right. Yeah.
33:43
We one of the beautiful things about this movement is that, you
33:46
know, LMS again, are are a higher form of abstraction.
33:53
And, you know, before if you had to unbundle them, you'd have
33:56
probably had to write code to sort of simulate the outputs of
34:00
what you're getting to do with LMS. But today all that is now
34:03
has now been pushed into the weights of these models, which
34:06
basically means that non-technical folks can do a lot
34:10
more, right? Because you have LLMS that have
34:14
that get you pretty far along. So as a result, they're able to
34:18
build and construct, you know, bespoke applications, bespoke
34:23
workflows without any help of a developer, without needing to be
34:26
technical, because LLMS now get them pretty far along.
34:31
Yeah, I do see different people wanting to take some of these
34:38
low code no code solutions or features or whatever they create
34:41
and then infuse it into the product that they're working on.
34:46
And that's where you get this gigantic divide because you go
34:49
from a low code, no code MVP type thing that works when you
34:53
hit run and there's one or five people on the team and they're
34:57
using it sparsely versus this is now a feature in your production
35:01
application, right? And so that divide and that
35:05
disconnect, I, I think is still not super close, but who knows
35:13
where we'll be a little bit down the line.
35:15
And the, the, the other thing that I was going to ask, just
35:21
when you said LMS and it brought up in my mind, how are you
35:28
feeling about LLM providers in general?
35:32
I would say it's just like an interesting battleground there,
35:36
right? So, you know, on one end, you
35:39
know, you have open AI and Anthropic that are pushing the
35:41
limits of research and coming up with all sorts of new, you know,
35:45
methodologies from the way they handle data to the way, you
35:50
know, obviously the algorithms that they're using under the
35:52
hood. And then, you know, I think they
35:55
also even want to go down to the data centre level and and kind
35:58
of own the optimization of compute and networking even at
36:01
that layer. But then you have, you know, the
36:04
matters of the world that are, that are using a bunch of the
36:07
free cash flow that that their core business generates to outdo
36:10
these guys, right? And so it's, it's, it's very
36:14
interesting to see. And so far, you know, by all
36:16
benchmarks, open AI and Anthropic are still ahead of the
36:19
open source ecosystem, but that gap is only closed with every
36:23
epoch of model released. And and, you know, again, like
36:26
coming back to together together is anchored on the fundamental
36:29
belief that open source models would be good enough for the
36:32
enterprise such that, you know, enterprises will will use them
36:36
at large. And then we'll need to find a
36:38
way to serve them efficiently and as a result, use platforms
36:41
like together. So it's a very interesting
36:44
battleground. And you know, again, like, as we
36:48
know, models are the fastest depreciating asset.
36:51
You pour all this money into them and they're impressive for
36:55
a couple of months and then someone else comes out with with
36:57
something that's better or better according to a certain
36:59
type of benchmarks. And then you can ask the
37:01
question here, like, was that investment worth it?
37:04
What's interesting with open AI strategy is that they, they've
37:07
been smart about it. They've, they've been
37:09
integrating their models into ChatGPT, right?
37:11
And ultimately and, and monetizing via subscriptions,
37:14
which is, you know, higher quality revenue than than API
37:17
calls. And that is a good strategy
37:22
because at least you know, you can, you can make the ROI case
37:27
for for model training and, and for having to, you know,
37:31
essentially burn more cash on, on bigger and bigger models.
37:35
But there is this open question as to will the scaling laws hold
37:39
and then what happens if they don't and how exposed of the
37:43
model provider companies if they don't hold?
37:45
Exactly. And that's where I'm always
37:48
wondering is we're gonna hit that point of diminishing
37:52
returns. And then I like the idea of,
37:55
yeah, they're exposed. Now you find out that when the
37:59
tide goes out, who's not wearing their clothes?
38:02
Exactly. And it's a scary thought for
38:05
these companies, right, because again, all of their competitive
38:08
manage thus far has been from being able to provide these
38:11
models with, you know, billions and trillions of parameters.
38:14
But you know, if we're if we're reaching this saturation point
38:17
for reasoning right, then it's scary, right?
38:22
And, and, and again, an open EI has been smart by at least
38:25
having this kind of brand value and, and distribution with which
38:28
Archie PT that will still allow them to be a sustainable
38:31
business. I think Entropic needs to think
38:33
about what their angle here is. But you know, to me, I think the
38:39
two interesting questions are one, you know, is this test time
38:42
compute approach the next frontier or do we actually just
38:46
need to have something that goes beyond the transformer?
38:48
The transformer has been very successful, right?
38:51
And we stretched it to its limitations.
38:53
But do we need something new? Which has been what I've been
38:58
thinking a lot about and in the same camp.
39:01
But you're almost expecting like a potentially like a once a
39:09
lifetime event to happen because yeah, we need to go beyond the
39:14
transformer. That's probably, I imagine when
39:18
deep learning was all the rage in the 70s and then the 90s, in
39:22
the 2000s, early 2000s, there was that same kind of idea.
39:27
And it happens at least decades apart.
39:33
So thinking that all right, this is going to happen before these
39:37
companies get exposed is a really wishful thinking in my
39:42
eyes. And and I also wanted to just
39:46
mention that there are these other companies out there like
39:50
the Mistros and the Coheres. And I really wonder what's going
39:54
to happen to them, right? If the Anthropics and Open a
40:00
Eyes are going to get burnt, then where is Coherent Mistral
40:06
in the conversation, if at all? Totally.
40:08
Yeah, I think that's that's a big, it's a big question for
40:11
them and their investors. And, you know, they were always
40:14
playing with fire a little bit by trying to essentially outdo
40:17
these guys from a model competence standpoint.
40:21
And you know, it's, it's always been hard for these independent
40:26
providers to to outdo these these models.
40:28
And they may have, you know, their moment in the sun for a
40:30
couple of minutes, but but it's quickly washed away.
40:33
So I'd be worried if I were them.
40:37
Yeah, yeah, yeah. So all right, getting into the
40:40
idea of Co pilots versus Co worker, what are your main
40:46
thoughts or assumptions on how that's going to play out?
40:50
Yeah. Look, again, I think coming back
40:52
to this point about reasoning, it's about how do we, how do we
40:55
cross the bridge from what we have today to, you know, the
40:59
next level of reasoning? Because ultimately, again, this
41:02
whole AI wave is exciting if we can stretch the limits of
41:06
reasoning and get these things to act like human beings and do
41:10
productive knowledge work as a result.
41:13
And to me, a couple of just underpinning concepts that I've
41:17
been thinking about is, you know, how do you, how do you
41:21
essentially leverage things like reinforcement learning, right?
41:24
And how do you leverage techniques like, you know, Monte
41:27
Carlo tree search and what have you that have been successful
41:30
in, in things like A0 for for chess playing as an example, and
41:34
kind of bring that to these models to get them to be able to
41:37
be self aware, reflect intelligently prune a search
41:42
space, right? That to me is the next frontier
41:45
reasoning. And I think I think we need to
41:49
solve this reliably to unlock these Co worker products, right.
41:54
And if you can unlock these Co worker products, then you can
41:57
start pricing as a fraction of, you know, wage spend, which is
42:01
then when the markets become really big.
42:02
Because again, if we look at these these really exciting AI
42:06
companies today, today they're great.
42:08
And in some cases they've created new categories out of
42:11
dust. But they're SAS companies,
42:13
right? And and that's fine.
42:15
Nothing wrong with SAS. But you know, for all the
42:17
investment that's pouring into to AI companies, if what we're
42:21
getting out of it are SAS companies with pretty known
42:23
multiples, it's not going to end well for a lot of investors.
42:26
So you actually need these Co worker products to start hitting
42:30
the market to justify the investment because that's when
42:33
they start tapping into a much, much bigger pool of dollars
42:36
which is essentially spend and labour.
42:40
So a bit of a tangent from what you just said that has
42:44
constantly been going on in my mind is if you think about agent
42:50
companies right now, it used to be back in the day when you had
42:56
engagement and you had people that were using your product,
42:59
that was a very good thing. But now because of the way that
43:04
the pricing structures are, if someone is using your product,
43:09
the more that they use it, that cuts into your profit margins.
43:13
And I saw this very clearly when we were doing the agents and
43:16
production conference and they, the guys from the OLX team were
43:20
showing me their new agent OLX Magic.
43:24
And they were giving me this demo.
43:26
And I thought it was hilarious because I said, OK, now I'm
43:30
thinking about buying a new car. Let's.
43:32
And they said, yeah, yeah, we can do that here, watch new What
43:36
kind of car? And I told him Toyota Hybrid.
43:39
So we typed that in and it was based in Poland, so it was in
43:44
Polish Schlotzsky's and forgive the pronunciation for anybody
43:48
that's based in Poland because I love calling them Schlotzsky's.
43:52
Just go with it. All right.
43:54
So then I said, well, can you how much is that in euros?
43:58
And they said, Oh yeah, yeah, we can ask that.
44:01
So they ask we get the prices in euros, and then I said, OK,
44:07
we'll now ask how can I convert a Polish licence plate to a
44:12
German one because I'm in Germany.
44:13
And they go, no, no, no, we have guardrails on that.
44:16
You can't just use this like ChatGPT, man, give me a break.
44:18
And I'm like, but that's related to my question.
44:21
Like if I'm going to buy a car in Poland, I want to know all
44:24
these things. So they said, well, yeah, but
44:25
you got to go to some other provider for that.
44:28
We can't just be basically giving you this for free.
44:32
And so that was a huge click in my mind on when you're building
44:36
these products, you can't let the user do everything that they
44:41
want or you don't have that ability per SE.
44:45
Because if you're rolling it out to millions of users and then
44:50
each user is doing on, use 7 or 8 calls to a model, that can add
44:59
up really quickly. It's a great point.
45:02
It's totally a great point. It's it's like every inference
45:05
is essentially, you know, cogs, right?
45:09
You heard you. They're asking another question.
45:12
Oh, it's so in a way you're like, yes, I want them to use
45:15
it. And they told me they have a
45:17
metric that if it's two or three, it doesn't do anything to
45:23
conversion. And if it's after 7, it also
45:27
doesn't do anything to conversion because it's just
45:29
somebody that's playing around with it and trying to see what
45:31
it can get. But from that three to seven,
45:35
that's the sweet spot. And they see conversion really
45:39
went up. So it made it worth it for them.
45:41
But I am constantly thinking about these seven plus where
45:46
maybe you do want to know all these answers and, or do you
45:49
have all these questions, but they, they are trying to think
45:53
about ways to cut you off from after you've asked 7 questions.
45:57
They're like, yeah, actually this is down for the moment.
46:00
Yeah, it's a, it's a great point.
46:02
And it kind of if you walk that backwards, it actually points to
46:06
how under optimised we are at the data centre level, right?
46:09
Because ultimately these applications are being attacked
46:12
to the model providers in the form of API calls.
46:15
Why are those model providers charging what they are?
46:17
It's because they have their own underlying costs with how much
46:19
it took to train the model and and serve it, right?
46:23
And then it points to you, OK, then how is model training
46:25
happening under the hood? It's so under optimised, right?
46:28
Like you essentially have, you're burning a lot of compute,
46:32
right? And you know, alternately, the
46:35
networks, at least at the moment, in terms of the way
46:37
they're designed are pretty under optimised for these kinds
46:40
of workloads. And that's only pushing up
46:42
costs, right? So we and then GPUs are
46:46
expensive because of the whole supply demand imbalance.
46:50
And we just need a lot of pieces to come together at many
46:53
different layers, right? Like AI would say, we need like,
46:56
you know, some challengers to Nvidia's dominance that we can
46:59
bring overall prices of compute down.
47:01
We need network fabrics to be a lot more efficient, right?
47:05
So that LM training costs go down.
47:07
And then you just need to be able to have these, you know,
47:12
chips get ever more powerful is that you can rely on a few of
47:16
them, which again will push LM costs down and that trickles
47:19
them all the way up to lower costs for for a model to be
47:24
served. Yeah, and lowering reliable,
47:27
like the GPU's are so unreliable.
47:29
And I think my favourite way of hearing this was when Todd
47:33
Underwood gave a talk at the AI Quality conference.
47:37
We did and he said you can look at a GPU the wrong way and it'll
47:41
go offline. You can walk around a data
47:44
centre and because of the way that your feet are hitting the
47:48
ground, you just blew out a whole cluster right there.
47:52
And obviously he's making a joke, but it shows you how weak
47:56
these are. And if you read any of the
47:58
blogs, you can see how GPU is going offline is going to happen
48:04
no matter what and folks are just trying to figure out how
48:08
they can deal with it and quickly get them back online or
48:13
minimise the blast radius when it goes offline so that they can
48:16
switch those training jobs quickly to other pools.
48:20
So true. And it's, it's, you can imagine
48:22
how complex this problem is at scale, right?
48:24
When you have, you know, you're trying to train the biggest
48:27
models that have been distributed across thousands of
48:30
GPU's. And, you know, you, you first of
48:32
all are trying to get Max out the utilisation, the FLOPS in
48:36
particular from these GPUs. And if it goes down, then it can
48:39
actually impact entire training runs, right?
48:42
You, you, you really can't afford for them to go down.
48:44
So, yeah. And that, that by the way,
48:48
contributes to higher costs, right?
48:49
So the IT to me it feels like, you know, training costs are
48:52
like, you know, 3-4 X what they, what they could be if we had
48:57
more reliability of GPUs. Well, knowing all of these
49:01
inefficiencies in the training in the GPU's, does that not make
49:05
you want to go and try and invest at that level?
49:09
Or is it because there's only a handful of model trainers?
49:14
It's like, yeah, I don't know if it's worth it.
49:18
I do think investing close to the data centre is interesting,
49:21
right? And and again, I think venture
49:23
investors who put money into a company like Vast have, have
49:26
done well or they're pretty happy.
49:27
And then again, you know, people who've invested in some of these
49:30
compute providers have done well.
49:32
So there's something there. I actually find networking
49:36
pretty interesting. I actually feel like for the
49:38
first time in a long time there, there's a clear why now to
49:42
invest in in a networking company, right?
49:44
Like the way, you know, at least the hyperscalers are are
49:47
approaching LM training is they're literally provide
49:49
provisioning separate networks just for LM training.
49:54
But even still, that's inefficient, right?
49:56
Take the hardware could be a lot better than what it is today.
50:00
There's a lot of talk about, for example, whether network
50:02
interface cards should be smart. There is a lot you could do in
50:06
terms of, you know, how you provision a network and
50:10
essentially how do you get it to be a lot, you know, higher
50:12
throughput, higher bandwidth. So there's something interesting
50:16
there for sure. Go to market will be very tricky
50:19
for companies operating at this layer and I think that's what we
50:22
have to factor in as investors. What?
50:26
Because you have, you first of all, have on one end, you're
50:29
selling to a pretty narrow customer set.
50:31
That's one. The second is you do have
50:34
existing players, right? You have, you know, the
50:36
Broadcoms, you have the Aristas, you have even to some extent, I
50:41
guess like a Cisco that are trying to move into this space.
50:44
You have some of these guys that are also doing their own work,
50:47
right? Like Azure and, and, and I, I
50:50
even know Facebook, it's kind of like investing in, in innovating
50:54
themselves at this layers. Then you know, what is it that
50:57
you're going to do as the quote unquote wedge, right that all
51:01
these guys buy from? And then what does that get you?
51:03
I think that's those are going to be two interesting questions
51:05
to answer. But do you think there are
51:11
credible solutions out there that are able to increase the
51:16
throughput? Because I I can imagine if you
51:19
did have something that you could just show, hey, look 2X in
51:23
the throughput here, you're golden.
51:26
Yeah, I think people are working on this.
51:28
I wouldn't say there's some there's a silver bullet just
51:30
yet, right, Because you can show this at like, you know, sub
51:32
scale, but can you prove it at that kind of scale?
51:36
I think that's. Hundreds of thousands of GPU's,
51:39
that's right, is a whole different beast than just
51:42
connecting 2 GPU's that are right next to each other
51:44
together. That's right.
51:47
Yeah, yeah, I can see that. NVIDIA, NVIDIA itself is, is
51:51
doing a lot of work here, right? So you have to overcome that E
51:53
as well. Like as we know, like when when
51:56
a box ships, it's typically got 4 to 8 GPUs that are, you know,
52:00
interconnected via the single NV link, right?
52:02
So then you're so you're not going to really beat that.
52:05
Then you you have to probably play in this kind of like into
52:08
node space, trying to how do you make communications across these
52:12
nodes for LM workloads a lot more efficient.
52:15
Yeah. And you need to just go after
52:18
folks that have lots of nodes. So you're instantly looking at a
52:24
really long sales cycle. You're looking at how are you
52:28
going to get someone to commit to using you it I can see the go
52:32
to market would just be really difficult.
52:35
So it makes it hard to want to invest in something like that.
52:39
And how much can you saturate a market like that?
52:43
I wouldn't know because I am in no means by no means an expert
52:49
in this. But I would think like after
52:52
you've sold to Open AI and Meta and Anthropic, maybe you're kind
52:58
of good, then all right, you can go after XAI, you can go after
53:02
Mistral, and then who else? Or maybe there is that, well,
53:08
once it becomes a standard, everyone's going to use it
53:11
because it's better. So it doesn't matter if you have
53:16
5 GPUs or if you have 5000. Yeah.
53:19
And look, I think if you can overcome some of these
53:25
challenges that we just talked about, I would say like yes,
53:30
while a lot of investors will say customer concentration is
53:32
bad, like, you know, there are successful companies we can look
53:35
at that that have immense customer concentration, but.
53:39
They're doing great. But but it didn't great.
53:41
You know, NVIDIA is not NVIDIA arguably is a company with a lot
53:44
of customer concentration, right.
53:46
So I think the the question is how do you kind of overcome some
53:51
of those hurdles? And if that results in you
53:54
having, you know, a few handful of customers that are paying you
53:57
an insane amount of money, that's not bad.
54:00
And then by the way, you know, inference ends up, in my
54:03
opinion, being the big opportunity because you know,
54:06
ultimately every business is going to have LM applications,
54:09
as we know. And and you know, inference is
54:11
just going to be such a pervasive workload, you know,
54:14
many, many or as a man to bigger than training.
54:16
So if what you built is gives you the foundation to go after
54:20
inference in some shape or form, then by all means.
54:24
Yeah. So we had a sponsor.
54:28
We have a sponsor of a big company, big cloud company that
54:32
I won't say exactly who it is, just to keep them unnamed.
54:36
And I was talking to them about why they wanted to sponsor us
54:41
and they were saying, yeah, well, we have a lot of people
54:44
doing AI on our cloud and we want to know more about what
54:48
they're doing and we want to get closer to them.
54:51
So I asked, Oh, yeah, really? We have so many people that are
54:57
training models, Is that what's going on?
55:00
And they said, no, no, no. The majority of the money is
55:02
being made on inference here. Don't get it twisted.
55:05
This is where things are happening.
55:08
The training is great. We love it.
55:10
We're not going to turn them away.
55:11
But the inference is where the real cash is being made.
55:14
That's right. And and you know, I think with
55:16
this whole test time compute paradigm at least seeming
55:19
interesting, I would say the jury's out so far in terms of
55:22
how effective it can be. But if that paradigm continues
55:27
to yield exciting results, then but no doubt inference is going
55:30
to be way, way, way bigger than training.
55:33
It's going to Eclipse training as we know it.
55:36
Yeah, yeah, it's fascinating, man.
55:38
Well, before we jump, anything that you are super excited about
55:44
something maybe that you just recently invested in or that
55:48
you're looking to invest in the types of companies?
55:51
We talked about the application layer, we talked about customer
55:55
success or the coding. You kind of got tools that you
55:59
already invested in there. So if somebody's doing stuff in
56:02
that space, I don't recommend that they reach out to you.
56:04
If somebody's doing stuff, in what space should they reach out
56:07
to you? Yeah, I I'm really excited by
56:11
people that are trying to push the frontier of what agents are
56:15
capable of. And again, comes back to what we
56:17
talked about, can you build a application that is able to
56:22
truly automate a useful unit of work, right?
56:26
And so just to give you some examples, right, Like there are
56:28
companies that are trying to, you know, automate the work of
56:31
security operations analyst. There are companies that are
56:34
trying to, you know, automate the work of Bpos.
56:37
There are companies that are trying to automate the work of
56:39
site reliability engineers. These are just like really
56:42
interesting potential applications where the
56:46
underlying technology is, is very hard to build because you
56:50
have to, again, get these things to be able to reason reliably
56:54
over, you know, large swaths of data, multimodal data sets.
56:58
And anyone who can crack that will find themselves in a, in a
57:02
pretty big market. So, you know, people who are
57:05
thinking along these lines, trying to push the frontier of
57:08
reasoning for, for agents where there's a pretty tangible, you
57:13
know, value proposition. At the end of the day, I'm very
57:16
excited to meet those people.