Latest / Agentic Conversations (formally mlops.community) / The Unbearable Lightness of Data // Rohit Krishnan // #295

Transcript

The Unbearable Lightness of Data // Rohit Krishnan // #295 · Agentic Conversations (formally mlops.community)

0:00
Name is Rohit Krishnan, company is Bodo Bodo AI title as chief
0:05
product officer and I usually go for a latte although these days
0:10
I'm having a lot more decaf. All right, a little hat tip to
0:14
my man Rohit, because he was the first person to turn me on to
0:19
the idea of when a model is thinking.
0:22
Why don't we have the ability to click into the specific steps
0:28
that it is going over so that we can help guide it better?
0:33
I'm Dimitrios, your host for the ML OPS Community Podcast today,
0:37
and this conversation was fascinating.
0:46
I got to tell you, I'm a huge fan of Strange Loop Cannon, your
0:50
sub stack where you've been writing.
0:52
I feel like you are a original thinker in this world of
0:59
parrots, and so it's nice to get a splash of differentiation
1:05
whenever you drop a newsletter, I guess we could call it, or
1:10
blog post. Right.
1:12
That, that's very kind. Thank you very much.
1:15
It's a labor of love. Yeah, well, we're going to talk
1:18
about a few of these different blog posts that have really
1:23
captured my imagination. But the first thing that I want
1:27
to dive into was your insight that I saw on Twitter.
1:30
You are like one of the main reasons that I still check my
1:34
Twitter. That's very kind.
1:36
The thing that you put on Twitter that I absolutely loved
1:42
was I think it was talking about DeepSeek, right?
1:45
Or it was a Quinn model and it was the reasoning behind it.
1:49
And you mentioned the UI on this could be better if we could
1:54
click into each one of these steps, these reasoning steps or
1:59
each one of these passes that it's doing and then adjust the
2:05
prompt or directed in a different way.
2:09
And I just felt like that was something so in front of my
2:14
face, you know? But at the same time I had not
2:18
ever heard anyone put it like that.
2:23
That's very kind. Yeah.
2:24
I think it was about deep sea, the R1 if I'm not wrong.
2:28
Before it was cool, it was way before deep sea.
2:32
Fanboy for a little while. The way I think about reasoning
2:36
models is that why is that interesting?
2:41
The interesting part is like when we started out in the pure
2:45
Chadbot world, right? You ask a question, it's an auto
2:48
regressive model. It gives you back an answer in
2:51
some sense, that's what you do, right?
2:53
It gives you an answer, you don't like it, you kind of poke
2:56
it product, push it somewhere else and it kind of goes and
2:59
does that thing for you. Then we started introducing
3:03
tools that they could use and then you could even say like,
3:07
remember there was a little while, but like ChatGPT when it
3:09
started introducing the tools, but like it just wouldn't do the
3:13
right tool. So you had to kind of stop it
3:15
from searching the web or you had to say like, no, no, no, go
3:19
use Python. Like write a script.
3:21
Don't just do the math in your head.
3:25
I feel like with reasoning models, we are at the first part
3:27
of that curve a little bit, right?
3:29
Because clearly if you're reasoning in the sense that you
3:31
can see the thought process flow through, you know, where it's
3:35
getting closer to sort of something interesting now
3:38
because ultimately it is a model still.
3:40
And it's not like, you know, able to understand the entire
3:45
context that you have in your mind.
3:46
Sometimes you do want to just stop it and be like, no, no,
3:49
just wait right there. I want to pick it up from there
3:52
because that is the truly interesting part that I want to
3:55
go off of now in like mechanical, not mechanical in
4:00
like scientific queries that might be prevalent to finding
4:03
that and saying like, I want to take that and research it.
4:06
I want to take that and go search for it.
4:08
I want to take that and write a piece of code to test that
4:11
assumption out, what have you, but it the same if you want to
4:14
do stuff for literature, right like it found whatever an
4:17
interesting illusion about something something Shakespeare
4:21
and you're like, that is an interesting.
4:23
That is a cool thought. I want to pick up from that
4:26
point and extrapolate because quite often otherwise it might
4:30
just go back or go sideways or go somewhere else.
4:33
And ultimately, we are consumers of these models.
4:35
So I think the more control we have of figuring out how we can
4:40
take it for a spin, the better it is for us.
4:44
The flip side is all of these models get far better at
4:47
figuring out what we want as users because that's what
4:50
they're being trained for. And as they're getting better at
4:54
figuring out what we want as users, the need to do this
4:57
perhaps reduces a little bit for the normal models, But for
5:01
reasoning models where fundamentally we ask really
5:03
detailed questions and the answer back is not like a
5:08
message, but like a report, then the need to do that increases
5:12
dramatically. Because I don't know about you,
5:14
but every single time, whatever at work, whatever, you get a
5:19
report back from somebody, even if it's extraordinarily well
5:23
done, you have questions. You're like, I want to
5:26
understand that a little bit more.
5:27
I want to poke more here. I want to increase the sort of
5:31
magnification that you put on this.
5:32
This part is not that interesting.
5:33
So let's whatever. I think that dialogue is really
5:36
important and that dialogue is really hard to do with like a
5:39
10,000 word, you know, output that deep research throws at
5:43
you. It is much easier for you to say
5:45
like, that's an interesting reasoning point.
5:47
I want to give you insight as to what is useful here while you're
5:51
thinking through. We'll get there.
5:53
I'm 100% confident we'll get there.
5:56
Really like making it a collaborator is something that
6:00
is cool to fantasize about because when you're doing it as
6:08
it is, spitting out these reasoning points or the steps
6:13
that it's going through and you're able to, I think the way
6:16
that I visualized it was you're able to click on one of those
6:20
and then add more context or as you said, spur multiple
6:25
different actions from that one piece of reasoning that.
6:31
Ultimately these are chains of thought, right?
6:34
It is, it's, it's rolled out chains of thought.
6:37
And if it is having a chain of thought, then we should be able
6:40
to pause it or poke it and kind of take things from there.
6:43
I think that just that is the correct mode of interaction with
6:48
that particular thought process. We can ask it to think more from
6:53
that point, you can add more context.
6:56
You can just say like, that is really interesting.
6:58
Why don't you go search against that and come back with some
7:02
sort of insights like Because not everything needs to be equal
7:06
valence. Yeah, it's almost like
7:09
checkpointing in a way, or you're giving it more feedback
7:14
as to what is important or where the signal is versus where the
7:18
noise is. Correct, You do want the model
7:23
to do the best job, but also like fundamentally all of these
7:27
models to a large extent, they are blind as to their context.
7:32
I mean, you have to provide. That's why like models like O
7:36
one or whatever do so much better if you give it more
7:38
information because like you just how would it know?
7:41
There is no way for it to know. So like you just have to
7:45
continually provide a huge amount of background before it
7:48
gets really good at providing those answers.
7:50
And the way to provide a large amount of background might be
7:52
deposit in the middle and kind of add more information.
7:55
I think that would be a phenomenal way for us to figure
7:57
out how to steer them in the direction that makes more sense
8:00
for us given a question, given an answer, right?
8:03
It also means like you might be able to side label them together
8:06
saying you have a report generation.
8:08
That is a fundamental Co creation aspect that is being
8:12
built on, but there's also the part where it's not just a
8:17
report creation. You actually have a
8:19
conversational agent where you have a conversation
8:21
simultaneously so that you know what things to put in the
8:24
report. I don't know whether you tried
8:26
out the new deep research feature from open AI, but like
8:31
it provides you a it has a couple of questions it asks in
8:34
the beginning and then refines the plan and then it goes up and
8:38
just does it work for like whatever 1015 thirty minutes.
8:42
Now the output is amazing. However, most of the time the
8:47
interesting stuff that it surfaces is somewhere deep
8:49
inside that process off the 30 minutes.
8:52
And if we had better steerability, would that be
8:54
better or worse? I think it'd be better because
8:56
you can actually get a lot more, like you can ask it to go deeper
9:01
into spaces that are most interesting to you within a
9:03
report as opposed to sort of asking you to sort of generally
9:07
create a report. And then having this like, you
9:10
know, 5008 thousand page output, the 8000 word output that you
9:14
then need to refine by saying, oh, guess what?
9:16
In section 4.5, why don't you? That's a very bad way of
9:20
analyzing it. At least that's what my
9:21
hypothesis is here. It feels like there is that
9:26
potential product that you have the report and you can just go
9:32
and drop comments in and then it will spur more progress or more
9:39
action from the model and try and refine these different
9:43
parts. Or there's this idea where you
9:47
have the transparency as things are happening under the hood and
9:53
you get to see it as it's happening and then like you
9:56
said, steer it in the way or or try to figure out if this is
10:03
valuable or if this is not. I guess the hard thing in my
10:06
mind would be it will probably take you a while to recognize
10:10
what's important. And so as it's creating this
10:14
report, it will probably spit out the report.
10:17
And then you'll want to retroactively go under the hood
10:20
and say, like, it's really right here on step seven that we
10:24
should take a left turn instead of a right.
10:29
Precisely because ultimately, like you do want to have that
10:32
input into things that you think are important because you're the
10:34
one asking the question, they're usually to take an action for,
10:38
to learn something, to use it somewhere else, whatever it
10:40
might be. The the goal is for you to
10:45
utilize that information for something, right?
10:47
Even if it's purely for edification, it was so, you
10:50
know, just because you like it. And that requires more
10:54
interactivity with the output that is coming your way.
10:58
And the cool thing about R1 is that it's the first reasoning
11:02
model where we can literally see it's entire chain of thought.
11:07
And it is just like a fascinating thing to read
11:10
through on its own, right? Because the number of times that
11:12
it kind of goes like, oh, but wait, or like, let me backtrack.
11:16
It's, it's a partially because of how it is trained.
11:19
It's a very human friendly way of reading and understanding
11:23
what it's thinking. So it's almost like you're
11:25
picking into the mind of somebody who's doing the work
11:28
for you. And if that's the case, then you
11:32
want to kind of steer it a bit better to kind of guide it in
11:34
the right direction. You don't want it to fall into a
11:36
cul-de-sac or like something that you're not excited about or
11:39
interested in, right? So like the more input you can
11:42
give it, the better. And why should all the input be
11:44
front loaded as opposed to sort of stuff that you can give it
11:47
along the way, so to speak? Especially if it's acting as an
11:51
agent, it should be possible. I think a lot about the idea of
11:57
cognitive load and how if this is going out to users and you as
12:03
a user have to front load all of this information and get it
12:07
right and then wait 15 minutes and see if the output is correct
12:11
and then refine it and wait another 15 minutes etcetera,
12:14
etcetera. The way that I see this
12:19
manifesting that is one potential future, is that just
12:23
like when you use Notion AII don't know if you use that much,
12:27
but I've used it very much like clicking.
12:29
Yeah. And you don't have to prompt
12:31
much unless you want to. And so there's probably you
12:36
click into one of these chain of thoughts and then you have some
12:40
basic things that can be done, the most common things where
12:45
it's like go deeper here or search the web for and then you
12:49
can add whatever you want. So there's a lot of cool product
12:54
manifestations of this. I do wonder about the technical
12:59
feasibility on the model side. Like do you feel like it would
13:03
be that hard for a model to pause and just cut that context?
13:09
Because I know that with something like the streaming API
13:13
for voice, you do have that capability built into the model
13:18
and you can have that cut off and then the model understands,
13:22
OK, I cut off here. I didn't say everything that I
13:25
wanted to say. So I'm not going to assume that
13:28
the person understands all of that stuff that I was about to
13:31
say. Well, I think it's feasible and
13:33
the reason is primarily that like ultimately the multiple
13:37
roll outs of the chains of thought in order to select the
13:40
right path that it should go down is relatively
13:43
interruptible, so to speak. What has happened before or the
13:47
things that it has rolled out doesn't necessarily just vanish,
13:49
right? As long as we can sort of
13:50
capture it somewhere. If you want to kind of reuse it
13:53
or start from that, that's possible.
13:55
And if you think about the smartest models that exist at
13:57
the moment, they you can, you can choose your reasoning effort
14:02
to be high, low, medium, whatever, which to me indicates
14:05
what that like, not just to me, which shows that you can just
14:10
decide to not push it to the edge if you wanted to.
14:13
Like once you thought it through, at some point, it kind
14:16
of gives you an answer. It's just not the best answer
14:18
considering the question that you asked, which effectively is
14:20
a way of saying like at that point, at every point you have
14:23
the choice of like searching more, which means rolling out
14:26
more and more or taking in more input.
14:29
And there's no reason taking in more input should fundamentally
14:32
be, you know, architecturally difficult or super difficult,
14:36
right? I would be extraordinarily
14:39
surprised if that's not being worked on, especially because it
14:43
is be like people are trying to introduce tool use into these
14:48
rollouts anyway. And if you introduce tool use
14:51
into these rollouts, then almost by definition, you're
14:54
introducing a way for the model to do something.
14:57
Now, tool use can be internal, like writing a piece of code and
15:00
verifying it or whatever. But it can also be external,
15:03
right? It can be doing rag over a
15:05
corpus of documents, or it can be going and searching or going
15:08
and finding or like, you know, whatever those are bringing
15:11
extra and external pieces of information back in.
15:14
So, you know, you could even do a dirty version of at this point
15:18
if you're unsure because of whatever reason or like if a
15:21
user interrupts you, take that input in and then roll it out
15:24
from there. Totally feasible.
15:27
I love this topic. I'm very excited for the future
15:31
of how we interact with LMS for this very reason because like
15:36
you said, I would be very surprised if there aren't folks
15:39
already working on this and if they aren't like get on it
15:42
because it's we're asking for it and it's not.
15:47
If we could see it then I'm sure the top research labs can also
15:52
see it, correct? What other wild stuff do you
15:57
have that you ponder about daily or weekly?
16:04
I mean, one thing that I keep thinking about is especially
16:08
since O1 Pro, I would say that like we've hit the point where
16:11
the models actually are ridiculously good.
16:16
I think we've, we've hit that point now something I think it's
16:22
like the models will keep getting sort of better and keep
16:25
getting faster, smarter tool use, etcetera.
16:27
Like all of that stuff will happen, which effectively means
16:31
that at some point in the not too distant future, we will
16:36
start being able to actually use these things for real life work
16:42
more and more because we can already use them.
16:45
But they're kind of like right now it's more jobs in bicycles
16:49
for the mind usage as opposed to, you know, autonomous usage.
16:53
But I feel like we're getting to a point where we can kind of see
16:58
a path towards doing that, right?
17:01
People would say they've already always seen it, which might also
17:03
be true. But to me, it's like, you know,
17:06
understanding what's going on inside a code base and
17:09
contributing APR is the kind of thing that like you can
17:12
visualize a model being able to do given the right kind of tool
17:16
kit for it to get the information from that and then
17:18
be able to kind of go into that. In such a scenario, I kind of
17:23
keep thinking like, you know, I have a almost a socio economic
17:27
question of what does that world actually look like?
17:31
Because a large number of existing normal white collar
17:34
jobs will transform quite dramatically.
17:38
And if it transforms quite dramatically, I don't know what
17:41
the 2nd order implications of that are likely to be for the
17:44
economy. Like, you know, yes, sure, I'm
17:47
not a doomer. So like I feel like we'll come
17:49
up with new ways to actually utilize the extra manpower and
17:52
inside that we have lying around and that's great.
17:56
But is that it? Or like, you know, do we have a
18:00
fundamental realignment in terms of what types of jobs basically
18:03
just go away? Like the, you know, I had this
18:06
post at some point about, you know, computers used to be a
18:10
job, right? It used to be a job description
18:12
for people, but it's no longer. Now it's a machine.
18:14
And similarly, you can kind of envision maybe analysts or
18:18
researchers being a job description for people today
18:21
that kind of goes away entirely. In which case you have to have a
18:25
very different format of interaction with them, with the
18:28
AI such that you're able to capture the information or
18:31
utilize it in some fashion. I don't know what that looks
18:34
like, right? I mean, would you end up having
18:36
a lot more orchestration necessities as the primary kind
18:39
of way of interacting with the models are unclear, but I think
18:43
that is an interesting thing that I spend a bit of time
18:45
taking about. So I do love in one of your
18:50
blogs how you talked about some jobs that could spur off of this
18:54
are high quality data labelers that come through and we know
18:59
that. All right, we're going to, I, I
19:01
forget how you put it, but we know that this is eventually
19:05
going to be a job that will be done by a machine, but we have
19:10
to get a large amount of data before that can happen.
19:14
So we're going to recruit labelers in this specific field
19:20
and you're going to be almost like temp workers for a bit.
19:24
But one of the things that I think of is that it's not even
19:27
data labelers per SE, right? I think it's almost like if
19:32
these transformer based models are ridiculously good at
19:35
identifying patterns and utilizing them in order to
19:38
create sort of the output. If that is true, which it seems
19:42
true, then it stands to reason that any ongoing process or any
19:48
ongoing piece of work given sufficient data should be made
19:52
more automated, correct? Because like you have that data
19:56
and if that data actually gets captured, especially the process
19:59
data gets captured, you should be able to automate it.
20:01
And if that's the case, then effectively no matter what you
20:06
do for work, part of your job ends up automatically entailing
20:12
the requirement for you to be the training wheel for the AI
20:17
that will come in the future. Like I, I don't see how we get
20:21
away from that First, like extremely simple tasks, maybe
20:25
you can just get data labelers, that's fine.
20:28
But you know, if you're doing sort of high value task that
20:32
can't realistically be done by an AI, then what does that
20:35
actually mean? I mean, does that mean that you
20:37
kind of just do your job and there's data capture happening
20:40
simultaneously and at some .2 years, three years down the line
20:44
or whenever they get actual amounts of data, then you
20:47
actually get converted into the AI world.
20:49
It's feasible, right? I mean, this kind of what I mean
20:51
by what is the economic reality of 2040 kind of look like
20:57
assuming we don't do recursive self improvement and everything,
21:00
you know, gets crazy, which is possible, but then that's very
21:03
hard to kind of predict or understand.
21:05
But to me like this seems, this seems possible, but like, I
21:09
don't know, name a job, right? I mean, they all have to go
21:12
through a certain set of pieces of work and every single person
21:17
doing that job almost inherently ends up being a data labeler for
21:21
some models that'll get created at some point in the future.
21:24
I, I, I don't see what I mean. I think that's feasible.
21:27
One thing that I think about is how much noise the models create
21:33
right now and how much the output makes information cheap
21:39
in a way. And because of that, going back
21:44
to the like, no cognitive load or a lot of cognitive load, if
21:49
you're expected to read a 8000 word report for everything and
21:57
then have to tweak it and have to go back and forth with it,
22:01
that type of stuff can get very noisy.
22:06
AI is trusted to do a lot of these things that we're talking
22:12
about like in in any job or field.
22:14
But let's let's take this example.
22:16
I really like the one of understanding the code base and
22:18
then submitting APR. Sure.
22:21
The person who has to review all these PRS is going to go fucking
22:25
nuts because there's going to be a lot of shitty PRS, correct?
22:31
And so maybe you say, well, we'll get another AI to review
22:34
the shitty PRS and then only the good ones will come up.
22:39
And then you think, what is that going to get rid of all of the
22:45
noise? You know what I mean?
22:47
Like. I, I see what you're trying to
22:48
say. I mean, I don't think of, I
22:50
don't know if noise is the right framing.
22:53
I, I like the framing, but I don't know, I think the, well,
22:57
first of all, of course you will get it to review the PR as well
23:01
as something like, it just makes sense because like in some way,
23:06
reviewing the PR is probably easier than writing the PR in
23:10
the 1st place. And it's also true that like
23:14
many of them might not, especially as we are on the
23:17
curve, many of them just might not be great.
23:20
It's a fact. However, what we have done in
23:24
that world is that we have effectively commoditized PRS,
23:29
right? Like PRS are no longer a big
23:32
thing. It's just like whatever, it's
23:33
like a tweeting, you know, anybody can kind of do it.
23:36
It's like it's a, it's, it's not a high effort, intensive,
23:40
complicated research activity. It's just something that anybody
23:42
can actually contribute to. And if that is the world, then
23:47
noise is fine, right? That's like a lot of noise on
23:51
Twitter. It's still fine because like we
23:52
use it for some reason effectively because we are able
23:56
to gather the signal from that somehow.
23:57
And the work that goes in just because it has higher volume and
24:01
higher variance doesn't really matter because net net stuff
24:04
just works out. And BRS are become like, in some
24:08
ways, like you break it, you just do another one and you just
24:11
break it. You do like you're not, you're
24:12
not scared anymore that like you're just going to like break
24:15
stuff because suddenly the velocity of you being able to do
24:18
that just goes up. And I think that's true of a lot
24:21
of different types of pieces of work.
24:23
I think the place where it becomes hardest is things like
24:27
academia or something where there is inherent roadblocks in
24:30
terms of how much you can actually create.
24:33
So you know, highly productive academics like whatever Asimo
24:38
glue in economics, he writes like an extraordinarily large
24:42
number of very highly rated papers in a very short period of
24:46
time. I think that was a while that he
24:47
was writing, I think like one a month or something that kind of
24:50
goes into one of the top journals, you know, co-author,
24:52
etcetera. And I'm like, OK, but today if
24:56
you're able to speed that up and say, like, you can do a top
24:59
journal entry once a week, what does that do to the entire
25:04
system? It just kind of breaks it down,
25:06
right? Because suddenly one a week
25:09
means like, no, no other part of the system can capture you.
25:12
Like, nobody can review it fast enough.
25:15
Journals don't publish that fast.
25:17
Like all you got left is like, throw it up on archive, right?
25:20
And at which point you have effectively, you know, broken
25:24
through the existing hierarchies in terms of sort of how this can
25:28
work in a very similar way to like whatever agile, etcetera,
25:32
kind of broke through the six month, one year release cycle
25:35
hierarchy that used to exist. Because if it was 1-6 months,
25:38
one year, then like there's only so much you could do.
25:39
But now if you can push something every day, every week,
25:43
every two weeks, suddenly like, you know, the way you do the
25:47
work itself changes and some types of work just goes away
25:50
completely. I, I think that is going to hit
25:53
multiple different other parts of the economy.
25:55
So I think noise is A, to me, seen in this fashion, a certain
26:00
level of noise acts as a feature, not a bug, because it
26:04
demonstrates that you have a sufficient velocity that we are
26:07
willing to kind of take the hit of that noise, if that makes
26:10
sense. Yeah, it's really makes me
26:14
ponder the idea of the human on the other side that is meant to
26:20
be ingesting all of this information.
26:24
Like if we're publishing these very in depth research reports
26:29
once a week now as opposed to once a month, the attention of
26:35
the folks who are reading that is so valuable because there's
26:39
going to be, we think there's a lot of information right now and
26:43
there's a lot of shit that we can spend our time on and our
26:45
attention on, but it's going to get exponentially worse or
26:53
better and depending on how you're looking at it.
26:56
I think one of the funny things is like I am a fairly firm
27:00
believer in in the fact that like market mechanisms work
27:06
remarkably well and market mechanisms doesn't need an
27:09
actual financial whatever dollar market.
27:12
It just means like checks and balances are phenomenal.
27:15
It just works right. Like, you know, I remember sort
27:20
of for the last like decade, I've been hearing about the rise
27:23
of deepfakes in some sense. And it used to be, you know,
27:28
it's a hi now, it used to be Photoshop, whatever.
27:30
Like there's all of it. There's always a new technology
27:33
that that makes it easier and the same for like cyber or
27:35
whatever, right? And, and you know what, they're
27:38
all right. Like deepfakes are insanely easy
27:41
to do today. It's just a fact.
27:43
However, it's also true. Like we are all, we are not all
27:48
like kind of walking on egg shells trying to figure out, we
27:52
kind of figured out a way around it, right?
27:54
There's enough signal in the noise that if I see something
27:57
weird on Twitter or whatever, I have this at least this thing
28:01
saying like, is that real? Like, and that seems to kind of
28:05
help because then you can you have a bit more logical or like
28:08
common sense to kind of capture those, capture those things.
28:11
And to me that's like a kind of checks and check and balance.
28:14
I think the same thing applies for like most of these large
28:19
scale interventions that might come from AI changing the way
28:23
that the world works for for for people.
28:25
Like AI makes writing research report super easy.
28:30
OK, that means the value you place on a research report might
28:34
go up or down. Great.
28:35
But it also means like you suddenly have very different
28:38
yardsticks and benchmarks and heuristics for figuring out if
28:42
one of these things make sense for you to do or not, right?
28:46
Like, I mean, you kind of choose your battle, so to speak a
28:50
little bit. And as a, you know, in a weird
28:54
way, as a human, what information should you actually
28:57
look to ingest is a non trivial question, right?
29:01
Like what is your job? Is your job to read all the
29:03
reports that AI is writing? That seems pointless.
29:05
Like why would you do that today?
29:07
I read sort of stuff that people write because I read the things
29:12
that like my team writes because reading that is how I get the
29:18
information about what they want.
29:21
But in the future, if like their AI write something and my AI
29:27
read something and summarizes the things I need to know, like
29:29
in a bullet point list, that's fine as well, right?
29:32
Like, I mean, it changes the velocity, but the way I consume
29:35
it will fundamentally differ in terms of sort of from from the
29:39
way that it works today. I don't.
29:41
I'm not saying that's the way it should be or will be, but I can
29:44
easily see it moving up a layer of aggregation.
29:48
Sort of. You read the information coming
29:51
towards you. There is another thread that I
29:54
wanted to pull on, which is in the blog post that you wrote
29:59
about AGI and like what that actually would look like because
30:03
I feel like it is kind of playing on what we're talking
30:06
about right now, right? And it's, it's thinking about
30:10
what, what does it look like assuming things continue to get
30:16
better and get better and we are able to leverage the benefits
30:22
that we already see. And one of these you kind of
30:28
breakdown different vectors on how it could get better or how
30:32
it could maybe not get better. And I like at the end of the
30:38
article you mentioned it's probably going to be a a Little
30:41
Mix of all of these. Can you explain what these
30:45
vectors are in your head and then how they can get better or
30:50
worse? Yeah.
30:52
I mean, I think the, I should say the primary reason I ended
30:56
up writing that essay was I think it's the one that said,
30:59
like, what would the world with AGI look like?
31:01
I think that's the one. Yeah, yeah, yeah.
31:03
Is that a large amount of the conversations around this future
31:08
world that we have revolve around extreme assumptions and
31:13
extreme assumptions means that almost every extrapolation that
31:17
you can make and it just goes up in smoke.
31:19
Like once you assume that, you know, AI will make creating
31:25
better AI super easy to the point where we are not
31:29
realistically going to be bound by energy or resource
31:32
constraints, then kind of we don't need to like, what's the
31:36
point, right? I mean, you can, you can say
31:38
anything at that point and there's like no way to kind of
31:40
prove or justify or back solve it at all.
31:44
So my question was like, there are some fundamental
31:46
constraints. We kind of know barring crazy
31:49
breakthroughs, which will happen.
31:51
So, but like, at least for a base case, we should be able to
31:54
look at this and go like how much power is going to be
31:56
required? How much, you know, how many
31:59
chips are going to be built and using those things?
32:02
Can we at least back solve to say like, even if you know one H
32:07
100 was all that's needed to run like a human level agent
32:12
continuously, what does that mean?
32:14
Is that good? Is that like what is the change
32:17
in the labor market or what's the change in the world?
32:19
That was my like setup for the question just to like do a base
32:22
case scenario. And The thing is like, you know,
32:28
people got to talk about trillions of AI agents floating
32:30
around, and that has to be like, there's no AI agent in the
32:33
ether, right? They have to run on some
32:35
substrate. So like if they're running on H1
32:38
hundreds, then like H1 hundreds need to be created.
32:41
And we have better chips now. Fine, H100 has some average
32:45
power consumption that happens today.
32:47
It has a service life which is actually not that large.
32:51
So I think it's like three years or something running at like
32:54
reasonable utilization. If you're running it at 9000%
32:57
utilization, the life falls even further because there's failure
33:00
rates, right? I mean, Lama paper talks about a
33:02
very high failure rate for these GP us when you connect them
33:05
together, which is fair enough because like everything in the
33:08
world has depreciation. Why should this be one of the
33:10
few things that doesn't have? So now you have the number of GP
33:15
US, you have the amount of power that is created, you have some
33:17
failure rate, which means you can have a rough end of like
33:21
either from a CapEx or an OpEx point of view.
33:22
How much does this cost for you to kind of run it?
33:25
And if you run it 24/7, how many agents are you kind of talking
33:30
about roughly? That's the kind of math that I'm
33:32
trying to get that I was trying to do.
33:33
And I think the number that I got to was something like, I
33:35
don't know, 4045 million full time agents given all of these
33:39
assumptions. And the point of that is not to
33:42
kind of say that's too high or too low or even too accurate,
33:46
But the point was to say that like, look, these are the
33:48
assumptions that need to be broken.
33:50
If that 45,000,000 needs to become 45 billion, right?
33:53
Like if an AI agent suddenly becomes as efficient to run as
33:56
humans are like a few watts, whatever, then then it'll break.
34:01
But then guess what? We need to, we need to get
34:03
there. That is a fundamental constraint
34:05
that needs to be broken or we need to say that like a
34:08
substantial proportion of the world economy basically becomes
34:12
chips and energy. It just has to, there's no way
34:15
out, right? Like, I mean, think about it, if
34:17
you added an X, if you double the labour workforce across the
34:22
entire world, that's a major shift.
34:25
But they have doubled the entire workforce of the whole world
34:28
before. And that's been fine.
34:30
It doesn't mean all of us are kind of jobless, right?
34:33
It doesn't mean that at all. It means we have found new jobs.
34:37
If you triple it, still we'll find new jobs because like we
34:40
there's enough stuff for everybody to do that tripling
34:44
the workforce doesn't necessarily mean that we end up
34:47
in a dystopia. So I think all of the the the
34:50
conclusion therefore is some version of the fact that we are
34:53
bottlenecked by certain very clear, very strong physical cap
34:59
X investments, for lack of a better word.
35:02
Some of these things are getting better.
35:03
I had some numbers about like I think I assume thousand X
35:06
improvement in some of these numbers from from like O3 to
35:10
like running an ATI on H100 or whatever.
35:14
But these are the kinds of extrapolations you will need to
35:16
make if you want to figure out what the world actually actually
35:19
will look like as opposed to sort of assuming the conclusion,
35:23
which is to say, what if chips get, you know, a million times
35:27
more powerful and a million times cheaper?
35:29
Like, OK, but like you got to push that through, right?
35:33
Like in the sense of making a chip today requires ASML to
35:38
create certain lithography machines.
35:40
And there's only so many they can make.
35:42
And if they can make it and DSMC has a certain number that they
35:44
can manufacture, like the the world, the world of atoms is not
35:49
infinitely elastic. You kind of have to pick and
35:53
choose where it comes from and figure out where it leads.
35:55
And also answer the question of like, hey, is, you know, a
36:00
substantial proportion of the world GDP going towards just
36:04
creating semiconductors and energy is sustainable
36:07
equilibrium. And I'm not sure it is like what
36:12
Like we're all happy with NVIDIA stock prices going up, but you
36:16
know, it's at 3 trillion. Are we happy when it's 10?
36:19
I don't know. Are we happy when it's 20?
36:21
There is some breaking point where we kind of go like that is
36:24
too large a percentage of the economy that is going into one
36:27
company or 1 industry, right? I mean like that's just the way
36:30
that all S curves eventually bend.
36:33
And I just want to kind of draw a line a little bit around what
36:35
that's what bending cycle is. Yeah.
36:39
And and then you mentioned how there's there's these different
36:42
pieces. Yeah, it can get better here or
36:45
it the assumption that it's going to run on H1 hundreds can
36:49
be one assumption. Maybe we drop that down and it
36:52
all just runs on our CPU's, but we still need these other.
36:57
We still need to think through the bottlenecks and where we're
37:01
going to have different bottlenecks if we're trying to
37:03
game out these scenarios. Correct.
37:05
Bottlenecks don't disappear just because we want them to.
37:09
I think they exist, they move, but like they 100% exist and the
37:13
story of the world economy has been one where we keep
37:16
discovering new bottlenecks and breaking through those and going
37:18
to the next one. That which is why we effectively
37:21
have somewhat of a linear growth curve despite exponential impact
37:25
effort going into each individual part of it.
37:28
Yes. So right now you hear a lot of
37:31
talk about the bottlenecks being the energy and chips and just
37:36
GPU's. I've also heard some people come
37:39
on here and they say it's not necessarily GPU's that are the
37:42
bottleneck, it's more the memory on the GPU's that are the
37:46
bottlenecks. And so folks are trying to
37:49
figure out clever ways to disrupt that, assuming that we
37:57
get, as you said, this expansion in GPU's, we have to look at
38:03
like what steps would we need to take for that to happen?
38:07
I know there's a ton of investment that is going into
38:10
this right now, but these kind of things are going to play out
38:14
over decades. Yeah.
38:17
I think the investment kind of tells you a little bit about
38:20
what is required for us to kind of push that boundary forward a
38:24
little bit, like the amount of effort that's required for one
38:27
extra data center, 1 new nuclear power plant, 100,000 new chips,
38:32
which just shows you that when you're playing at the margins,
38:35
the numbers get very large very fast.
38:38
Yeah. And then how do you, I, I really
38:40
like the idea of how do you justify when enough is too much
38:47
or too much is is too much? What does that look like?
38:51
Where you say, all right, do we really want to invest half of
38:56
the country's GDP into this one big bet?
39:01
Yeah. I mean, I don't think we kind of
39:03
this is one of those things where we don't, we won't know
39:05
until we hit it. But I'll ultimately like, you
39:08
know, right now everything is kind of going up and to the
39:11
right, like the investments are bearing fruit and we are clearly
39:14
being able to utilize these tools to get amazing results.
39:18
Even if we extrapolate that thing out, there is this
39:21
limitation on the amount of investment that we can
39:23
sustainably take on board versus the amount of output we can
39:26
sustainably utilize. And like, it's a constant tussle
39:30
between those things, right? In order for us to get to the
39:33
future, which I completely believe we will.
39:35
But it's just, you know, we, if you believe something else, that
39:40
is totally fine. But like we would need to
39:43
justify it with some similar assumptions saying like, this is
39:46
what I expected, it's likely to happen.
39:48
And then you got to justify that.
39:50
And the justification can't just be, or maybe it is that like,
39:55
hey, no AI will make everything, you know, a million times
39:58
cheaper, which is like, if that's a justification, then
40:01
let's talk about it. But like, to me, that's kind of
40:04
hand waving the issue away, right?
40:06
You're assuming the conclusion. And it's so vague and it is so
40:11
ungrounded because I don't know about you, but it's not like AI
40:17
has made much in my life cheaper, right?
40:22
I'm actually spending more money on subscriptions now than I've
40:25
ever spent before. There's a lot of AI
40:27
subscriptions to spend money on. So yeah, it's, it's funny to
40:33
just like you said, be a bit hand WAVY and think that just
40:38
because we have AI, we are going to see a ton more productivity
40:43
and B, it's going to make our professional lives cheaper or
40:51
it's going to make things that we need to get done in our
40:54
professional life cheaper. Correct.
40:56
I think like, you know, we have to have, we have to work towards
40:59
doing that. I mean, one way to think about
41:02
the current kind of deep sea bruh haha, is that it's the
41:04
first time, at least that I've seen that.
41:07
Maybe not the first time, but it's like the first major time
41:09
that like efficiency was a even came up as a KPI as opposed to
41:16
performance. Yeah, yeah.
41:19
Because it's, it's like we've been throwing everything we need
41:23
at it correct to get the performance.
41:26
But now we're looking at like, oh, can we get that performance
41:29
without a blank check? Correct, correct.
41:33
And that has implications, right, Whether if you have the
41:36
ability to do that, that has certain implications on what we
41:41
might expect to get in the future.
41:44
And how we can again tune those different vectors that you talk
41:47
about and or at least just be aware of them.
41:50
Precisely. And look, things getting more
41:53
efficient is part of this. Is part of this anyway, but like
42:00
GPT 4 level models when GPT 4 came out was much more expensive
42:03
than it is today, right? However, that there is like
42:08
there is the natural reduction in cost due to further and
42:12
further advances when we learn how to do these things better.
42:15
And that you have to kind of account for that versus like if
42:18
you think that trend is going to get substantially broken and
42:21
there will be a discontinuity, then that needs to be assessed
42:24
or analyzed. Now, fact of the matter is even
42:27
the current trend of decreasing cost that happened over the last
42:31
few years, if you extrapolate that out to like the next 10
42:34
years, you could start seeing some pretty crazy numbers.
42:37
I don't know if that's even feasible, but like that is the
42:42
that is the crux of the question.
42:43
Effectively, when is the breaking point for some of these
42:46
things like Moore's Law held over, I don't even know how many
42:50
orders of magnitude, so it kind of held for a very long time,
42:53
right, and fall in solar cell prices kind of held over a very
42:58
large order of multiple orders of magnitude.
43:02
Is this one of those things that I can hold there as well?
43:04
Maybe. But then we can go back to the
43:05
question of like, can we run an O3 level model on ACPU and kind
43:10
of work backwards in order to get some of those answers?
43:12
Well, tell me about what you're doing for your day-to-day
43:18
besides being a complete philosopher.
43:20
I know you've got a company. I would love to hear a bit more
43:24
about it. Happy to so the the day job like
43:28
I want to spend my time on. I'm the CPO, the chief product
43:31
officer of company called Bodo Bodo AI, which won't surprise
43:35
anyone. Bodo is basically the best way
43:39
to do data engineering and data analytics.
43:42
If you want to work in Python, that's the easiest way to say
43:45
it. So you have large scale data
43:48
sets that you want to do any kind of analytics or engineering
43:51
on transformations, processing, pre processing, what have you.
43:55
Then rather than needing to use anything external like, you
43:59
know, Spark or buy Spark or whatever, or Dask, you can just
44:02
do it in Python. And we make that happen.
44:05
So we're much, much faster, much more efficient, much easier to
44:08
use than pretty much anything else on the market, which we
44:12
recently went open source, which is one of the like big, big
44:17
hurdles that was in front of us. So pretty excited about that.
44:20
Nice. Congrats.
44:21
And thank you. Yeah, that was, that's been a
44:24
little while coming. So like we're finally glad that
44:26
we managed to push it out there. And the goal is we want to get
44:29
as many people as possible to try and play with it, use it,
44:33
you know, get an understanding from the examples that we have
44:37
already put out there. See the benchmarks that we put
44:39
out there, knock on it, figure out where it breaks.
44:42
Basically make make the world of Python great again.
44:46
You know, because current like Python's came up as a lingua
44:49
franca in the last several years for pretty much everything,
44:52
especially AI. But if you want to do kind of
44:55
any hardcore data stuff with Python, you kind of need to end
44:58
up stepping out of it and going and using something else.
45:02
We just wanted to kind of break that cycle and keep it inside
45:04
Python because that also means that a much larger group of
45:06
people can contribute and work on it.
45:09
And so you're talking basically like data frames.
45:13
Correct. So we if you work in, I don't
45:17
know, pandas, numpy, whatever, so I could learn it gets
45:22
dramatically faster then you can see some of the benchmarks in
45:26
the in the in our in our repo, but the numbers are pretty
45:30
pretty upset. It's like 20 to 250 X faster,
45:34
depending on sort of what you're comparing it against.
45:36
Wow. And that's important because
45:38
like, ultimately, if you want to do, I mean, if you want to train
45:40
an AI model, right, you start with a bunch of usually raw data
45:44
lying around somewhere. And then that needs to be
45:49
processed, cleaned, trained, tokenized, whatever.
45:52
Like there's a set of, you know, deduped, there's a set of things
45:56
that you need to do to that data before you are getting ready to
45:59
the point where you shove it into Pytorch and click drain dot
46:01
pie. That is the part that is the
46:05
most cumbersome, takes the most effort, takes the most sort of,
46:09
it's the most painful. And if you can just make that
46:11
part faster, then I think it's going to be pretty dramatically
46:15
useful, not just for us or for AI, but like for data
46:20
engineering across the world. I think that's the goal.
46:23
Agreed. And it's funny that you mention
46:26
that because I've heard actually in one of our most recent
46:31
podcasts with a friend of mine, Flores, he was talking about how
46:35
he identifies as an AI engineer these days, but he comes from
46:39
the data science, ML engineering world.
46:42
And I was like, what's the difference?
46:43
What do you have in your mind that makes an AI engineer
46:47
different than an ML engineer? And he said, well, now I do a
46:50
lot of software engineering and I have prompting sprinkled on it
46:57
and I have a little bit of a eval sprinkled on it, but it's
47:01
mainly software engineering. Back when I would consider
47:05
myself an ML engineer, it was a lot of the data stuff.
47:09
So that data processing those data pipelines, that was his day
47:13
in, day out. I mean, and you even once you
47:18
are in the AI engineering world, you still have to do it.
47:22
It just like sometimes it is abstracted away for you by
47:26
someone else was kind of done it.
47:28
Like if you think about the you know, what, what was it a couple
47:31
of years red pajama data set, kind of like there's a bunch of
47:33
these things that exist in the world, right?
47:35
Or all of these are effectively things that you still have to
47:38
put in the effort for it to get to a point where you can
47:41
actually use it for AI training. So we we have we have an example
47:45
that I think is pretty neat to try and use the pile the the
47:49
large. Open sourcing corpus.
47:51
And just to show like how you can pre process it in different
47:54
ways. And like, I think we're doing a
47:55
benchmark at the moment to compare that against other
47:57
existing large scenario, large kind of competitors and, and
48:02
other things that exist to kind of do data data engineering
48:05
work. And like that's, that's a crux
48:09
of the problem that your friend is talking about.
48:10
No Of. Course, I can only imagine you
48:14
being the product guy. How many times a day do you
48:19
think about how you can add AI to the product?
48:23
Don't even get me started. A lot.
48:26
A lot across the board from using it by ourselves, making us
48:32
incorporated far closer to the various kind of AI use cases
48:37
across the board as much as humanly possible.
48:40
Right? Because it's like it.
48:42
It just supercharges everything if you use it for yourself.
48:45
And I'm a believer in the world as you know.
48:48
So like it's kind of it's not going anywhere.
48:50
There are different ways that you could plug it in, and I like
48:55
this idea of just bringing it to that world.
48:58
Or maybe you add a feature that is an AI feature potentially
49:04
where you're helping people dedupe or you're helping people
49:08
do what they need to do inside of the product.
49:11
So how do you how do you look at that?
49:14
The core question for us is like there's a set of use cases where
49:17
we can directly go and help those things out.
49:20
So that's kind of the classic one.
49:21
We're kind of doing that, you know, like I said, sort of pre
49:23
processing of the data. We can even speed up our
49:26
paralyze LLM inferencing, which is something intriguing that we
49:29
discovered because it's borders built on HPC like, you know, MPI
49:34
technology. So effectively, we can paralyze
49:36
the inference calls, which is kind of like a neat thing that
49:39
we discovered. But the the flip side of that is
49:44
like, are there aspects of how people can actually utilize our
49:48
product or our, our repo that we can make it easier from an AI
49:53
perspective? So one example that I'll tell
49:55
you here is that that we can working on is can we create like
49:58
a CHEAT SHEET that we can give to any AI that we speak with or
50:03
talk to, to say, can you make this code Bodo compatible?
50:08
And like it's something that we're working on because
50:10
effectively if you believe that the future will involve a lot
50:13
more a is actually working, then guess what?
50:15
The importance of read me's and those kinds of documents go up
50:18
rather dramatically. Yeah, that would be a great.
50:20
That's an example of that. Yeah, I was just talking to my
50:24
buddy Fausto about this and he was saying how Hopefully now or
50:33
very soon PDFs come with ALLM readable metadata piece.
50:41
Because that would be lovely. Man, PDFs are the bane of so
50:45
many people's existence, and especially when you're trying to
50:49
ingest them or you're trying to make them.
50:54
Parsing PDFs is surprisingly one of the still largest problems
50:57
that exist in the world. Which is weird, but it's true.
51:00
It's it's so weird and it's so difficult.
51:03
That is the thing like parsing them and then keeping the
51:08
context or if there's forms. I know a lot of people who are
51:11
in the finance realm and they say they really try hard to use
51:16
LMS, but the PDFs are the hardest area and if they could
51:22
just get around that parsing PDFs piece then everything would
51:27
be much smoother and so. I completely agree with that.
51:30
You feel like there's got to be a new standard.
51:32
Hopefully there's a way that we can add that metadata to PDFs
51:38
from now on so that it's easier for an LLM to understand.
51:44
Yeah, I think that's the, I mean, I think we will get there.
51:47
I think Jeremy Howard started this thing to add LLM stock TXT
51:50
to every website so that it's easier to scrape.
51:54
I feel like similar things are going to happen to a lot more of
51:56
these different normal existing formats as well.
51:59
Because ultimately like if that's how you read information,
52:04
then you got to make it easier for the LLM's man.
52:06
I mean like it's, it's really hard otherwise.
52:09
If you want an LLM to read it, because that's the other
52:13
philosophical point that is like, do I want to let the LLM
52:19
have all of my skews and the categories and everything that
52:25
I've worked so hard as a platform, this e-commerce
52:28
platform to aggregate, Then if I just give it to an LLM, is that
52:33
going to make my Moat not? I think it's a classic question,
52:39
but I feel like even if you don't want it to get out, you
52:43
might still want to do it so that your internal LLM can read
52:46
it and make your life easier. So in some sense, like I don't
52:49
think there is a Luddite defense here that works scalably, right?
52:55
I mean, you kind of need it's the train's only going in One
52:57
Direction. Yeah.
52:59
Well, we had a whole panel discussion on this because what
53:04
does that world look like? Where is it a Google agent then
53:09
calls your platform of whatever e-commerce agent?
53:13
Or is it that the Google agent goes from end to end?
53:16
Or is it that it's even before you go to Google and you've got
53:20
your own agent on your laptop? I mean it could be any of it.
53:25
Like in some ways you can easily see it being any of the above.
53:28
I feel right now I feel agents are still not strong enough to
53:32
be able to kind of what get autonomously deployed.
53:35
But if you replace agent with like situationally aware
53:40
function call, which is which, then 100% that's going to happen
53:44
now, right? Any piece of information that is
53:46
coming onto my laptop or any piece of information, if you're
53:49
Google that is coming onto your servers, you can have a function
53:53
like you can have a context aware call to kind of figure out
53:57
what you want to do with it or proactively as well as
54:00
reactively. I think that will change the way
54:03
that we actually deal with these things.