Latest / Agentic Conversations (formally mlops.community) / A New Way of Building with AI

Transcript

A New Way of Building with AI · Agentic Conversations (formally mlops.community)

0:00
But assume that the AI was the main driver of producing and
0:03
maintaining software, what we should design.
0:06
We actually asked the AI model, you know, if you wanted to add a
0:09
table, what you should do. And the AI model was like, yeah,
0:11
I'd like to have a function that adds tables.
0:13
We're starting to see more understanding in the ecosystem
0:16
of how to connect AI to software.
0:18
Now, a lot of sandboxes, they are usually like Jupiter style
0:21
kernels. They run, they disappear.
0:24
And it's like, oh, very firm, very firm.
0:27
Like the state is gone, the variables are gone.
0:29
It's not great because that's not how we interact with
0:32
ejecting systems. We chat with it, we come back
0:34
many hours later and we keep going again.
0:37
Or we go take a nap or wake up and go to sleep in the morning
0:39
and like can my coffee and start going in and like, your state is
0:42
gone now. That's terrible.
0:48
Even though there's much activity, it's also sometimes
0:51
hard to figure out what's working.
0:52
Yeah, but then it's so exciting. I guess it's like almost the
0:55
early days of, you know, just new technology, people adopting
0:58
it, getting it out there. And so I think for us, what we
1:02
are seeing, at least from our users, our point of view has
1:04
been, you know, this really kind of people who just want it to
1:07
work. And there's a lot of AI that's
1:10
really fancy, generates really cool things.
1:12
But end of the day, what we are seeing is that I just wanted to
1:15
do some work for me, you know, pass my emails, read my
1:18
invoices, get my air table in shape, you know, stuff like
1:21
that, right? And sometimes the, even though
1:25
we are seeing all this amazing fancy things out there, I think
1:29
the basics are actually important.
1:30
Like can it do the basic day-to-day operations that we
1:33
care about, right. And I think that's, that's not
1:35
what we're saying. Like people just want things at
1:37
work. And I think, I think that's kind
1:40
of come to like come back to the ground again of like there's
1:43
height, there's reality. Let's bring that together and
1:46
make your expectations are aligned, yeah?
1:48
Yeah, 'cause you can get into trouble if you're just going off
1:51
of the hype and then you're promising the world and not
1:54
delivering on much. Exactly.
1:56
Exactly. That there is all these
1:58
different potential use cases, but the ones that actually work
2:02
are few and far between. Have you seen some pretty common
2:08
use cases that are reliably? Working so for so I do try.
2:12
What we do is that we, we are building ATI agent that
2:15
completes tasks across your apps also be like this January agent
2:18
that works, you know, understands how they work across
2:20
different software, use emails, CRM, spreadsheets, so on.
2:25
We're starting to see actually a lot of people understand how to
2:28
use it and get a lot of success with it.
2:30
And I think the nature of the task that we normally see
2:33
working of those that you, you know, just data in one system,
2:37
you've got to pull it out. You've got to transform it in
2:39
some way using AI normally and you've got to put it back in a
2:42
different system. For example, get my invoices
2:45
that are coming through my emails, pull out the different
2:48
line items, get it into my accounting software or spread
2:51
spreadsheet. I'm working with QuickBooks or
2:52
whatnot, right? And it turns out that, you know,
2:55
if you think about that, that makes a lot of sense because our
2:58
software services are separate and silos, right?
3:01
They don't work well together. But it turns out that the AI has
3:04
the ability to now understand how to interact with which piece
3:07
of software you work with and be able to glue them together,
3:10
right? So we're starting to see that
3:12
that kind of work, that kind of processes work really well.
3:15
So the thing I just described is actually a process that works
3:17
really well on Nutra. But I think as we start to, you
3:20
know, I think what we're trying to see is 2 things in there.
3:24
I think 1 is that models, especially the current
3:26
generation of generation of models, have gotten really good
3:29
at understanding software. So code generation, cursor when
3:34
service it a lot. But at the same time, that
3:37
ability to produce software can now be something that everyday
3:40
business users can harness to automate work.
3:43
And #2 I think is that we're starting to see, you know, more
3:48
understanding in the ecosystem of how to connect AI to
3:51
software. You know, like, so we have been
3:53
working on this for a long time. And I think at the core of it is
3:57
figuring out what is the right AI computer interface.
4:01
It is, you know, I don't, I don't think it's going to be the
4:03
point and click. I don't, I don't think it's
4:06
going to be the raw API's. It has something in between that
4:08
is abstracted away for the AI to work with, you know, so MCPS,
4:12
things like this coming to bear. So I think when you have all
4:14
those pieces start to come together and click, you know
4:17
things are starting to work quite well.
4:19
Dude, it's funny you mentioned the ETL process, but now for AI,
4:25
I'm I'm sure you thought about that.
4:26
You realize, like, oh, it's just the same ETL stuff we've been
4:30
doing in data engineering, but now we have more ability to
4:36
decide what we're extracting and what we're transforming and then
4:41
where we're going with that. Maybe I'll give an example of
4:43
like some fascinating use cases and down the lines.
4:46
So one of the things that we found is that the the challenge
4:50
is that's your point, right? We used to be all stuck in
4:53
tabular data tools. You need to have a schema
4:54
database around it, but a lot of real world data is not tabular.
4:58
It's not in the schema like that.
4:59
So here's one example of a end to end process that, you know,
5:03
we see some users trying to do, which is I have a spreadsheet of
5:07
people sign ups, you know, maybe an area table or Google sheet.
5:10
There's some missing data in there.
5:12
You know, some of the form completely with, you know, maybe
5:15
the name, their addresses or their, you know, budget for
5:18
something, you know, and then I got to e-mail this 100 people
5:21
and say give me that data point you didn't put into your sign up
5:25
form. This cannot be known.
5:28
And now it's a ETL process because what you do is that you
5:31
read the spreadsheet and then you produce emails, drafts for
5:35
each of the individuals based on how are they filled in.
5:38
And then you send it out now, then they reply.
5:42
Now is a different ETL process. You take the emails that they
5:45
reply to you extract out using AI the data they're giving you
5:49
now and you fill it back into the spreadsheet and error table
5:53
where you need it. Now those those actually both
5:55
ETL processors. But if you're thinking about it,
5:57
those actually real work processors that we do every day,
6:00
you know, getting some form of a, you know, table of data.
6:03
Missing data? Having your own follow up?
6:05
Yeah. Exactly.
6:06
So, so fascinating that you start thinking that concept and
6:09
saying, because AI understands how to work with different
6:13
applications, so you don't have to program it and which API's to
6:16
call and say, here are the tools you can use and #2 because it
6:19
can take that unstructured data from like, e-mail content and
6:23
get the right pieces of information out.
6:25
Yeah, you know, you can start really ETL ING things, you know,
6:29
like extracting, you know, extracting and being, you know,
6:31
figure out how to use the tools, transform being using AI to
6:33
transform the data and then shove it into a different
6:36
system. Right, but I like how you made
6:38
that a verb. We are ETL ING a whole lot of
6:41
stuff now. You're not confined anymore.
6:44
How do you think about integrations?
6:47
Got it. Makes sense.
6:48
So the way I think about integrations actually, what is
6:52
the right level of abstraction actually to give an AI so that
6:57
it can be successful using a tool.
6:59
So I think a lot of integrations that we have today, AP is were
7:03
not designed for AI. They're designed for developers.
7:06
Now here all the error codes, if you do read it, you know,
7:08
program it deterministically to do all of these things, you
7:11
know, and then for an AI model, sometimes that doesn't make
7:14
sense. So give it to maybe 2 examples
7:16
in here. 1 is that we worked a lot with the Google Docs API
7:20
because people want to create documents and edit them.
7:23
Add to that, you know, use AI to write documents for yourself.
7:27
It turns out that there is a way to get the Google Doc API to add
7:30
a table into the document. Now, if you can figure it out,
7:34
kudos to you because it was impossible to figure out.
7:37
A lot of tinkering. And what he had to do was, you
7:40
know, create a table and then kind of like backwards compute
7:42
how to fill in every cell. And it's like insane, you know,
7:45
like you need, it's possible, but you need to like sit down
7:46
and really work through the details of that.
7:49
And any AI models going to struggle with that.
7:51
So the abstraction level that we want to get to for an action
7:54
like that will be one level higher, add a table to the
7:57
document and behind the scenes, we as engineers, developers
8:01
figure out what all the Garni and gnarly details and doing it
8:04
right. And so as we're doing
8:06
integrations, it's really thinking about that, right?
8:09
Because that's just one use case, right?
8:11
One use case and there's millions.
8:14
There's millions, there's millions of Google Docs with
8:16
adding a table instances that you have to backwards engineer.
8:21
Yes, so it sounds like that. I don't think it's so if they
8:24
look at the level of abstract getting different levels of
8:26
abstraction, I don't think it would go all the way to such
8:29
fine detail like we need to support this table, that table,
8:31
we just have to support tables, right?
8:34
So it's a bit one level higher than the raw API's, but not so
8:37
far abstracted that it only does 1 task, right.
8:40
So just one of the level to get to.
8:42
The thing about that is that the way we think about it, actually
8:46
it's like kind of in a kind of funny way, like instead of
8:49
sitting down and designing an API for the AI, we actually
8:53
asked the AI model if you were to wanted to, if the, you know,
8:56
LM, you know, if you wanted to add a table, what you should do
8:59
then the AI model like, yeah, I like to have a function that
9:02
adds tables. OK, great, let's do that.
9:04
You know, so it's almost observing, like if we give it a
9:07
set of integrations and then actions, what are the
9:10
inclinations of the model? What does it prefer to do?
9:14
And then we kind of actually adapt the way we design the
9:18
actions to what the model prefers.
9:20
So we actually designing around AI preferences, which is a very
9:23
different thing to do. 1 very one surprisingly non trivial
9:27
thing. Spreadsheet filling.
9:29
Yeah, OK, very common task. I have a spreadsheet.
9:31
I need to fill in data for every role.
9:33
Maybe do some research on the web, fill it in.
9:36
There are many ways to go about doing that.
9:38
What is the right abstraction to give an AI model to say this is
9:41
the way to update a row? Yeah.
9:43
Do you give it just the functions to, say, update a cell
9:45
with coordinates? I was going to say yeah, maybe,
9:48
yeah. Turns out that if you do that,
9:50
you're going to get off by 1 errors.
9:51
It's going to be like, oh, I think there's a header row plus
9:53
one off by one, everything. Oh no.
9:56
So now if you give it an abstraction in which I'm going
9:59
to give you the input data that represents a row and you tell me
10:03
the columns to update based on the no header as keys, now it's
10:08
going to work. No more off by 1 errors, no more
10:10
issues in there. So once you did, once you figure
10:13
that out, for example, then the errors that the models make go
10:17
away because it's sometimes impossible to make the same
10:19
errors. And then you can see like, you
10:22
know, there's a model like this some ways, right?
10:24
Because you give the model, this is the way to update it.
10:26
Is it getting it right or is it trying to do something else?
10:29
You know, what this makes me think about is how the you're
10:33
giving. I always think about trying to
10:38
give the LLM the least amount of scope as possible and really
10:41
directing it and almost ordering it to do one thing and do that
10:46
one thing well. But you're coming at it with
10:49
this vision of what let's work together on it.
10:54
You have a certain level of understanding and capabilities
10:58
that work fairly well. And so take that into your
11:03
understanding and do it. You're you're just giving it
11:06
more scope than I would have been if I were to attack this
11:10
problem. Exactly so.
11:12
So this is actually the one of the key principles we use in
11:14
developing like AI integrations. It's like, look at what the
11:18
model is doing, but the model is not doing what you think it
11:21
would do. Don't go against it.
11:23
In fact, take a step back and go like, we should redesign our
11:26
interface so that that thing that it wants to do works.
11:28
And then actually it becomes very easy because like you're
11:30
not fighting the model now, they're just doing what is
11:33
really naturally programmed to do or have learnt to do in data.
11:37
And it turns out that funny enough, because everyone trains
11:39
on the same data, all the models exhibit the same, the same
11:42
tendencies because of that. And, and so we actually see that
11:46
across the platform where, you know, once we get it working and
11:48
you know, one model from Open AI, the same system, then it
11:51
works also with entropic models and Google models because the
11:54
tendencies are very similar in all of them.
11:56
So then what are some other ways that you've almost created
12:02
workflows or programmed with the model?
12:05
There are many different ways to get models to interact with
12:07
applications. 2 very common frameworks are React, which is
12:10
you get it to move function calling over and over again.
12:13
That's the most popular one. And then I think the other one
12:15
is called Kodak. You know, you get it to write a
12:18
bit of code and you run it and see the outputs.
12:20
So we are a big fan of the Kodak's approach.
12:23
So we have been doing that actually before the paper came
12:25
out. And that should be the approach
12:26
to be because we saw the first demos of GPD 4 was about, no, I
12:30
think Greg Brockman is on stage. You know, demoing like Co
12:33
generation is like, wow, that's going to work.
12:35
This is like very early days. So we have always been big on
12:38
that, that approach. And it turns out when when you
12:41
take the Kodak approach, there's a lot of things you can start
12:44
doing now. So for example, you say, hey,
12:48
model, I want you to produce code and I need the code to
12:52
respect types. So when you call this function
12:55
and this function and that function and you, you know, pass
12:58
the data, you know, data around and you have a variable that
13:00
goes around, you know, we can't infer the types because it's
13:03
just code. Yeah, we can say that type must
13:05
be X. And when you pass it down, it
13:07
has to be a string. Or at this point of time you got
13:09
to commute it to a number and pass it down.
13:12
And in that approach, you can say, produce a little snippet of
13:14
software code. We can check whether the types
13:17
match up and it doesn't match up.
13:19
We can go like, not good, go back and do it again.
13:21
Yeah, right. And then you can see them and
13:22
the model will fix itself. And so now again, it's just
13:25
putting this guard rails on and saying the model like you gotta
13:28
stay in the guard rails. And the guard rails come from an
13:30
external system that it's about understood type tracking and
13:34
programming. So the model gets better and
13:36
better is over time too. Yeah.
13:37
And that's actually a really good thing that that works
13:40
really well in practice. Are you using pedantic AI for
13:43
this? Because they that's a whole
13:45
value prop that they have. We are.
13:47
We're using Pedantic for a lot of like type tracking and typing
13:50
things for sure. We use it a lot for schemas and
13:53
serialisation of data. We use some degrees of pyrite, I
13:57
think for type tracking, but then it's also interesting in
14:00
that that's a balance too, right?
14:02
So here's the here's the fun part.
14:04
Yes, type tracking is great. Get models to do type track
14:06
code. It turns out that in Python
14:10
variable shadowing, which is reusing A variable name that you
14:13
once used, you know, it's a so it's very common thing, you
14:17
know, like, you know, you reuse the index in loops or something
14:19
like this. Type checkers don't like that
14:23
because like you said, it was a string.
14:25
Now it's not a string anymore. What's going on guys?
14:27
You know, it has to be the same type in the whole function.
14:30
So it turns out that going back to our first principle, the
14:33
model keeps producing code that doesn't respect that particular
14:37
rule. Yeah, great.
14:40
Throw, throw that rule out the window.
14:41
Oh, wow. Throw that rule out of the
14:43
window, but keep the rest of the rules.
14:44
Yeah, right. So there's some rules that you
14:46
can be kind of like also guided on like a great, the guardrails
14:48
are there, but this guardrail, the model really hates it.
14:52
You know, then you kind of really say like, is it a good
14:54
guardrail? Yeah, it's not a good guardrail.
14:55
Throw it away. Right.
14:56
So it's like really like AI first, kind of like AI first
15:01
system design. If you were to design A
15:04
environment for coding, for execution, for software to run,
15:08
but assume that the AI was the main driver of producing and
15:12
maintaining software, what we should design, right?
15:16
And that's kind of the the mindset we have over here, yeah.
15:20
Having the guardrail, but if for some reason it keeps not liking
15:24
the guardrails, see if you can format that guardrail
15:28
differently, yeah? Yeah, make it in something
15:30
that's actually like not valuable for the model, right.
15:32
So and so like, yeah, a lot. That's a lot of the ways to
15:35
think about it. Like what does the AI model want
15:37
to do? Is this something reasonable?
15:41
Is guardrail A guardrail? That's really important and you
15:43
to bring that up. Yeah, because there's probably
15:45
like hardline guardrails that you cannot change, but then
15:48
there's other ones that you just have there for some reason that
15:52
you thought it would be more useful.
15:54
You thought that the model output would be more consistent
15:58
or more accurate. And if for some reason it's not
16:01
liking that, then well, how can we work around it?
16:04
How can we maybe work with the model as opposed to trying to
16:07
put yet another guardrail on it? Yeah.
16:09
So I think when you see, I mean the two approaches, right one
16:12
side, go back to the prompt and prompt it more to do what you
16:15
wanted to do or change the way you're set up the system.
16:19
So it just accepts what the model is doing.
16:21
And the latter turns out to be a lot more effective we found
16:24
because it's you're not fighting the model anymore.
16:26
And so just really figuring it out.
16:28
I think it's been been a journey for us actually, because we're
16:31
happy. It's like the models keep
16:32
changing, keep evolving, getting good and bad in some things.
16:35
This episode right here is sponsored by the good folks at
16:38
ML Flow, the open source platform for teams looking to
16:42
track, manage, and deploy ML and generative AI projects at scale.
16:48
If you spend hours keeping track of different LLMS, prompt
16:52
versions, debugging agent hallucinations, or filling out
16:56
spreadsheets of experiment results, ML Flow is the right
17:00
tool for you. This year the ML Flow team
17:03
launched ML Flow 3 Point O, the first major version update in
17:08
four years, along with a fully managed version now available to
17:13
everyone, built and maintained by the original creators of ML
17:17
Flow. Shout out to Ben Wilson, we see
17:21
you man Managed ML Flow gives you access to full-featured
17:26
enterprise ready ML OPS platform with 0 setup required.
17:31
You get effortless experimentation, tracking,
17:33
automated model versioning, evaluation and seamless
17:37
deployment. All that plus the cutting edge
17:39
features for Gen. AI workloads like automated
17:43
tracing, agent evaluation, and one click monitoring dashboards
17:48
for your apps and agents with 0 infrastructure headaches.
17:52
Ready to try it out for free? Well, visit mlflow.org to get
17:56
started. That's mlflow.org.
17:58
Let's get back into this episode.
18:00
Every tool has its own way of working.
18:04
And I can imagine you are constantly thinking about we
18:10
need to create more integrations with more tools.
18:14
How do you look at that? And I know you mentioned MCP
18:17
before. Is that something that you're
18:19
just like, all right, well, cool, Soon enough we're going to
18:22
have MCP servers for all these different applications and we
18:26
won't have to think so hard about the integrations.
18:30
So put things in there. I think MCP is very new and we
18:34
do see a nice, you know, potential long term future,
18:36
right? It does become a standard that
18:38
lots of people adopt. I do see a world in which every
18:40
company that has an API right now has an MCP server next to it
18:44
that is somewhat like the API. It may be a bit different and
18:47
that is, you know, molded around how AI should use it with
18:49
prompts and a set of functions that they create.
18:53
But it's not, we're not there yet, right?
18:54
And we're still figuring out what that should look like.
18:57
So for example, MCPS today, they're not output schemas and
19:00
we think figuring out the right output schema of MCP call is
19:04
really important. So the way we've been thinking
19:06
about it is like, one, we have become MCP compatible at Lutra,
19:10
so you can actually use all MCP servers with Lutra today.
19:13
But two, we want to make it possible for people to also
19:16
create integrations today without waiting for MCP server
19:19
to be available. It turns out that what are
19:22
integrations? Integrations are a translation
19:25
of how, you know, our external system normally APIs work and
19:30
how that should be adapted to our internal way we integrate
19:34
with AI. Translations are very open to
19:39
being, you know, AI translated. And So what we have built and
19:44
shipped in the last last month actually was a way to take API
19:47
docs from any system, have a service that looks at our
19:52
documentation on how we set up things, how we configure
19:54
authentication, also generate an integration, all the code needed
19:58
for that, and then immediately, instantly test it out.
20:03
And then upon testing it out, because it's an important part,
20:06
it's going to not always work because of the edge cases and
20:09
everything. Be able to have the Model C live
20:12
data come in, fix the integration.
20:15
And I rolled it out. So there's something that we've
20:17
been doing it's it works really well with like, you know, silver
20:20
based databases that you can bring into Lutra CRUD based APIs
20:24
just out-of-the-box right now. Yeah, it gets a bit more tricky
20:28
when you start to work with really complex systems like
20:30
CRMS, but we're getting there. Yeah, so with.
20:33
You know what, it's so funny that you're saying this because
20:35
back in 2022 or 2020, early 2023, right after Chachi BT came
20:42
out and there was this boom. And it was before folks were
20:45
really doing stuff with code. Someone was in the ML OPS
20:49
community Slack and they said hey I built this framework and
20:53
all it is is you talking to an LLM telling it what you want,
20:58
but instead of the LLM generating code, the prompt was.
21:07
What would this look like in code in your language?
21:10
Like and so the LLM would just output its own basic gibberish
21:15
and its own code. It's and then it would create
21:19
its own language in a way. And then it you could re put
21:24
that into an LLM and and for some real weird, really wacky
21:30
reason, it understood it. Yes, I think the models actually
21:34
have some tendencies, right? They like like to process it in
21:38
some ways or structure code or some things in some ways.
21:41
Yeah. And I think that's when you
21:43
describe that, it actually reminds me a lot of almost like
21:46
chain of thought reasoning, right.
21:47
The first chain is actually to produce some pseudo code which
21:50
is like this infernal gibberish. And then after that, you know,
21:53
or maybe a plan, and after that produce the actual thing you
21:55
want to do. And then it kind of understands
21:57
that too. Yeah, so we've seen, we've seen
21:59
that a lot. I mean, it's fascinating that
22:01
reasoning models work very well today because that whole process
22:05
is now embedded into the reasoning model.
22:07
And you know, we definitely see like a big step up in
22:10
performance when we move to that.
22:12
Yeah. And so how are you using
22:14
different reasoning models so. We, we decided to be pretty
22:18
model agnostic from day one. So we use models from different
22:22
providers in our back end. In fact we stop between them and
22:24
like falling back when model providers not available.
22:27
And it works very limited actually works pretty well.
22:30
And and the way we think about reasoning models in particular
22:34
is on models in particular is that every model actually has
22:38
different strengths. So it turns out like for example
22:41
Gemini flashing models are really good at data extraction.
22:45
Give it a bunch of data really fast, really high quality data
22:49
extraction comes out nicely. Even the multimodal data even.
22:53
The multimodal exactly stuff is really, really good at that.
22:56
But you know what they're not, I don't think they're great at
22:58
writing, you know, and then these clot models, she write
23:01
really nice content. You know, ChatGPT now is the
23:04
best image model. And so we think of it as can we
23:09
help users or dynamically figure out the right model for the
23:12
right tasks when they need it? All right, so if someone comes
23:16
to our platform and says I got a bunch of PDF files, I need to
23:19
extract the data and put it into a spreadsheet and.
23:21
You're like, Ding, Ding, Ding. I got just the model for you.
23:23
Exactly. You say like we're going to use
23:25
Gemini Flash to do all the extractions.
23:27
We're going to use Sonnet to figure out how to map the data
23:30
because there's a lot of reasoning behind mapping the
23:32
data. And then, you know, if you run
23:35
into any image stuff, maybe use a different model there, right?
23:37
And so that's what we, that's what we're doing in practice.
23:40
And then the reasoning models, that part really comes in
23:44
critical at understanding how to tie different systems together,
23:48
right? So it's like I need to work with
23:49
system A, system B. These are the setups, the API
23:52
sections I have. What is the way for me to tie it
23:54
together? Reason about that a bit.
23:56
Write some code, run it. No, maybe it works great, maybe
23:59
it fails. And if it feels reasonable why
24:02
it fails, try again, Go. So that reasoning is really
24:05
important in orchestration and that's why we use visit mostly.
24:09
So is it also orchestrating what models to be using?
24:13
So yes, in practice, yes. But what we do is that in the
24:16
back end, we actually we do our own testing and predefined like
24:21
no data extraction. We like Gemini Flash.
24:24
Now we're looking at them, you know, but then at the top level,
24:27
it's figuring out what is a task, is a task creative
24:30
writing, report generation, web research, or is a task, you
24:34
know, data extraction. And then depending on the task,
24:37
we have hand selected the models based on just evolve sets we
24:40
have, yeah. That's such a great way of doing
24:44
it and also not sticking to just one because you're like, oh,
24:49
well, I like Gemini the best. No, it's Gemini's great for
24:53
these tasks, so let's use them. Let's predefine that we're going
24:57
to use them for that type of thing.
24:58
And then if we need others, then we'll go and and find others.
25:03
How are you thinking about super complex and long running jobs?
25:09
Yes. We the longest task someone has
25:11
to try to do 2. I think 2-3 days to run really.
25:14
It's like, imagine you press enter, it's like for three days
25:17
it comes back. No way but.
25:21
I think that there are different kinds of long running task.
25:24
They're the kinds, they're one kind, which, like you're doing a
25:28
bit of it. You need to see what it is and
25:29
you're always re planning re, you know, reorienting around the
25:33
outcomes and then doing it more and more and more.
25:35
That's one kind of long running task, very high complex
25:39
knowledge work, correct that maybe are doing a deep research
25:42
report about something and you got to make talk to young
25:45
people, click data. That's pretty hard.
25:48
There's another kind of long running task which we think AI
25:51
is well suited for today, which is do that same damn thing, but
25:55
over 100,000 rows do the same thing over repeated over many,
25:59
many things. But this, you know, verify me
26:01
that's doing the right thing. So right now where we are I
26:04
think is so we're really good at that, right.
26:07
And then over time the formula will get really good at too.
26:10
But the latter is actually where we are today, so.
26:12
It's almost like if you can do it once, well, yes, you can do
26:17
it. You verify that it can be done,
26:19
and then you just do it a lot. A lot.
26:21
Yes, so, so for example, the task in this case was to some
26:25
extent I have a spreadsheet of accounts, websites I want to
26:29
research on, maybe all the people attending a conference,
26:32
maybe all the nameless or something I got from somewhere,
26:35
every single person in MLS community.
26:36
What are you doing right now? Pretty big task.
26:39
I go to every single person, web research them, look up that on
26:43
LinkedIn, look them up on Google Scholar, put all the data pieces
26:47
together, maybe even score that like give me a final score on
26:51
like, hey, should this person is be a good candidate to interview
26:54
on this podcast? I need that, you need that.
26:56
And then, but then I was like, OK, I got a 5000 people or maybe
26:59
100,000 people in my community. Run it on 10 people.
27:03
I like the scores. Go run it on 100,000 people
27:05
right now, right. So now this is where,
27:08
interestingly, this is where we tied back to a bit of like, how
27:11
do you identically automate work?
27:13
And this is where a bit of, you know, why we think matters like
27:16
code act shine versus React. Because now if you're producing
27:21
software code, you can go like, I want to write the function
27:24
that does it for one row. I'm going to run that function a
27:27
few times for you, show you the output.
27:29
That's great. Now I'm going to write the for
27:30
loop that goes through all the rows and just keep running until
27:33
it's done. You can start imagining now
27:35
that's only like 4 to 5 model interactions and you're off the
27:39
races. Now if you did something more
27:41
like the React framework, every single row is a single LLM call.
27:45
You're never going to finish this because it's like.
27:46
And it's so much more expensive. Much more expensive.
27:49
Doesn't make sense. Yeah.
27:50
All right. So this is where, you know, we
27:51
have already been focused on that approach because it scales
27:54
very nicely to many different kinds of real world tasks that
27:57
people do. So the thing I just described is
28:00
actually a task people do like all the time right now. 100% and
28:04
if you're in sales, even more. And now think about that as
28:08
like, why is it input a spreadsheet?
28:12
What if the input was my HubSpot CRM?
28:15
Oh, what if they input my hubs or CRM on a particular list of
28:18
target accounts I carry about? Can it just read my data from
28:22
that system directly, do this work and lock a note back into
28:26
the CRM directly? Go.
28:29
There's no difference conceptually from the CRM and a
28:31
spreadsheet. No high the high, very high
28:33
level, right? They're all database.
28:35
Yeah. But the hard part traditionally
28:37
has been I need to figure out the APIs and integrations.
28:39
But I was like, oh wait, I have the.
28:41
If I have the integrations ready to go into AI understands it,
28:44
I'm just a prompt away. Yeah.
28:46
So we have hotspot integrations today and that's actually
28:49
something they will do on their tasks, yeah.
28:51
I'd love that. Yeah.
28:52
You're thinking about attributes or whatever it is, these custom
28:55
fields that you have in HubSpot or Salesforce or whatever.
28:59
It is the entire object with properties like.
29:00
Could be nested, could be, you know, arbitrarily complex what
29:03
you want it to be. Turns out that this is where
29:06
integrations get hard too, because the more system allows
29:10
you to configure things to a very complex degree, the more
29:13
the AI needs to understand the schema behind the configuration.
29:17
So we spend a lot of time on our end as well figuring out like
29:20
how do we communicate user defined schemas from external
29:24
systems into our AI agent so that when our AI agent operates,
29:28
it's able to understand the schemas.
29:31
For example, if you had HubSpot or Airtable, one of these things
29:34
you say, I want to update a drop down field with the value, you
29:38
know XYZ. It's not one of the drop down
29:41
options. The the system will reject it.
29:43
So you need to know the schema to work with it.
29:45
And then now you know the it's going to make the mistake of
29:48
doing the wrong name. Maybe it puts a space between,
29:52
you know, qualified lead and the the thing in HubSpot often have
29:56
no spaces. It's qualified lead without
29:58
spaces there. So how does your AI agent then
30:01
understand that error comes back, fixes itself and go again,
30:04
right? All of those things are things
30:06
that we spend a lot of time thinking about and architecting
30:08
for. So simple on our end, we look at
30:11
it and it's like qualified lead, no fucking space.
30:14
And then for the agents, it's a little bit different, yes, man,
30:21
but that is that is so cool to think about how how much it can
30:25
unlock when you don't have to say export as CSV and you can
30:32
just let it run loose directly. This makes me wonder about
30:39
taking this data and mixing your processes of the agents steps
30:46
with traditional data processing and how you're looking at that.
30:52
Because I imagine you have you, you mentioned you have certain
30:57
models that you like to use for certain tasks.
31:00
Obviously there's certain tasks that are better done with Pandas
31:05
or things like that or Spark or whatever it may be.
31:09
How do you like integrate the two of these?
31:13
So this is where it's actually really, it's like it's really
31:17
beautiful in some ways when you tend to use going back to get to
31:21
the Kodak framework. So if you're in a world where
31:24
you're orchestrating between apps, not just by function
31:27
calling, but by code generation, Pandas, numpy, whatever can just
31:32
be part of the output. And it is.
31:33
So today when you use Lutra, it actually produces, you know,
31:36
Pandas pipe plot all the stuff that you like.
31:39
It is code as well. So you could go be like, OK, for
31:42
this part, do this thing that cause the AI function that
31:45
extraction. For this part, we use pandas to
31:48
do some kind of analysis for this part, use test model to do
31:51
a regression analysis. Or for this part, I generate a
31:54
HTML file using a model. So it's a nice pretty website
31:57
that you can present the data now.
31:59
So now you can start composing across all your existing tools
32:02
you like that do data processing with the AI tools that are new
32:07
and bring it all together into one place, right?
32:09
And so that's really nice. The hard part of this turns out
32:13
turns out to be actually runtime.
32:15
So if you think about it, most of the time in the while, when
32:18
you see people doing a code execution approach, the code
32:23
runs in a very tight lockdown sandbox.
32:26
A code interpreter from Open AI that's been around for a while,
32:30
you can use Open AI to HGBT, say write some code for me, do that
32:33
because it's running it in Python.
32:34
That Python environment is in a lockdown sandbox that can do
32:38
nothing. It's like run some Python code
32:40
very fast, not really nothing. But you can't really, we cannot
32:43
access the web, cannot call APIs, it cannot call out to
32:46
other AI functions or, or external systems.
32:51
Now that's where we took a step back, you know, like how do we
32:54
then make it possible, you know, and how do we then consider all
32:57
the security implications of making a sandbox that can access
33:02
the external world such that the AI is working in a sandbox, but
33:07
it shouldn't be able to access these key secrets and keys, but
33:11
it can call out to external world.
33:12
It can bring data back. Wow, that looks like right.
33:15
And then so that's what we what we've been planned a lot and
33:17
designing on. And then second thing is that
33:20
now a lot of sandboxes, they are usually like Jupiter style
33:23
kernels. They run, they disappear and
33:26
it's like, oh, very firm, very firm.
33:28
Like the state is gone, the variables are gone.
33:31
It's not great because that's not how we interact with ejetic
33:34
systems. We chatted that we come back in
33:36
many hours later and we keep going again or we go take a nap
33:40
or wake up and go to sleep in the morning and be like, can my
33:42
coffee and like start going in and they're like, oh, your state
33:44
is gone now That's terrible. But we also cannot keep, we
33:47
cannot also no reasonably keep all the machines running all the
33:51
time. So how do you make an
33:53
environment that is more stateless, right?
33:57
So we design, we have designed actually more of a stateless
33:59
environment. So a stateless secured sandbox
34:03
iconic that enables the AI to operate it.
34:07
So that's actually the core of the the core.
34:09
That's the hard part, I guess. The core of it all, yeah.
34:12
There's so much that you just said there that I want to dive
34:15
into, especially on this, this stateful and stateless part and
34:19
how you're enabling that because there's sometimes that you need
34:23
it. There's sometimes that you
34:24
don't, and if you get the wrong time you can either lose data or
34:30
you can lose processes or or a long running job, or if you keep
34:36
it on too long you can just spend a lot of money.
34:38
Exactly, exactly. So there's there's a lot of
34:41
design decisions in that one, but at a high level, if you if
34:45
we just, you know, abstract ourselves into a set back go,
34:47
like what are computer systems? You know, you know, like what's
34:51
what's the state? They were talking about state
34:53
food stateless, right. The state is actually like
34:54
what's in memory really, And what are the instructions in
34:58
memory don't execute against. That's all the state that we
35:01
need. We don't have to keep the
35:02
machine around. What we have to keep around is
35:04
what's in memory. The state, yeah.
35:06
Is the memory state and the current instruction state, which
35:08
is what's in the, you know, instruction stack, what's on my
35:10
functions on the stack, What can I run, what I can call?
35:13
What are all the variables on my stack, right?
35:16
One easy way is to try to serialize the entire machine
35:18
state and say, spin up a new machine, load it up and go
35:21
again. Kind of slow.
35:22
You know, you can't imagine how that works.
35:24
Whenever you you chat and finish, just shut down the
35:26
machine, saves the state away. Kind of a poor man solution
35:29
there. Yeah.
35:29
Another way is to, you know, be able to, you know, bootstrap
35:32
machines and load up states much faster.
35:34
Another way is to rethink a whole runtime and like, let's
35:38
design a whole new runtime that is designed around being
35:41
stateless. And then, you know, execute
35:43
code, run it, writes it to memory, but really control the
35:47
entire, you know, interpretation of the code and how you start at
35:52
the variables. That's another way to do it,
35:54
right? And so we are much closer to
35:56
that approach where we actually, you know, we don't just exact,
36:00
no eval, eval, eval. The Python code that comes out,
36:04
we actually inspect it, we look at it, we run it.
36:08
Yeah, we actually integrate it in the way we want to.
36:11
And that gives us a good amount of control, right?
36:14
So for example, one of the challenges that we quickly run
36:17
into when we deployed this was someone loads up their 100,000
36:21
row spreadsheet. Go do try do this for me.
36:24
It starts running. If you ran Python out-of-the-box
36:28
or any programming language, it's like there's no output.
36:31
You know, it's like like the terminal is like, you know, like
36:34
there it's running and like, is it running?
36:36
I don't know, you're on top. You can see like CPU is running.
36:39
So I guess it's good. Wait, no, it's been going.
36:41
But you really want your execution runtime environment to
36:45
give a feedback. Yeah.
36:47
How long has it been running for?
36:49
What actions is it taking? What is it doing right now?
36:52
I didn't want to see a real time indication of what's it doing.
36:55
Yeah, in the UI. Yeah, that would be brilliant.
36:58
Of course, just like we see the streaming of the ChatGPT giving
37:02
us the answer back. But exactly when the code runs,
37:05
you want the stream too. Yeah, but for the execution of
37:07
the code. And then maybe you even want an
37:09
estimation. Can we, you know, do static
37:12
analysis and say, oh, this is how the code is laid out, This
37:15
is the loop. It's going to be, you know, 9000
37:17
iterations. Each iteration has been running,
37:20
you know, looks like it's about 5 seconds for one, one
37:22
iteration. Yeah.
37:23
Oh, yeah. So like 9000 * 5 seconds.
37:25
That's how long it will take. Yeah.
37:26
Like, that'll be nice, right. So just sort of things that
37:28
we've been, you know, looking at thinking, you know, designing
37:30
around. And so if you use Nutra today,
37:32
you start to see hints of that. So even when Nutra starts to run
37:35
a piece of code, you actually see real time live streams of
37:38
actions it's taking. You start to see indicators of
37:41
like time estimates, you know, and then can make all that
37:44
better and more exact. And then also more importantly,
37:48
AI functions are expensive. Yeah.
37:49
So this thing is running for you 9000 times.
37:52
And in the middle is calling some model.
37:54
It's going to rack up a lot of tokens soon.
37:56
Yeah. And so estimating not just the
37:58
time but also the cost. Oh I love.
38:00
That so this might cost you, you know, 5000 credits, AI credits
38:04
to run because you're going to consume that many tokens, right.
38:07
So that's all the things that had to come in the picture when
38:10
when you really start thinking about designing around, you
38:12
know, AI operating software. Yeah, yeah.
38:16
I love that so much, how on the cost.
38:18
And it also feels like you have a chance to warn people or set
38:24
up alerts or do something where maybe the first couple
38:28
iterations you don't do anything, but you take a pause
38:31
and you say, hey, by the way, if this is going to go 9000 times,
38:36
it's going to probably cost you this much.
38:37
Are you OK with that? Exactly.
38:39
And then you can have that human say yes, which you don't get in
38:44
traditional sequel and stuff. I remember a Horror Story of a
38:48
friend who did some kind of a join and he was working at
38:52
Spotify and then a week later somebody knocked on his door was
38:58
like, hey, why did we just spend $40,000 on something with the
39:04
like this? You know, our database bill just
39:07
spiked and he was like, oh, that was a big join that I did.
39:11
I didn't realize how big it was, you know, And so something like
39:14
this where it's conscientious and the alerts are built in and
39:20
you're ready to say you can say if I am about to spend over X
39:27
amount, yes, warn me. Exactly, exactly.
39:31
And then I said this, this exact the things that we think a lot
39:33
about because we had versions of Lucha earlier on that didn't
39:37
warn and the users like I send one prompt and oh, my credits
39:41
are gone now. What happened?
39:42
They were like, oh, shoot. Because it's, it's just a
39:45
prompt, right? Yeah.
39:46
But meanwhile, there's a whole lot of stuff going on in the
39:49
background. Exactly.
39:50
So we are entering a world in which a single prompt, you know,
39:54
could set off a whole chain of processors.
39:57
No, today, you know, I think there's some reports and
39:59
articles like can do, you know, you know, minutes, hours of work
40:02
for you. In the future, I can do like,
40:04
days of work for you, months of work for you.
40:06
But start thinking about it. Wait, that's.
40:08
There's a lot to design around that.
40:09
Yeah. Because if one prompt can just
40:11
do that, you want to have all this, you know, ways to help
40:14
users understand the implications of that one prompt.
40:18
Right. And today we know.
40:19
And that's why we're designing around here.
40:21
We're really thinking hard about.
40:23
Yeah. And I can also imagine a world
40:25
where you would want to try and give some advice or or help say
40:30
where maybe it's like, here's what we've decided, right,
40:35
here's the choice. And you can click in and maybe
40:40
you're like, well, maybe all, not all 9000, let's bring this
40:43
down to 2000 and we'll just see if I can find any value there.
40:48
Exactly like start start small to a few 600.
40:51
Let me review it first or even better, AI.
40:54
Can you review own work? And if it's good, let's do the
40:57
rest of that right. And if I could think, you know,
40:59
so application developer, this also is really nice because you
41:03
know, you know, from a, you know, even like a business model
41:06
perspective, a lot of AII think will be more usage based, right?
41:10
The more you use it, the more that what the AI does for, you
41:13
know, the more you only charge for that too, right?
41:16
And so if you can help people understand the value of the work
41:19
and the magnitude of the work, it also means that it's a very
41:22
natural business thing to do where start, you know, paying by
41:25
the amount of work it does. And so it's, it's really nice
41:28
and it aligns expectations right and, and you do not want it to
41:31
go off and do 9000 rows of work if they're all going to be
41:35
really bad. And that's the worst thing you
41:37
can, you know? Yeah.
41:38
Then you're paying for it. Yeah, it's like.
41:40
You know that you hired an intern, the interns like doing a
41:42
terrible job by working really hard.
41:43
And it's like, ah, got to redo everything.
41:45
That's not good, right? But then I think that's, you
41:48
know, that's all challenges to design in this space right now,
41:51
especially you. And you know when you say 1
41:53
prompt can do days of work? Yeah.
41:56
It's just long running jobs, man.
41:59
Wow, you are thinking a lot about system design with AI.
42:04
How do you feel like system design will change in the coming
42:08
years? And especially if we're thinking
42:11
AI is only going to get better, the models, the underlying
42:13
models. When I say AI, that could be
42:16
interpreted as many things, But if we're looking at the
42:18
underlying models getting better and being able to understand
42:22
systems better, how do you feel like system design is going to
42:27
change I? Cannot view quality of models in
42:30
a few dimensions. The first very obvious one is
42:34
context window length, but it's very important.
42:37
The reason for that is that, you know, back in the day of GBG 4
42:41
when it was like, I don't know, was it 8K or 32K?
42:43
And I should be people remember this.
42:45
It was tiny context windows and we had to do a lot of
42:49
workarounds. All this work around the rag was
42:52
actually a lot of it was because the connects windows are too
42:55
small. So our snippets in the rag are
42:56
very small. But now it's like shove the
42:58
whole document in the PDF file and go with it, right?
43:01
So as that gets bigger, that really changes the way we think
43:04
about things and what we can give the model as context if it
43:06
can reason about them really well.
43:08
Second one, I think is oddly enough costs.
43:13
No cost and equality and equality and equality of the
43:15
outputs. And the reason why cost is
43:18
actually very interesting and important is that essentially
43:24
it's more salient. Actually, if you compare it to
43:26
maybe a year or two ago with GBD 4, it used to be that a single
43:30
call to GBD 4 was, I don't know, on an order of like $60.00 for
43:35
million output tokens, that's only $120.00, like some crazy
43:37
amount. And now it is on the order of
43:39
like $3 to $5.00. So it's about like 1020 X
43:42
reduction. So put it this way, if if you
43:45
were to ask the AI to do a task and it had to make 20 calls to
43:49
figure out the right things to do, and each call was about, I
43:52
know $0.50, right, 20 calls, $0.50 to $10, too expensive.
43:58
And I don't want to ask any AI to read my emails and summarize
44:01
them with $10. I just do it myself.
44:03
But then if that's a, that's a few cups of coffee right there.
44:06
But. If you take $10 and divide it by
44:08
20, suddenly that's, you know, that's way cheaper, right?
44:11
Like $0.50, right? Yeah, sure, $0.50.
44:13
I will do that. I can do that now 20 Times Now.
44:16
You know, that's great. But then when you start thinking
44:19
about that, why do you want to do that?
44:20
It turns out that a lot of the times the models are not
44:23
perfect. They're not one of single shot
44:24
give you the right answer and the more you can get the models
44:27
to decide like it. I'm going to do a task.
44:31
What do I need to get? Somebody can now based on data,
44:34
I guess, what should I do next? That iterative way of using a
44:37
model over and over and over again is very powerful.
44:39
If we know it works well, it's very agentic, but it's a ton of
44:43
calls. So if the cost of models keep
44:46
going down, then more developers and more users like myself will
44:50
be very encouraged to call them a lot.
44:53
And calling them a lot actually improves the performance a lot.
44:57
So cost there's actually like a big thing right now because I
45:00
think the malls are really powerful right now.
45:01
Like if you if you put on cost by 10X today, I suspect we will
45:04
see no a lot of gains just you know like how people do
45:09
reasoning because just keep calling it more.
45:11
The third one thing is just like no quality of outputs, reasoning
45:14
traces and so on. I think it's gonna be better
45:16
too. So a lot of the work we have
45:19
done just looking back has been working around those things.
45:22
How do we work around context windows being too long?
45:25
How do we work around being, you know, saving on number of calls
45:29
we make? And then how do we work around
45:31
on like, you know, trying to be smart about which models we use
45:35
from a quality perspective when right.
45:37
But moving forward, those are not considerations then should
45:40
just be using them like, you know, it's like we'll stop
45:43
worrying about what to put in the connect window.
45:45
Just put as much as we can, we'll not worry about how often
45:49
we make calls. We just call as much as we can.
45:51
And then quality wise, I think it's going to be no increasingly
45:55
better, more selection, more selection too.
45:58
And so that's kind of like the things, those are the three
45:59
things like kind of like pay attention to from like a very
46:02
high level perspective do. You think that's where the
46:06
biggest bottlenecks are right now?
46:08
I think so, yeah. So I think it's like the quality
46:10
cost trade off curve is I think the thing that everyone's
46:12
pushing on. So does this frontier on like
46:15
you know what, what that parallel optimal, you know,
46:17
quality cost frontier is I think Google is at the frontier right
46:20
now. So sound cool.
46:21
Yeah. But I think that's there's so
46:23
much more we want to use it for. And I think, yeah, I think we'll
46:28
just, you know, like for example, you know, when we
46:32
generate integrations today, we're like, OK, we got to go and
46:35
figure out the API docs for the new system and integrate.
46:40
We got to figure out new trust API docs on what we are doing
46:44
and generate code in the middle, OK, test and run that code, self
46:48
debug it a few times and then put it in output, right?
46:51
We now have to be very prudent on like what do we sharpen as
46:55
the API docs from the other system?
46:58
Do you have an everything wholesale?
47:00
Do we need to strip things out? Do we need to format it nicely?
47:02
Do we need to be very careful what we put in there?
47:05
If the context windows are very big and they're effective, we
47:08
don't really have to worry that much.
47:09
We're just saying put it on, put it on.
47:11
We give it, we'll give it more context than it needs, probably
47:14
because it's not a concern to us.
47:16
Same for the our own documentation.
47:19
Now when we produce some kind of integration, we need to test it
47:24
that runs in the loop. You just continuously let it go
47:27
until you're super confident. And now it's like when it's
47:30
testing itself, we want to give it debugging information.
47:33
So it's going to make an API call to an external system,
47:36
observe what's going on and say it is great, that's fine.
47:40
It is bad. We want to give it as much
47:42
debugging traces as possible, right?
47:45
They not just, you know, send it out, send an error.
47:47
They're not just like, you know what's wrong, but actually like
47:50
the the traces in the calls itself.
47:53
You know, maybe the headers are coming back.
47:55
Maybe handle all the details that you don't normally see as a
47:58
user or in even in the terminal. Search whatever stack overflow
48:02
for this. Area exactly so we'll be much
48:06
more open to just grabbing all of that stuff against the
48:09
contacts too now. But it requires the models to be
48:12
able to accept non contacts reason about them in a good way
48:17
and be cheap enough was to do that because it was going to
48:19
cost us a dollar each time. We're not going to do that.
48:20
It was going to cost us like $0.05.
48:23
Great once. And Oh yeah, no brainer.
48:25
As much as you want, you know, like it's us there.
48:27
Yeah, it's funny that you mentioned that because I wrote a
48:31
blog post probably six months ago on how the price per token
48:38
is going down, but price per answer is going up.
48:42
And it was that very fact of OK, per token, cool, it's dropped,
48:48
it's plummeted. But now because it is plummeted,
48:50
that leaves us creating more complex systems and sending more
48:55
calls. And so no longer are we just
48:58
doing a back and forth, one off, hey, give me a poem in the style
49:02
of Bob Dylan about my last earnings call.
49:06
We're creating these complex systems.
49:09
And since we have these complex systems and there's many calls
49:13
being made, our price per answer has gone up.
49:16
Exactly exactly like it's AI mean Javon's paradox.
49:20
Yeah, work. The cheaper you make it, the
49:22
more you want to use it and then price per token goes down.
49:26
But we stuff so much more context that.
49:29
You can go back. To the same, same price again
49:31
and we're like, we want to do more so the price goes up.
49:34
Yeah, right. And I think there's just like,
49:36
you know, just amount of demand for the sentiment.
49:38
So I think it's great in that the connect input context
49:41
windows have gotten very large. Things like COM caching, we use
49:45
a lot to spell. That's super important.
49:47
And I think there's also a trend now that the output windows are
49:50
getting quite big too. That tells you you know that
49:53
trend is only going to keep going.
49:56
Are there any other trends that you think are going to keep
49:58
going? Let's see multi model that's a
50:02
big one yeah, of course you know I mean a year or two ago we were
50:05
like blew away by Dali. Remember like Dali first came
50:08
out we were like Oh my God, where's images?
50:10
And now we're like, Oh yeah, GPT image 1 ChatGPT things like of.
50:14
Course like table stakes that's. Like wow, that's like the the
50:19
bar just gets higher so quickly, multi model video, all that we
50:22
just keep coming to. I think we will see a lot more
50:28
agentic type things out there. Atlantic systems I think were
50:33
just from a quality cost perspective not feasible.
50:38
Definitely two years ago when GBT 4 first came out a year ago
50:41
was really expensive. This year we're seeing in prices
50:45
plummet in a way that they become feasible.
50:48
That too. What about you mentioned how
50:51
you're using Gemini for like the image understanding or
50:56
extraction? Do you ever try with videos?
51:01
Not yet. So we but we don't know it
51:03
works. Yeah, I haven't tried that
51:05
either, but I realized that, oh, that's something cool that
51:09
potentially also could be. I don't know what the use case
51:11
would be. I don't know if it's just like
51:13
cool and then it doesn't have any real applications.
51:15
But that's a fascinating one, too.
51:18
You can imagine like now think this podcast recording put the
51:20
video in. I think they support like 3 to
51:23
4000 frames out-of-the-box, which is like, you know this
51:26
place up the video with the audio too.
51:28
And then you can say great, figure out all the parts that
51:32
were awkward and then just cut them out and then it just
51:35
produces for you. But it gives.
51:37
Us a minute long video and I go no I need an hour shit.
51:41
But it's all the time stamps, right?
51:41
And then you're like, OK, run it through FFM, pack those
51:44
timestamps and then can now for the parts that the audio sounded
51:47
bad, I run it through some other filter to improve the audio.
51:50
So I said imagining like, hey, some of these things that we do,
51:53
do manual editing and everything, the models can start
51:57
to reason about it. And that's fascinating, right?
51:59
So. Actually, one thing that I
52:02
always love to do, the real professional podcasts do this
52:06
because they have a whole staff and team behind it.
52:09
But they will, if you have an interview, they don't give you
52:15
the interview that you see, or I should say this a different way.
52:19
What the end product that you see is, is not the interview how
52:23
they recorded it. They slice and dice it and move
52:26
this part to the front and move that part to the back.
52:29
And oh, the story fits better like this.
52:32
And so I never do that because I never have time to really think
52:36
about it and get that creative and say, oh, that question
52:38
actually should have been here instead of in the end of it.
52:42
But yes, there that's a potential use case because you
52:46
could. Ask the model to give me like
52:48
time points to like slice the videos can.
52:50
Now I've ordered slices, generate a description where
52:53
every slice can now give me a reordering of all the slices and
52:57
then. Would this narrative go better
52:59
here or I won't? I think this question or this
53:02
answer, we, I jumped around, I, I was talking about
53:05
integrations, then we talked about something else and then we
53:07
went back to integrations or whatever it may be.
53:10
So. You tried with this video?
53:12
Yeah, I I might actually. Yeah, no, that's, that's
53:15
fascinating. Yeah.
53:16
But but yeah, I think those are the some things in there.
53:18
I think also I think there is this bigger trend thing that I
53:24
think people want to do solve as well, which is, you know, we
53:26
hear a lot about computer use. I think it's very early,
53:31
honestly. I used operators like I was a
53:33
bit, you know, and actually I was like super high, like, oh,
53:35
this is going to be so cool. And like, I ran it.
53:37
I was like, yeah, through my thumbs kind of kind of getting
53:40
there. Oh, needs my help now.
53:41
Ah, kind of get it there. Then after like a day, I was
53:43
like, stop using it. But I do see a world in which,
53:48
you know, the systems, you know, they, they can be trained to
53:51
understand our world as we see it.
53:53
You know how we use computers, how we navigate around the
53:56
physical world and everything. So sort of, I think we'll start
53:59
to see a lot of developments there too.
54:04
Very early days, but I think definitely some no promise and
54:08
how that works. And oddly enough, I think it
54:11
just still goes back to the fundamentals, which is are the
54:14
context windows big enough to put in all the frames or history
54:19
of interactions to reasonably predict what to do next, right.
54:24
So that's kind of interesting. I think the other interesting
54:26
part on that is I want to be using my computer.
54:29
So would it be some VM or background process that's
54:32
happening in a different sandbox or a different almost like it
54:37
would be some different VM there and then when it gets stuck and
54:43
it needs my help, it comes to the forefront of my computer?
54:46
Yeah, totally. I mean like I can.
54:48
I mean, there's so many ways where where in a there's this
54:52
article I read yesterday of by by Pete Coleman.
54:54
I think on AI Hostess carriages. I've seen that one.
54:59
Oh, that's great. I think we are in that world
55:02
right now where this we all, most of the AI applications are
55:06
like horses, carriages, you know, like the guy in front of
55:08
the, you know, trying like on the carriage.
55:10
And the AI is just like the machine, the engine is machine
55:12
in the back. Like why is it there?
55:14
It's like a bit retrofitted in ways that we don't expect it.
55:17
And you start to reimagine, you know, 5-10 years from now where
55:24
the technology is cheap enough to be deployed everywhere.
55:29
Voice recognition is amazing, multimodal is amazing.
55:34
And then you go like, you know, what should the UI be for a
55:38
computer now? I think the GUI that we have,
55:42
you know, that came from like, you know, I remember, like
55:44
fondly remember Windows 3.11 days to 95 and everything to
55:47
today with my Max and L. They've been designed for humans
55:52
to operate, you know, like their windows, like start buttons,
55:55
like, you know, layers on navigation and all that very,
55:58
you know, humanistic way to help us understand how these things
56:01
work, you know, and but AI doesn't need many of that,
56:04
right? The AI needs operates a more
56:06
fundamental level. You know, you can write
56:08
software, you can operate that. So I think we're still in this
56:11
Hostess carriage world where we haven't really figured out what
56:14
the real interaction model should be yet.
56:17
For example, if we wanted to do this task of getting the machine
56:20
to cut up your video for you, what if you say you give it the
56:24
task, you give it the MP4 file, say go do it.
56:27
But you don't want, you don't want the AI to work without
56:29
telling you what it's doing, showing, you know, clicking
56:33
around the UI and trying to get it to do work.
56:34
It's kind of nice. That makes what you're doing
56:36
very understandable. But those are not very
56:38
inefficient. Yeah.
56:40
What it really should do is like run some FM pack commands, do
56:43
some slicing, run another model, blah blah blah, do all this
56:46
stuff. But what if on the fly it
56:48
generated its own UI? On the fly, because you know the
56:53
models can produce websites, you can write its own software.
56:56
What if on the fly produce a representation of what it's
57:00
doing to you that is not your video editing software?
57:05
That is not just the raw commands it's running, but
57:08
something intermediate that represents what the AI is
57:11
working on so that you can understand what's happening.
57:14
How would that be interesting now?
57:15
So not, not even the VM, right? It's like, it's almost saying
57:19
create your own one off experience to explain the task
57:24
you're working on in a visual way so that a human user can
57:27
understand what's going on. So it's like, OK, I'm going to
57:29
write a little mini app, Electron app to show you how I'm
57:37
cutting up the videos. OK, the videos are cut up this
57:39
way now. Oh, great, I'm not going to edit
57:41
them. I want to modify my electronic
57:43
on the fly to show you how I'm modifying the videos.
57:45
So seeing that go, it's like, so you're not watching it use your
57:49
current applications, you're watching it do something really
57:51
rich. But I'm showing to you how it's
57:54
doing it in a very intuitive fashion.
57:57
You know what the problem I have with that is, though, is the,
58:01
because I've thought about it and I've had all kinds of
58:04
conversations with people and the, there's certain
58:10
applications that we use that are very specific and very
58:16
niche, like video editing. The pros, the pro video editors,
58:21
they know what words describe certain actions inside of the
58:27
camera and they know all right, when I'm editing, if I want
58:31
different colors, if I want different Luts, all of that.
58:35
Even LUT, for someone who doesn't really video, do a lot
58:39
of video editing, they're just like, I just want it to look
58:42
better. And maybe the AI can give you
58:46
something. But if you want just one little
58:50
thing changed and you don't know the word to describe it, to
58:53
prompt it, then it's really hard.
58:55
But if you know that, all right, I can go in and I can either
58:59
Normally what I do as a very, very amateur video video editor,
59:05
I'll just go and type in YouTube.
59:07
I wanted to do this and I can't see.
59:09
And then I'll search around in YouTube a bit and try and figure
59:12
out what I'm looking for potentially.
59:14
There's a world where, yeah, I don't search YouTube.
59:16
I just and prompting it back and forth saying no, I want
59:20
something more like this. And I but usually it'll be me
59:23
going through different YouTube tutorials to see this is how you
59:28
do it. And there's these specific tasks
59:33
that you want something, one thing changed.
59:37
And I'm sure you've had it with cursor or an any experience
59:41
where you just want one thing changed and that one thing it's
59:45
changed, but the rest is blown up.
59:47
Totally. No, it's it's funny, like you
59:49
remind me of the conversation I had at my designer on my team
59:53
recently. So if we use image and models to
59:55
create like, you know, illustrations for some of the
59:59
automations people make on Dutra, and then he writes up
1:00:02
this really long Proms that work really well.
1:00:05
And clearly, you know, he he understands how to express a
1:00:09
certain kind of design language. Way better than I do because
1:00:13
when I try, it's like it looks like garbage.
1:00:16
He laughs at me. You know what's going to happen.
1:00:18
And then now that's cool, but it's but it's, but you know, to
1:00:21
that point, right? It's like there are few tokens
1:00:24
that we use and what words we choose that have a huge impact.
1:00:27
You know, like if you would say that particular design pattern
1:00:30
to cursor, if like, ah, I know what you want, do that design
1:00:32
pattern for you. And so this is interesting,
1:00:35
interesting thing in there that I do wonder if, you know, I
1:00:41
mean, at the two thoughts in my mind here, I guess the first
1:00:42
thought was what is the role of people?
1:00:47
You know, what do we do with AI doing a lot of work for us?
1:00:50
It turns out that I think 1 is that having good taste.
1:00:54
But then even if you have good taste, how do you describe your
1:00:57
taste? It turns out that describing
1:00:59
your taste is having those tokens.
1:01:01
You know, like everyone, everyone now knows the Studio
1:01:04
Gibby token. Easy one, right?
1:01:07
But so many other ways to express style or form or design
1:01:11
pattern that I think is really hard.
1:01:13
But #2 I think is I don't wonder as well as like, can can the
1:01:20
machines help us understand the styles we like, Right.
1:01:24
So for example, you know, you look through all the YouTube
1:01:26
videos and you go like, I like that kind of style.
1:01:29
I mean, why not just fit in all the videos and say I like that
1:01:31
style, Don't like the style, just my mood board.
1:01:33
Tell me what how she described this.
1:01:37
I think it will actually do a pretty decent job at describing
1:01:39
it. Now the the hard part is things
1:01:41
still figure out what styles you like.
1:01:42
Yeah, right. And that fine tuning too, where
1:01:46
you have those, that last mile of oh, I need this.
1:01:49
But I do like that. I think I've seen a tool like
1:01:52
that where it's almost like a mirror board, but instead of it
1:01:56
being a a random mirror board, the stuff that you put on the
1:01:59
mirror board are ways to give the prompt more.
1:02:05
Yeah. And it's more enriched prompt.
1:02:07
It's not just saying the words. But you know, I mean back to the
1:02:12
horses carriages thing, are we in a world where the problem
1:02:17
that we are seeing is that most of the software today has been
1:02:21
pre designed for us to use? So whatever knobs that exist in
1:02:24
our software, some developers sit down like say that I like
1:02:27
those knobs, I'm going to give it to you.
1:02:30
But one of those knobs that were given to us was generated on the
1:02:34
fly. What if your video editing
1:02:37
software looked at your video and say, oh, this is the
1:02:39
podcast. Oh, here the fine knobs you
1:02:41
want. Oh, this is a, you know, very a
1:02:43
video recording of a commercial out on the street.
1:02:46
These are the fine knobs you want.
1:02:47
Oh, this is an interview with someone.
1:02:48
This is the fine knobs you want. And one of those knobs that it
1:02:51
gives you are dynamic based on a small knowledge.
1:02:55
So what if you don't have to know the words, but the words
1:02:58
come to us, right? And that's not like that one.
1:03:02
I'm just ideating because I'm like, that's a different world
1:03:05
where software software isn't just, you know,
1:03:09
one-size-fits-all design and shipped software adapts to the
1:03:13
context, right. I don't think there's seen that
1:03:16
kind of software yet. Just we just have nots And so it
1:03:19
does. I don't know, I got a bit
1:03:20
thinking, wondering what that would look like because I'm I
1:03:22
can imagine, you know, where we say the word personal computer.
1:03:27
I think the word personal actually came from the genesis
1:03:30
of like computing that that's what we want to do.
1:03:34
But this days the personal computer runs the same software
1:03:37
for everyone. It's not very personal except
1:03:39
for hi, not typing my my data in there, right.
1:03:42
But if it's truly personal computing, then, you know, is it
1:03:47
AI that looks at what we're doing and then make suggestions
1:03:49
for us? You know, it's like you wanted
1:03:51
that particular style. You don't know what is it
1:03:53
called? You don't even need to know
1:03:56
about it because the AI has looked at 1000 podcasts and
1:03:59
says, yeah, that's a style you should consider.
1:04:01
And then you're like, click on it and say, OK.
1:04:03
And then you're like, that's not the one I want.
1:04:07
Inspire me. Show me 5 more.
1:04:09
And then you start to learn, right?
1:04:11
Because the fact of the matter, I think, is that the models have
1:04:16
seen more data than any of us as humanly possible can, right?
1:04:22
They've seen every single piece of data out there in the web, in
1:04:27
the world. They've ingested it all the
1:04:30
tokens we're talking about that are taste tokens are all in the
1:04:33
model. So what better system to tell
1:04:37
us, educate us, maybe help us understand some of these things
1:04:41
than the models themselves? So, yeah, more food for thought,
1:04:44
I guess. It's like, I don't really know
1:04:46
if that's going to happen, but yeah, exciting worlds.
1:04:48
I love it dude.