Latest / Agentic Conversations (formally mlops.community) / We're All Finetuning Incorrectly // Tanmay Chopra // #304

Transcript

We're All Finetuning Incorrectly // Tanmay Chopra // #304 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Tanmay Chopra. I'm the CEO of emissary.
0:03
We're an AI infrastructure platform for model optimization.
0:08
And I take my coffee. I actually do a double shot
0:11
latte in the morning and then in the afternoon everyday.
0:17
We're back with another ML OPS Community podcast.
0:19
I'm your host Dimitrios, and today we got into how
0:23
traditional ML systems can inform and help you level up
0:29
your game on the new Gen. AI systems. 10 May is an
0:35
original thinker in this space, and I really enjoyed the last
0:38
part of our conversation when 10 May talks about the Pythonic
0:41
universe and how you really got to have it when you are dealing
0:45
in AI. Let's jump into this
0:48
conversation. I just posted something and it
0:58
said if LMS are so smart, why are AI products so dumb?
1:02
And I cannot take any credit for that because I completely stole
1:06
it from one of the agent hours talks that we had and it was
1:10
from Shrivu Shankar. But I think that resonates with
1:17
you, right? What were you saying 1?
1:19
100%, I think that's kind of the big mismatch we're seeing right
1:23
now, right? We see that these systems have
1:25
so much potential, but realistically in production,
1:29
we're actually not seeing them utilize any fraction of that
1:32
potential. You're seeing this now from, you
1:35
know, the leaders of large tech companies saying this is
1:39
promising, but actually maybe not all of it works, which is a
1:43
kind way of saying, yeah, most of it doesn't work right now.
1:46
And so this sentiment of, you know, AI systems could be
1:51
amazing and AI systems are amazing.
1:54
We're kind of in the gulf between that right now.
1:57
Yeah, it's so funny that you mention that because just
2:02
yesterday I was talking to a friend who was saying, you know,
2:06
apps won't exist in a few years, it's just going to be all agents
2:11
doing stuff. And I'm like, dude, give me a
2:14
break. I've heard this argument a few
2:16
times and I really am having a hard time getting on board with
2:22
it because let's just take my favorite video editing software
2:27
like Davinci Resolve. I know where all the buttons are
2:30
on that that I need to press to get what I need done.
2:33
I know the hot keys. I know how I really like the
2:36
visuals in the color grading, all of that stuff.
2:40
If now I need to just tell a large language model or language
2:46
model or diffusion model, whatever model, it is a
2:49
foundational model that I want you to edit this podcast and
2:54
color grade it, then at the current rate, I don't think that
3:02
is possible. And then when I have to debug
3:06
it, it's going to take me more time than if I just use Davinci
3:10
from the beginning. So maybe helping me on a few
3:13
steps is really where I see the value.
3:17
But actually, like what this guy was saying yesterday is, yeah,
3:21
we're just going to spin up apps.
3:23
Like the agents will create apps anytime that you need something
3:27
done. And I'm like, dude, that is that
3:29
is very hard to like grapple with.
3:33
I do not think. And maybe I'm pessimistic, but I
3:37
don't think that's a reality that we're going to see.
3:40
Yeah, I think there's this really interesting distinction
3:43
there, right between like what's possible and what's probable.
3:46
I think it's very possible that this happens.
3:49
I think the probability of this happening in the next 5 years is
3:53
probably not that high. And so that's how we think about
3:56
it. If you think about ChatGPT now,
3:58
we're on year three, right? Like that's kind of the in the
4:02
post ChatGPT era we're on, we're on year 3.
4:05
And if you were to go back 2 1/2 years and say, hey, in 2025,
4:11
we're not actually going to have a lot of AI in production.
4:13
We're going to have a lot of funded companies, but we're not
4:15
going to have a lot of AI in production.
4:17
Most people would be surprised, right?
4:19
Because when this thing came out, everyone was like, hey, do
4:22
your timeline. We're going to automate out
4:24
particular jobs. And what we've really seen is
4:29
there's two personas, right? So you have sort of AI as your
4:32
supervisor and then you have AI as your assistant, right?
4:36
So the scenario you're describing is AI as your
4:39
assistant. You're a really good video
4:42
editor and you're now being asked to use this AI system for
4:47
better or for worse. And the other scenario is you're
4:51
a really bad video editor and you're being asked to use AI to
4:55
just get started, right? You have no idea where to go.
4:57
You have no hotkeys. You're sort of the zero to 1
5:00
user. And I think code Gen. is a
5:03
really good example of this. So this is played out in front
5:06
of me in like real life most of the time, all of the, you know,
5:12
fascination over code Gen. is coming from CT OS or CE OS,
5:17
right? It's actually coming from people
5:18
that don't write code as frequently as sort of a
5:22
frontline engineer. And I love code Gen. for
5:25
context, I'm a huge shell, huge fan.
5:28
But now if you go talk to an infrastructure engineer that's
5:31
dealing with like a genuinely hard infra problem, they have
5:34
the experience that you're high, right?
5:36
So given that they're an expert, the system actually slows them
5:40
down, but it only slows them down in the context of
5:44
infrastructure. If you ask an infrastructure
5:46
engineer to do front end engineering, it's actually
5:48
amazing. They all love them, right?
5:51
So for the bucket of tasks where you're not the expert, but you
5:55
have some verification capability, code Gen. is really
5:59
good and it's taken up so much of that space.
6:03
But you really got to think about like, OK, what is that
6:06
user journey when you're not an expert?
6:08
So when AI is your coach, and then that user journey where you
6:12
are an expert. So AI is your assistant.
6:15
And in the latter, repeatability, determinism,
6:19
accuracy, latency, these things matter a lot.
6:22
An expert is not going to waste their time trying to prompt
6:25
ChatGPT 4 times. They'll just go to the job.
6:27
But someone who knows nothing, they'll, they'll take that alpha
6:31
and actually they're learning in the process.
6:34
So as the system fails, they're also learning.
6:38
And so I think that's how I've been looking at these tasks more
6:41
and more. So where I think in my mind as a
6:44
user experience type of perspective that this whole
6:49
thing breaks down, especially if we are trying to do tasks that
6:56
were not necessarily experts in and we don't know all the lingo
7:01
and we don't know the best ways of doing them, is trying to
7:06
explain what we want done. That is very difficult.
7:12
Like I was just playing around with this tool this morning that
7:16
the whole thing is you create a prompt and it gives you a
7:19
pipeline. It will create the workflow for
7:22
you and then you can go in and you can tweak that.
7:25
Yeah. It didn't really work.
7:27
And even me explaining it and then going back and trying to
7:33
prompt it better and I knew what I wanted, I knew what the
7:37
workflow was going to look like, but inevitably I had to go in
7:42
there and create new nodes and create new calls.
7:46
And so it's this low code, no code experience that I had with
7:51
the GUI. And at the end of the day, it's
7:55
like, yeah, cool, the prompt kind of got me there, but I'm
7:58
not sure if it really saved me that much time.
8:02
Yeah, I think this is actually a big problem with generalized
8:05
systems, right? They're just not aligned enough.
8:10
And so if you look at a model that's sort of trained on
8:13
everything in the world, it's a great model for everything in
8:17
the world, but it's not the best model for trying to do what
8:20
you're trying to do. And so in an ideal world, right,
8:24
whatever system was generating that pipeline out of your prompt
8:28
would have seen hundreds of thousands of prompts and the
8:31
ideal pipeline that was generated out of them.
8:34
Even then, it's not going to be perfect, but your edit distance
8:37
from first output to what you finally ran is going to be a lot
8:41
shorter. So there's, there's a couple of
8:44
things there. The 1st is inherently AI was or
8:49
ML old school ML was about trying to digitize systems and
8:53
processes you could not explain. And so we are doing a little bit
8:57
of morphing here, right? Of like what this was meant for
9:02
machine learning was always meant for like you give me
9:05
inputs and outputs and I will converge to the right function.
9:08
So don't tell me your process because there are processes
9:11
better than yours that I will discover in this larger search
9:15
space. And so that was kind of why we
9:19
used ML, right? If you think about all the use
9:22
cases before, like I cannot write software to do
9:25
recommendations, there is not enough lines of code that I can
9:29
generate that would create a recommendation pipeline for
9:31
every user in the world. Great.
9:33
That's a machine learning use case.
9:35
If you physically cannot describe enough of the process,
9:39
it's an amazing machine learning use case.
9:42
So we are morphing that by saying, hey, actually now you
9:44
have to explain your process in the prompt.
9:48
Don't give me inputs and outputs, Give me your actual
9:50
like chain of thought process, right?
9:53
So whenever you start writing sort of the chain of thoughts
9:55
prompts, so that's one thing. And this is sort of a more
9:58
general philosophical thing. There's two others.
10:01
The first is, is the AI system itself good, right?
10:06
And then the second is, is it good for you?
10:09
And those are two very, not very different alignment vectors, but
10:13
they are distinct alignment vectors.
10:16
A lot of AI systems are still stuck at the first one.
10:19
So they're not even thinking about hyper personalization for
10:22
the user. They're thinking, you are like,
10:23
how do we be good? And I think this is coming more
10:26
and more to the forefront. This was sort of very popular
10:29
towards, you know, 2021 in ML, but now very popular in AI.
10:34
Is Edal's right and not LLM as a judge?
10:39
No knock on NLM is a judge. I've just never seen it work.
10:42
But The thing is, you need to know what good looks like for
10:48
you. The first thing we ask every
10:50
customer to do on our platform is to stop and think, right?
10:54
Like tell me what good looks like.
10:57
Does it mean I have the right files in place?
10:59
Does it mean I have the right text in place?
11:01
Like in your universe, what does success look like, Right?
11:06
And ideally, this evaluation is not gut paste, because if it's
11:10
gut paste, you're adding an other layer of uncertainty.
11:15
So we now have some folks that are just not evaling their
11:18
systems, and then we have some folks that are evaling their
11:20
systems on vibes. And the bucket of folks that are
11:25
sort of evaling their systems deterministically is actually
11:28
very, very small. But if you were doing that,
11:32
everything would get better overtime because then you can
11:35
start using math, you can use back propagation to start
11:38
building systems that are better over time.
11:41
So that's kind of where we think a lot about this.
11:45
But why do you feel like people aren't using evals?
11:49
Is it because they don't know what good looks like?
11:51
Is it because it is a new product and they want to get it
11:55
out there? And then once they get users
11:58
giving them that feedback, then they can create that eval loop?
12:02
What is it that? Is missing.
12:05
So I think it's two things. One, this is the obvious one,
12:07
eval is hard, right? So I don't think people are like
12:10
actively choosing not to do eval.
12:13
I think this thought process of what does good look like in some
12:17
cases, it's hard to describe. In some cases, it overlaps
12:21
between technology and business. So this is also why machine
12:24
learning used to be hard, right? You've got people to try to do
12:26
calc and then you need to kind of map that to a business metric
12:30
because you don't actually have any signal till the business
12:33
metric comes out. So that's number that's sort of
12:36
that number one piece, right, that this is actually a really
12:38
hard problem. I think #2 piece is a lot of
12:43
people are thinking about AI systems a lot more statically
12:46
than we thought about ML systems, or at least did for the
12:49
last couple of years. They would ship something that
12:52
looked decent and then kind of say, OK, this is it that works
12:56
in software. You can front load a lot of
12:58
thinking in design and then you ship it.
13:00
Because over the years we've gotten really good at creating
13:02
software abstractions that result in good output software
13:06
the first time or the second time or maybe the third time.
13:09
AI systems are inherently perishable, right?
13:12
So you kind of have to think about from moment one, how are
13:16
you going to keep retraining and improving this?
13:19
But this is very new, just as a paradigm, it's very new.
13:23
And so if you're a software engineer building AI, that's the
13:26
thing I always encourage folks to think about is how is this
13:30
different from your software development life cycle?
13:33
It's not one and done. It is inherently improving
13:37
overtime that like. That's a feature, not a bug.
13:39
I remember when we had, there was a great article that came
13:43
out from Google probably in 2021 talking about continuous
13:47
training, and it showed the whole maturity levels that you
13:53
could get to that would automate that retraining pipeline.
13:59
And it's a little bit going back to that.
14:02
Even though now we're using large language models instead of
14:05
these traditional ML models, you still want to look at it as as
14:10
soon as you get the model out there, in a way, you need to be
14:16
ready or you need to have a plan for how you're going to keep it,
14:19
rating on making it better and what you can do to fine tune it.
14:24
And I think a lot of people are having success with fine tuning
14:27
prompts. I know that you don't
14:30
necessarily like that idea. I don't think I dislike that
14:35
idea. I think I've just seen that sort
14:38
of hit a ceiling with the early movers from the last two years.
14:42
And so in some ways I tried to preempt folks struggling when
14:47
they hit that ceiling. But actually the, the approach
14:51
we take is we say, don't fine tune until you have to, right?
14:54
So do the best you can in the prompting world.
14:58
And if it's good enough, like fantastic, let's help you
15:00
maintain those systems through the retraining piece, right?
15:03
So part of what our platform does is it actually streamlines
15:08
the retraining piece because retraining used to be the most
15:10
painful part of machine learning.
15:12
If you have your own model, you're now like signing up to
15:16
retrain this model month over month.
15:18
And so this is actually why we discourage people from fine
15:21
tuning their own models because you need to know that it's a
15:25
lifelong investment, right? It's like having a child where
15:28
you kind of have to take care of that child month over month,
15:32
year over year. You don't get to just like give
15:34
birth to your child and then be like, I'm out.
15:37
And so I actually don't, you know, knock on prompting or
15:43
prompt fine tuning. I think it's more a function of
15:47
you should be intentional about when you're doing that versus
15:50
when you're functioning your own model.
15:53
So the big challenge with any optimizations at the prompt
15:56
level is that you can't change the objective function.
15:59
So the model still only cares about the next word.
16:04
Most enterprise tasks care about things other than the next word,
16:07
right? You might care about some form
16:09
of classification. You might care about, you know,
16:12
you generate an investment memo and you're like, OK, did this
16:15
memo map to reality? You don't actually care about
16:18
the words of the memo, right? Or like words of memo one versus
16:21
ideal memo. What you care about is like what
16:24
was the grounding in reality of memo one versus memo 2?
16:29
So improving your model or your system AI system based on your
16:34
objective function is something you do not have access to till
16:37
you start owning your own models.
16:40
That's the only reason that we encourage folks to think about
16:43
this is it can improve for you, right?
16:46
It doesn't just improve for words, it improves specific to
16:50
your AI system objective. So the idea here is that you're
16:56
doing as much work as you can with foundational models, and
16:58
then eventually you feel like going back to what you said
17:04
before, there's these two vectors that we need to look at.
17:07
Is AI good? Is AI good for you or is the AI
17:11
product, I guess that we wrap it up and is it good and is it good
17:15
for your use case? You can get pretty far, I
17:19
imagine with the foundation of models and indeed a lot of
17:22
companies that's as far as they get their AI center excellence
17:29
says we've got to win under our belt.
17:32
Let's go play around with other stuff and wait till the new
17:35
model comes out. But what you just described is
17:39
very much like going back into the old world in a way old
17:44
traditional ML world. So how do you look at that?
17:48
Is it not just a ton of burden? And when do you want to use
17:54
those types of use cases? Because I think when these AI
17:59
Center of Excellence's are talking about their use cases
18:02
that they're going to tackle, if you bring up, Oh yeah, we now
18:07
have to do AI like we did ML, it gives people cold feet.
18:14
Yeah. So that you bring up a bunch of
18:17
interesting things. So in terms of the burden side,
18:19
that's why we're building MSLE, right.
18:22
That's that's the whole purpose of our platform is to say, hey,
18:25
it was actually really hard to build models before and it
18:27
shouldn't have been. So let's try and make it easier,
18:30
right? Like how much easier can we make
18:32
it? And our bar to success is really
18:34
can we make it easier than prompting?
18:36
That's that is the level that we're pushing to go to.
18:39
We're pushing to go to say, if you want to do a GRPO based
18:43
optimization, right. So the the system that deep CQ's
18:46
to build that model, we want to make it as easy as it is for you
18:50
to prompt. In fact, in the longer run, we
18:53
want to make it easier because you don't have to be guessing
18:55
around removing a word here and there and being like, oh, did my
18:58
prompt regress? You can just be like, hey, math,
19:01
please take care of this, optimize, do this function.
19:04
So burden is is an infrastructure problem, right?
19:08
That's something that, you know, even if we don't solve, somebody
19:10
should and will solve if the industry becomes.
19:14
There's Sorry to cut you off, but I want to stress this point.
19:19
You said, hey math, go and figure this out.
19:22
So it's not that it's someone that is pulling something out of
19:28
their ass with their prompt and trying to explore latent space.
19:32
It is math equations which that resonated with me a lot the
19:36
first time you told me it because it was like that.
19:41
Seems like a much better way to systematically go about
19:46
improving your product. We have a deterministic way of
19:53
to reduce the loss, right? Like that's, that's all of what
19:57
machine learning was. We were like, hey, if you can
20:00
define what good looks like into some equation, we can use that
20:05
equation to start approximating outcomes based on your input.
20:10
This is what all of machine learning was.
20:12
Somewhere along the line, I think we kind of lost the plot
20:16
and said. Let's not use that.
20:19
Let's generalize that right and to to the next word.
20:22
And let's try and guess the series of words that result in
20:26
the right word outputted instead of actually trying to use the
20:30
math that we've been relying on for this long.
20:33
But. We made so many influencers so
20:36
much money with their prompting guides. 100% I, I still think
20:41
it's useful. So here's the thing.
20:43
I think the real LLM unlock is you need a lot less data to
20:47
bootstrap your first version, right?
20:50
So if you wanted to start doing machine learning three years
20:55
ago, your first call to action was I need to go to, you know,
20:59
scale AI or Surge AI or somebody like that and get 100,000
21:03
samples labeled. Now you can say, hey, let me
21:07
start guessing some words and that this is where those guides
21:09
are really useful. And let's get us to V0.
21:14
Let this V0, if it's good enough for customers to use, becomes
21:19
our pipeline to get better every day for the rest of our lives.
21:22
What only burden is the infrastructure burden.
21:25
And maybe you can, you know, check out emissary for the
21:27
infrastructure burden or you can check out other sources, but
21:30
there are people helping you with this infra burden.
21:34
And now you have this self improving system.
21:37
And I know there's a lot of founders or a lot of AI people
21:41
that say, hey, why don't you just wait for the next GPT on
21:44
what happens when there's the next GPT?
21:47
And this is my standard response there, right?
21:50
They don't care about every task in the world.
21:52
They cannot. They can care about every task
21:55
in aggregate, but they do not care about your specific
21:59
workflow. And so GPT 5 could be much
22:02
better, but will it be much better for you is a very
22:06
interesting question to ask. That's number one.
22:08
Number two is if they do start caring about your task, that's a
22:12
whole other like can of worms, right?
22:15
So you see with Anthropic code Gen. they started getting enough
22:19
about optimizing for code. And so now you're competing with
22:24
them and you're not improving your systems based on your users
22:27
interactions, but they are. And so you're now in a universe
22:32
where they came to your game six months or a year later, but
22:36
because they're better at improving day over day, they're
22:38
going to be better in weeks. That's the.
22:41
So actually you want to be as orthogonal as possible in terms
22:45
of better for you versus better from the foundational models.
22:49
If you want to keep space between what you do and a
22:53
foundational model sort of trampling on you, in some sense,
22:56
everyone has to go up against open AI or Anthropic at some
23:00
level, right? Like that is if you're building
23:03
an application layer company, you'd at some point they will
23:07
come for your application. You can't simultaneously believe
23:10
that code Gen. will make all software mode 0 and then go out
23:15
to public markets and say our mode is software mode, right?
23:18
Like those two things cannot exist badly.
23:21
So you they say code Gen. is going to be really bad and so
23:23
there will still be software mode.
23:25
Or you say oh shit we need something more than just a
23:29
software mode. Not the start, but over time.
23:32
It goes back to your last point or one of the points that you
23:36
made before with the workflow solution that wasn't working up
23:39
to standards. And that is because it is trying
23:43
to be generalized workflows. It's not a here's a workflow
23:49
that scrapes keywords from SEM Rush and then goes out and looks
23:55
on Reddit and looks at all the posts that are for those
23:59
keywords and then looks at all the blog posts that have been
24:02
written about those keywords. And then creates a intro or an
24:07
outline for a blog post that you can write.
24:11
And then it creates the intro and the body text and it and
24:14
then it has that run through some kind of SEO optimizer.
24:20
I'm just explaining like, yeah, the very deep, deep workflow
24:25
that I've seen my friends in marketing create for generating
24:29
a shit ton of SEO optimized blog posts from AI generated content,
24:37
right. And so whether or not that is
24:40
actually useful for the Internet is a whole nother conversation.
24:45
But it's out there, it's happening, it is super advanced.
24:50
And that happens because a marketer or a few marketers sat
24:55
down and they said, you know, it would be great if we had all
24:58
these different steps. And then if I go and I say I
25:02
want a workflow that will do all these different steps, even if I
25:07
know exactly which step is which and I prompt it or I try to
25:11
prompt it, the model isn't going to understand or the magical
25:16
workflow that is created from that isn't going to understand
25:18
because it's, again, going back to what you're saying.
25:21
They didn't have thousands of these prompts.
25:26
And so it couldn't know what I was trying to look for because
25:30
it is such a broad type of a product.
25:33
It's not that verticalized product.
25:37
Yeah. And what's really interesting
25:39
there is you added another layer of uncertainty, right.
25:43
So this is one of the I, I always like everyone laughs when
25:47
I say this, but you should use as less machine learning or AI
25:51
as you feasibly can to get through your system, right?
25:54
And, and obviously this is counterproductive to, to my
25:58
business. But the idea here is you had two
26:00
layers of uncertainty you prompted and that resulted in
26:05
the creation of a workflow. So you have what we think of as
26:08
like orchestration uncertainty, right?
26:10
And then that workflow needs to get executed.
26:13
So these are different pieces that need to get executed.
26:17
And so you have some level of execution uncertainty within
26:21
each piece. That execution is also volatile.
26:25
So you also have component uncertainty right now.
26:28
If you look at the folks that are in marketing that are doing
26:31
this, if they're marketers, the workflow is coming from them.
26:35
So they have no orchestration uncertainty.
26:37
They will say, do this first, then do this, then do this.
26:41
They have an SOP and they'll execute on that SOP.
26:45
They still have execution uncertainty.
26:48
But even within that, because they're experts, right, this
26:52
expertise is sitting in their head, they're actually a lot
26:54
better at aligning these systems.
26:56
So the whole game now is, is just a question of like who is
27:01
the best at aligning the models. We've kind of flatlined in terms
27:07
of how good we're going to get by throwing more computer at the
27:10
problem to a large extent, right?
27:13
And this happens every so often. And so I'm sure 10 years from
27:17
now or five years from now, we'll have a much larger model
27:21
that'll do much better than everybody else.
27:24
What's really interesting is a lot of value is created at the
27:29
next layer of foundation model, right?
27:31
So when CN NS came out, it was huge.
27:33
A lot of value was created, but actually most of the value is
27:36
captured between the times that one large model comes out and
27:41
the next larger model that blows the last large model out of the
27:44
water comes out. This is what we're very excited
27:47
about, right? Is the actual value capture in
27:52
in this journey has always been the applied AI people.
27:57
It's always the people who are like, these are the new tools
28:00
we've gotten this new large model, this new way of training.
28:04
How are we going to orchestrate these things to actually
28:08
generate value for someone that will be willing to pay for these
28:11
systems, ideally more than what it costs you to run them?
28:16
That's kind of what's really exciting about this period in
28:19
time. But yeah, there there is.
28:21
The more uncertainty you add to the system, the less likely it
28:24
is that the system will do well. OK, so getting back to the
28:27
question I asked before we went on this epic tangent around
28:33
these AI center of excellence is thinking, Oh no, I don't want to
28:37
go back in time to traditional machine learning methods.
28:40
Yeah. I think I, I would focus less on
28:46
method and more on outcomes. The thing we've been encouraging
28:50
centers of excellence to focus on is the bigger bats.
28:54
And what I mean by that is it's equally hard to build an AI
28:58
system for customer service as it is for something that will
29:01
completely change the unit economics of your model of your
29:04
business more often than not. And so one of the things we
29:09
focus really deeply on when we start working with folks is
29:13
what's the biggest problem you can tackle, right?
29:15
If you were going to commit 10% of R&D spend to a specific
29:18
problem or 20% of R&D spend to a specific problem, would it be
29:22
worth it? If not, just buy the solution,
29:27
right? Like there are companies out
29:28
there that are doing amazing jobs at doing the things that
29:33
are not worth 10 to 20%. And because this, these AI
29:37
systems are perishable goods, you're now putting somebody on
29:40
the hook for maintenance every time you build a new system.
29:44
If you can outsource that, you probably should.
29:48
You want to build the stuff where your feedback loop is
29:51
optimal. That's the shortest way I can
29:53
put it. If you have a differentiated
29:55
feedback loop, right? If you're an insurance company
29:58
and you write insurance policies, like figure out how
30:01
you can deal with the insurance broker shortage by building a
30:04
copilot for insurance workers, that's the alpha for you, right?
30:08
Not trying to figure out like, hey, can I build a marketing AI
30:11
tool? Because I think it's going to
30:13
cost you 100 bucks a month to buy.
30:14
It's probably going to cost you like $100,000 to get right,
30:18
right. Like the, the economics don't
30:20
work there. I, I think invariably going back
30:25
to your original question, I know I've been going a couple of
30:27
circles, but AI systems are going to become a component in
30:31
the ML system pipeline. So you're not going to forget
30:36
classifiers, but your classifier might sit on top of an LLM.
30:43
That's how we think about this universe standing out.
30:46
What does that mean? Sit on top of an LLM.
30:49
Yeah, how does that so say you're right, building a chat
30:51
bot, right? Common use case right now there
30:54
is like so much work being done on trying to figure out how to
30:58
get the LLM to only answer the questions that it should.
31:01
The easiest way to do that is to build a classifier that says
31:05
should I answer this question or should I not stacks on top of an
31:08
LLM. So basically upstream from an
31:11
LLM question comes in, should I answer?
31:13
Should I not? No, end the chat or you know,
31:16
politely in the chat, yes, let your LM answer instead of trying
31:21
to get this LLM to be this one unit that's perfect, just use
31:26
old school ML to solve the problems that old school ML
31:29
could solve. So I think there is that kind of
31:32
fear. That's actually one of the
31:35
things we've spent a lot of time doing is converting LLMS into
31:39
classifiers, regressors and other like old school models.
31:42
So you can still use their inherent knowledge, right?
31:45
That's what's exciting about them is the knowledge that sits
31:47
between the first layer and the last layer.
31:49
You can still use it, but you can use it for your tasks.
31:52
So now with 500 samples, you can build a classic.
31:57
We're really excited about this universe, right?
31:59
But I also understand that a lot of people are pretty hesitant
32:02
to, to sort of slip back into old school ML.
32:05
I don't think that's. I think you just have to get
32:08
over that fear at some level. Yeah.
32:10
Well, it's interesting you say that because it is a much
32:13
different scenario if it is only 500 examples, even though at the
32:18
end of the day you got to go and you got to have those chops to
32:23
know what you're doing, you it no longer becomes unpinioned and
32:27
API. Exactly.
32:31
And that transition is going to be rough, but that transition is
32:36
going to fundamentally change how you do business right within
32:39
the AI world because now you can get a precision recall curve.
32:42
This is the most exciting part about a classifier.
32:44
And I know I'm going to get a sale for saying precision recall
32:47
curves are exciting, but to me they are.
32:51
Now you have error margins, right?
32:53
Like this is game changing. You go from a world where you
32:56
just have to hope that the next token is the right one to a
32:59
world where you're like, oh, if your confidence score is below
33:01
.9, just don't answer the question.
33:04
And then you say, oh, we're making a lot of mistakes there.
33:06
Let's raise that confidence score requirement.
33:07
So if your confidence score is above .95, don't answer the
33:12
question, below .95, don't answer the question.
33:15
That's kind of the world we need to be in, right?
33:18
Because if you're an enterprise, you can't go out there and have
33:21
no AI is never going to be perfect, but at least you need
33:25
to know roughly how often it's going to go wrong.
33:28
And it's interesting, like classifier is one example.
33:32
The other example that I've heard from the folks at Process
33:36
for their AI agent is they want to know if their agents have
33:41
enough information to go and execute a task.
33:45
And that feels like something that could be a distant cousin
33:49
or a close cousin of a classifier.
33:51
Exactly same math problem, right?
33:53
You're, you're basically going to say, what's my threshold
33:57
confidence score above which I think we have the right answer.
34:00
You train the model over time to learn, hey, when I'm below .9,
34:04
I'm like never, right, OK, I'm going to be above .9.
34:08
It's like very simple to improve AI systems when you start adding
34:12
thresholds to them, because then you can just work in the world
34:16
where you're good, right? And you can say, hey, we'll
34:19
default to the old workflow when we're bad.
34:22
Right now, there just isn't that distinction.
34:25
The LLM is never able to say no. That's the that's the
34:29
manifestation of this problem that everyone's seen, right?
34:32
It it answers with equal confidence when what it knows
34:36
for sure and when it has no clue.
34:39
That was the biggest problem that they ran into in the
34:42
podcast that I did on their data analyst agent, is that folks
34:46
would ask it for data or how many partners have done XYZ in
34:52
the last month. And they could know that
34:57
information, but sometimes they wouldn't be given the right
35:03
amount of detail. And so the LLM would come back
35:06
with some kind of plausible answer.
35:08
It would go off and it would be thinking for a little while and
35:11
then would come back and would give you this full story.
35:14
And if you weren't really intimate with that data, you'd
35:18
be like, yeah, it seems about right.
35:19
Yeah. And so that is a really
35:21
dangerous scenario because you base decisions off of that,
35:25
which if you're not careful, can be completely fabricated. 100%
35:33
that is, the most dangerous chunk of answers is when it's
35:38
plausible but wrong. Right.
35:40
Everything else is actually in in that grade is actually pretty
35:43
fine. It's, it's that part which is
35:47
like, oh God, what do we do? And especially in regulated
35:51
industries, right? So if you're in finance, if
35:54
you're in healthcare, if you're in cybersecurity, you don't get
35:58
to make these mistakes more than once or twice before, like
36:02
meaningfully having to pay for them.
36:04
And so that's where we spend a lot of time thinking about how
36:10
do we make sure that when it's not confident, it's not
36:16
answering. That's the fundamental problem
36:19
you need to solve. No one's going to be mad at you
36:22
if you default to the old ways, right?
36:23
So if you have a search engine and it always answers the
36:26
question, but occasionally it says, hey, I'm sorry, here's the
36:29
search results, I don't think I can answer the question.
36:32
That's amazing. But every time it says made-up
36:35
answer, you start trusting it a little less.
36:38
Yeah, yeah. And it's really easy to lose
36:41
that trust. That's the other thing is that
36:43
it's hard to gain it and easy to lose.
36:46
But the the idea that you're bringing up to, I have heard
36:50
from Igor who works at Wise and he was talking about how LMS
36:57
should just be viewed as another node in a DAG that can do a bit
37:03
more fancy stuff. You don't need to think of it as
37:06
this whole revolutionary new product or new thing.
37:10
Let's just think about it a little bit more unsophisticated
37:16
and say, all right, cool. It can take unstructured data
37:20
and make it structured. That's what an LLM is great for.
37:26
Yeah. So I see it as two ways, right.
37:28
I see it one as wet clay. So it can sort of fit into any
37:33
position before you mold that node.
37:35
Anything can be an LL app. That's, that's the best part of
37:38
it being so flexible, right? You can treat it as a classifier
37:42
today. You can treat it as a regressor
37:43
tomorrow. You can treat as a generator the
37:45
third day. So as you're coming up with your
37:47
MVP system, it's amazing, right? It's a prototype solution at
37:52
that stage. And then when you harden that
37:54
clay, it's really good at one thing.
37:57
And that thing is fluency. So it's not good at decision
38:00
making. And if you use it for decision
38:02
making, at some point in your journey you will be like, oops,
38:06
maybe we shouldn't have done that, but it's really good at
38:09
fluency. So back when I was in grad
38:12
school, I used to build doing some work around persuasive
38:15
language modelling. So how do you generate text that
38:20
could potentially persuade people?
38:22
And our biggest blocker there was fluency.
38:25
So given a user query, we could come up with the best facts,
38:29
facts to try and convince you otherwise, but we couldn't
38:35
present those facts in a fluent way.
38:38
And now LLMS can do that. So this is four or five years
38:41
ago, right? And and that's kind of where we
38:43
were stuck with those systems. So if you start thinking about
38:47
LLMS as another node in this like AI systems world or ML
38:50
systems world, you start seeing a whole new set of problems that
38:53
can be solved. It was the same with sort of the
38:56
marketing use case, right? You could always targeting has
38:59
been good, retrieval has been decent.
39:02
The only thing we didn't have is fluency.
39:05
So if I could target you with the right material, how do I
39:08
actually give you that final piece of text that you're going
39:10
to read or video that you're going to watch?
39:13
Now he's unlocked. That's I think the hard clay
39:17
version of the LLM. So there is a purpose for the
39:19
LLM when you're demoing or MVP ING, and then there's a separate
39:23
purpose for the LLM when you're like in production.
39:26
What are some ways that you have seen folks actually having
39:30
success with AI and going back to what you were saying before,
39:35
having good AII think is the the key here and specifically good
39:40
for them too. Yeah.
39:43
So I'm a big fan of the code Gen. universe and specifically
39:50
for dev tools or MVPS, I've seen it actually.
39:56
I'm very, very excited about the no code low code dev tools
39:59
universe because basically what LLMS have or code Gen. has made
40:06
possible is having complete flexibility over your components
40:14
without losing any control. So without losing with still
40:18
having a lot of complexity abstraction.
40:20
Right. And and I'll rephrase that,
40:22
essentially the trade off always used to be flexibility for
40:25
complexity and that trade off is now gone.
40:29
That's what's really exciting to me.
40:31
You can have all your code. But.
40:35
You can also have it generated in a few minutes.
40:38
These systems, I still have yet to see them work well with
40:41
larger code bases. So I think code context
40:44
retrieval is a really big problem that pretty much
40:46
everybody in this world is is sort of trying to solve.
40:50
But when you have no baggage, when you have no code base is
40:53
dev toons. That's exactly you build a bunch
40:56
of these things every day or every week.
40:59
And now you can build them for a lot cheaper.
41:01
So I think like a dev broad team that now has five people can
41:05
ship so much more than it had protein yesterday that had five
41:09
people. So this one I'm very, very
41:12
excited about. I think the other one is
41:16
localization. That's been really hard for us
41:19
to do right. So I used to be an ML engineer
41:22
at TikTok and we spent a lot of time thinking about
41:24
internationalization because we were in about 150 countries.
41:27
I think that's one of those other low hanging fruits that
41:29
I'm very excited about. Can you localize content right
41:35
And and furthermore, hyper personalization is obviously one
41:38
of those big bets. It's always been a big bet in
41:40
advertising and now you can go even further.
41:44
To be fair, these things are both hard, right?
41:46
There are a lot of problems in both worlds with dev tools, you
41:49
might want to think a lot about like constraining universes.
41:52
So you don't want to use any database, but like if you use a
41:55
third party dev tool system, like a code Gen. system, it
41:58
could call any DB. And so there's, there's some
42:01
degree of like constraining that that needs to happen for a good
42:04
dev tool code Gen. company. And then with advertising, I
42:08
think it's just like a there's a huge talent shortage in
42:12
engineering for any industry that does localization, which is
42:17
usually sort of the content generation industries.
42:19
I'm not sure what the localization use case is.
42:23
Is that just serving the right ads to the right people?
42:26
No. If I create an ad in the US,
42:29
right? And I want to serve it in a
42:31
manner that appeals to people in Portugal, right.
42:35
So say it's about coffee. You probably have coffee with a
42:38
doughnut if you're in the US, but you'll probably have coffee
42:41
with a pasta if you're in Lisbon, right?
42:45
And so that's kind of the it's, it's a tiny change, but your ad
42:50
means so much more. And.
42:52
It's the same for blog. Portuguese it's going to be in
42:55
English. In the US it's in Portuguese you
42:57
have. I imagine you would want more
43:00
Portuguese looking people in the Portuguese one and American
43:04
looking people in the American 1.
43:06
So exactly they can relate to it better.
43:09
And I think that'll happen for news, I think it'll happen for
43:12
ads. I think it'll happen for blog
43:16
posts, right? Which case study are you serving
43:19
whom and what you emphasize on where There's a lot of these
43:24
like slices to that. We've always been trying to do
43:27
this. It's just the fluency is in the
43:29
blocker. Everything else in that
43:31
pipeline. I know where you're calling me
43:33
from. I probably know something about
43:35
your business, right? Identity resolution is pretty
43:37
good. I just, I don't have the fluency
43:41
engine and now I do. So that's kind of how what I'm
43:44
excited about. But the way that you can use AI
43:49
to get that last mile, if we're talking about, oh, I want
43:53
different looking people in the ad in Portuguese or in Portugal
43:59
versus in the US, unless you're doing video generation, then
44:08
you're not really going to get that right or not necessarily.
44:12
You can make it a retrieval problem.
44:14
So how you design your AI system is actually, it's a very
44:17
interesting industry, right? For example, here you could have
44:23
a collection of models from all over the world, and whichever
44:26
country you're in, you just retrieve that model and now it
44:29
becomes an interpolation problem.
44:30
It's not a generation problem. You're not generating a fake
44:34
Portuguese human. You're talking to human models.
44:37
When you say models, you mean human models or?
44:39
Yeah, yeah. Human models like the the
44:42
influencers, so you get like a Portuguese influencer, you get a
44:45
picture of them and you say, hey, we're going to please give
44:48
us the right to put your picture in our ad.
44:50
This is what the ads for. And then all you have to do is
44:54
in paint, right? Which is a much different
44:57
problem from generation. And actually in painting is a
45:01
lot more reliable than pure generation.
45:04
So now you also have a higher degree of determinism around
45:08
your system. And so we actually think a lot
45:12
about this, right? Like the way you design your AI
45:14
system can be very, very different from system to system
45:19
and you're trading off for different things.
45:21
So right now what we did is we turned the generation problem
45:24
into a retrieval plus infill problem.
45:29
And now we have a different pipeline where we can optimize
45:31
both of those systems. And the retrieval problem can be
45:34
purely deterministic. You don't need to use AI based
45:37
retrieval. You literally just go to a
45:39
database and say who are the models from Portugal and you can
45:43
infill. So you've reduced your
45:46
uncertainty drastically over like a one minute run
45:49
conversation. Yeah.
45:51
OK. And it's it's really thinking
45:55
creatively about that and thinking about that in these
45:58
different ways where if you've seen it enough, I'm sure you you
46:02
as a person who has worked with companies trying to do this or
46:06
doing it yourself, you've seen patterns.
46:10
And so you're able to go and say, well, yeah, I mean, you can
46:13
do it like that. But the way that I would do it
46:16
is, is like this. And so there's still is almost
46:19
like these design patterns that sound like they're kind of stuck
46:23
in your head and you need to get them on a blog post or
46:28
something. So I can go and scrape them and
46:30
then have an AI generate A blog post that has my name on it, but
46:35
it's really your blog post. Yeah, I think ML intuition is
46:40
very tribal. This is actually a big problem
46:42
that our industry has, right? Most of the people that have
46:45
been building these systems for a while, we learnt by failure.
46:49
So I failed 5060 a hundred times in building ML systems before I
46:53
was like, oh, I'm starting to get a hang of this.
46:57
I think enterprises that at large need to be a little bit
47:01
more patient with their ML teams or their AI teams because
47:05
they're also developing this intuition.
47:07
It's a matter of time. One of the things we're doing on
47:11
our platform is we're trying to sort of imbibe some of this
47:14
intuition into the platform. So the things that I'm telling
47:17
you, right, Hey, maybe we should design it this way, or maybe you
47:20
should use this base model instead of this one for this
47:22
reason. We're we're pushing some of that
47:25
onto the platform and we're, we're, you know, the goal is to
47:27
try and push as much of that as we can onto the platform.
47:30
So we don't actually think of, you know, emissary is, is sort
47:33
of a fine tuning platform. We think of it as ML intuition
47:37
delivered through infrastructure.
47:39
How do we make these AI engineers who are struggling
47:43
with this stuff, well, maybe one or two or ten fine tunes in or
47:48
prompt optimizations in to start getting access to this brain
47:54
that might have done hundreds of thousands of fine tunes or
47:57
optimizations. And that brain is is the sort of
48:00
the MSLE engine. So we are working towards
48:03
lowering that barrier because that is a big barrier right now.
48:07
But yeah, in the meanwhile we reach top on calls and say maybe
48:11
this is how you should try it. Until you can take those
48:16
opinions and bake them into the product.
48:19
And I really look at it as, yeah, the pattern recognition
48:23
that you've seen over the years. And I appreciate the fact that
48:28
you are calling out the obvious, which is it's a system.
48:33
And it's not just, OK, we've got this ML model or we've got this
48:38
AI model. We need to really look at the
48:40
system. It goes back to that old de
48:43
Scully post from back in the day that everyone used to quote
48:46
where it was a lot of different boxes in the infrastructure.
48:51
And only one of these small boxes was the actual model.
48:55
The rest were the different pieces of it.
48:59
And granted, it's, it's been updated since then because you
49:03
don't necessarily have the data pipelines anymore and you don't
49:06
have that training pipeline, but you need to think about the AI
49:13
product and the AI system that is around it as opposed to just
49:18
all right, cool, I'm hitting this API and then serving it up.
49:23
And that does feel like what a lot of people are learning these
49:27
days, one. 100% and actually they're learning it.
49:30
The one thing I'm always very excited about it is they're
49:32
they're learning it a lot faster than we did, right.
49:35
The ML industry we could decades to learn some things that like
49:38
the AI industry is learning in months.
49:41
And so it's very exciting to sort of see the pace at which
49:44
things are developing. You know, even evals like we
49:48
took so long as an industry to realize how important it was.
49:53
And I think it's taken maybe AI like a year and change from the
49:56
first systems in at least in demo to get to a point where
50:00
everyone's like, OK, let's stop and think about what a good
50:03
outcome is here and then we can sort of optimize towards that.
50:08
So I think that's that's sort of a big step in the right
50:11
direction. Yeah.
50:13
And realizing that certain evals mean nothing, like the leader
50:16
boards. And you need to have your own
50:19
evals that you keep close to your chest.
50:21
Otherwise in a few model iterations, they're also going
50:25
to mean nothing. Yeah.
50:27
That's you. You got to keep your evals, sort
50:31
of make them more robust over time, and your system has to be
50:36
learning from those evals. It means nothing if you're like,
50:39
my model is doing X or is that quality X?
50:43
If that's not feeding back into the model to make it X + 1,
50:47
right? Like at that point, maybe don't
50:49
do evals because you're just wasting time.
50:52
It's, you have to close the loop from generation evaluation to
50:57
improvement. Without that last piece of the
51:00
loop, your, your, your, your system is static, right?
51:04
And that's the sort of the worst thing you can have in a
51:06
perishable universe is you know, the baseline is going down and
51:10
you're not doing anything to to sort of keep improving it.
51:13
Perishable goods, man, I love that idea and the vision of as
51:19
soon as you put something out, it's already stale and we knew
51:22
that. It's so funny how speaking to
51:24
you, you're you're really bringing some of this stuff
51:28
back. Like as soon as we would put out
51:30
AML model to go and you would have to figure out the features
51:32
or you would have to retrain it as much as possible.
51:35
And you would be looking at the model drift.
51:38
And when it would hit a certain level of drift or if there was
51:43
some kind of a big thing that would throw it out of whack, you
51:46
would try and trigger a retraining and make that happen
51:49
as fast as possible. And then you kind of have that
51:52
champion challenger as you deploy one new model to see is
51:57
it working as well as this other one?
52:00
Let's see. And so now we're we're taking
52:03
that same idea, but we're saying, all right, well, what
52:05
about these prompts? Can we have champion challenger
52:08
prompts that will kind of work? Are they going to exactly and I
52:12
think once you. Start getting good eval
52:14
outright. It's very easy to start doing
52:16
those experiments. So the Step 2 here after that
52:20
Step 1 of what good and bad looks like is how, how can we
52:23
start experimenting, right? Can we do like a, for us, it was
52:27
so normal to launch a new model, do A5 like an offline eval,
52:31
which was on your existing regression set and then an
52:34
online eval on like 5% traffic, right?
52:36
These were like standard mechanics that we ran for every
52:39
model that we trained because you knew that a lot of the
52:42
ground truth was limited in offline eval.
52:45
So you had to see what this new Challenger prompt looks like
52:50
online. And you something get
52:52
comfortable with this idea that that machine learning is
52:54
iterative, that you're almost always going to be
52:56
experimenting. And I think that that sort of
53:00
scares folks a little bit because with software, you sort
53:02
of, you know, put your head down and then release.
53:05
And if you have to patch, it's a bug, it's a bad thing, right?
53:10
In ML, you're always patching like that is all you do.
53:14
You're just like sticking patch on top of patch.
53:17
And it's a feature. It's not a bug.
53:20
So I think that mindset is, is something that's go undergoing A
53:23
transformation there. And I'm very excited about it.
53:26
So basically what? You've seen as the graduation
53:31
from just hitting an API, calling some kind of open AIAPI,
53:35
Anthropic API, whatever, and calling it from any language
53:41
that folks want, and then embedding the prompt in that API
53:45
call. That's great until you get to a
53:47
certain maturity level and then you have to do what then you
53:51
have to start thinking about. What are the other steps that
53:53
you might want to do locally, right?
53:56
So you might want to have an AI server that's sitting right next
53:59
to you. You might want to embed locally.
54:02
This is like one of the simplest ones is you probably don't need
54:05
to call an API endpoint to embed your query when the user sends
54:08
it. You can just hold because the
54:10
embedding models are super small.
54:12
So you can host your own embedding model on the same
54:14
server. And now you've cut down the
54:16
network latency by about a second, right?
54:18
So if you're running this a million times a month, you're
54:23
reducing quite a bit of time for for your users.
54:27
And to do all of this, though, is actually pretty hard in any
54:33
language other than Python. This is kind of a big challenge.
54:36
Why is that, right? The ML community thrived in
54:39
Python, and so all of the support that exists is in
54:43
Python. That's not to say you can't load
54:45
a model in another language, right?
54:47
That would be sort of superfluous to me, say it's that
54:50
the amount of time it would take you to do that and do that right
54:53
is usually just not going to be worth it.
54:56
We have like a decade's worth of, you know, ML support in
55:01
Python, whether that's your Pytorge, whether that's
55:04
Tensorflow, whether that's now Transformers.
55:07
There's so many layers of abstraction that make your life
55:09
a lot easier. And so we're seeing this sort of
55:14
universe and, and it could be a good universe or, or a
55:16
challenging one. We'll know in a few months in
55:19
where folks are building AI systems into their existing code
55:22
bases. That is the fastest way to go.
55:26
But that usually means that once you hit the ceiling that you can
55:30
with foundational models, you have to RIP everything out,
55:34
create a separate Pythonic universe, and then start loading
55:39
everything in there, starting over from scratch if you aren't.
55:43
Doing it in Python exactly and so.
55:46
Now that we could get lucky and like there is support for ML in
55:50
other languages by the time you need to do that.
55:53
But This is why I'd like the general recommendation I tend to
55:56
give is, I know it's going to be a big lift to now start working
56:00
in a different language. But if you look at this AI
56:05
universe very seriously, then you should probably start
56:08
thinking about an AI back end, right?
56:10
What does it mean for you to have like a back end service
56:13
dedicated to your AI systems? Then you can hand off eval to
56:17
that system. You can hand off API calling,
56:20
model management, gateway management.
56:22
It just like the separation of concerns is so clean when you
56:26
start doing that. But if there is an upfront
56:29
investment of hey, are we really going to take on a whole new
56:32
language? So I think there is some back
56:35
and forth there to to be had. Yeah, because that was one of
56:38
the key things. Is that people aren't using
56:41
Python or not as much as you would suspect, especially if
56:44
they're not used to living and breathing in AIML world 100% I
56:51
think. People are using Python a lot
56:53
less than folks would believe because there is no immediate
56:57
reason, right when you, like, start hitting an API, you can
57:00
hit an API in any language. So maybe you're not using, you
57:03
know, open AI and topics. SDKS are very limited.
57:06
I think there's 3 or 4 languages, but you can just call
57:10
an API endpoint, especially if you're used to.
57:14
Coding in JavaScript, Why would you switch over to Python
57:17
exactly until you hit that? Limit.
57:19
And then that's what you're saying is that you're seeing
57:23
folks, you try and tell them in the beginning when they're
57:26
starting off their journey and then they say, yeah, yeah, yeah,
57:30
whatever you. Don't know what you're talking.
57:33
About kid, I think it's there's like an interesting.
57:36
Enterprise decision there, right?
57:39
So there is a big lift to move to Python.
57:44
So maybe it is optimal to MVP in your existing language as long
57:48
as you know that if the MVP works, you're going to have to
57:52
move, right? So maybe you say, hey, we don't
57:56
want to move to another language before we have we see some AI
57:59
value. But if you're convinced that
58:02
your business is going to run on AI or BAI native in the near
58:05
future, then I always recommend, you know, go all in.
58:10
If you're not convinced, then yeah, it might might actually be
58:13
worthwhile to stay. But think through that journey,
58:16
like make sure that you're cognizant of the decision you're
58:19
taking is sort of my focus there versus sort of pushing anyone in
58:24
One Direction over the other. Well, and what was it you were
58:27
saying about? OK.
58:28
Then you need to eventually bring your model in house.
58:32
And I was saying, well, it's probably through Microsoft Open
58:36
AIAPI or or the Anthropic AWS version.
58:42
But what was it? The the model isn't necessarily
58:45
the hard part here. Yeah, it's the, it's everything
58:48
around it. So that's why we were talking
58:50
about like the querying stuff, right?
58:52
Like how do you generate a query embedding or if you're trying to
58:56
generate like, rewrite your queries?
58:57
So query reformulation, there's no reason for you to call a
59:01
foundational model to do query reformulation.
59:04
You could just do it locally with a tiny model.
59:06
So you don't need to use Anthropics through AWS or Azure
59:11
Open AI. You just host that next to you
59:14
and it's like a dollar an hour. If you're using an A10 and it's
59:18
faster, it's cheaper, there's no reason that you wouldn't do
59:21
that. But if you're not living in a
59:24
Pythonic universe, you basically cannot do that, right?
59:27
Because you can't do that in Ruby.
59:28
Or you can't do that in JavaScript.
59:30
Do that in any language. So you could do this in Ruby,
59:33
you could do this in TypeScript. It just would be really hard,
59:38
right? Like you'd have to rewrite a lot
59:40
of libraries from scratch. That's kind of the trade off
59:45
there is if you're working in, you know, Java, for example, you
59:51
basically will need to rewrite a lot of libraries.
59:54
I actually went through this personally where I was doing
59:57
this for Go, right? And it became such a big blocker
1:00:03
that eventually I was like, you know what, we'll just call the
1:00:04
API endpoint. Like I really don't think I
1:00:07
should rewrite Transformers and Go.
1:00:11
And it's it's an interesting challenge to to wrangle with.