Latest / Agentic Conversations (formally mlops.community) / LLM Search, UI/UX challenges, Context Engineering and the 80/20 of Eval

Transcript

LLM Search, UI/UX challenges, Context Engineering and the 80/20 of Eval · Agentic Conversations (formally mlops.community)

0:00
People talk a lot about system prompt, tweaking the prompt or
0:05
choosing different model. What's the best model out there?
0:07
One of our hard earned lessons is I spent so much time on
0:09
context engineering with the teams.
0:11
Let me explain what context means, right?
0:18
So in my role, I build AI agents in these different industries
0:21
and these agents are kind of used by 2 billion customers
0:24
around the world. Talk to me about what agents
0:27
you've been building. So in general, there are two
0:29
classes of agents that I've been working on.
0:30
One is agents for productivity. So internally we built a tool
0:34
called Tokan, you know about it. This tool is used by 15,000
0:38
users internally at process. And these are people from
0:41
finance and designers, product managers, engineers and people
0:45
use it for different things. So we started building this tool
0:48
three years back. The technology has evolved so
0:50
much. Initially we did a lot of things
0:53
to make the LLM work so that you know, there was an intent
0:56
detection at the beginning and so on.
0:57
It blows my mind that you know, now you don't need like the
1:00
model itself is so better that you don't need these things.
1:03
So sorry. SO11 category is tools for
1:07
agents for productivity. So Tokan second is agents for
1:10
e-commerce and there I have experience in building agents
1:14
for online shopping assistant. So OLX is one of our portfolio
1:18
companies. So with OLX we built a shopping
1:21
assistant. The idea is that it helps
1:23
understand the user intent. So you can say that you know, I
1:26
want the latest headphone or you know, I, I'm going for a hiking
1:30
trip and I don't know what to buy help me.
1:33
So it can take very broad request from the user
1:35
understand, connect to the OLX catalog.
1:38
So that's an example. And currently I'm working on a
1:41
project with in a food delivery business where we are kind of
1:45
reimagining how people would order food in next year or two
1:50
years where, you know, things like not so usually if you think
1:56
about how people order food right now, they go on the search
1:59
bar, you enter burger, right? But how about it also, You know,
2:03
you can go and you can say that, you know, I want to have a
2:06
romantic dinner with my wife. And then it understands you, it
2:09
connects to the catalog and so on.
2:10
OK, so we've got 4 themes that we want to touch on.
2:14
Yeah. First one, which is on
2:17
everyone's mind, is the context engineering piece.
2:19
You've got some takes on that, yeah.
2:21
Totally. So Andre Kapati recently
2:23
published a tweet where he said that he likes the word context
2:26
engineering more than prompt engineering.
2:28
And you know, I was reading it and in my experience in building
2:30
these agents, I spent so much time together with the
2:33
engineering teams working on the context.
2:35
So I was completely, as soon as the moment you published the
2:38
post, I think the context engineering word got viral.
2:43
And when I think about context engineering, it reminds me of
2:45
the day with, you know, when you know, data engineering, data
2:48
scientist, you know, I was a hands on data scientist at some
2:51
point in my career. And you know, we always said
2:53
that, you know, data science or AI is the shiny thing, but you
2:58
know, if you're garbage in, garbage out, if you don't work
3:00
on your data context engineering for me is like that.
3:03
We are doing a podcast, I see people talking on Twitter on
3:05
different podcasts and people talk a lot about system prompt,
3:10
tweaking the prompt or choosing different model, what's the best
3:13
model out there recently about MCP tools and so on.
3:17
But for example, you know, if you think about the latest
3:19
models, if you think about the latest models, for my use case,
3:22
it's not, it's not really about that model.
3:24
If I use model A versus model B, and if both of them are
3:26
state-of-the-art, they already do a good job.
3:29
What makes the difference between A&B is A is with
3:33
context, B is without context, then A would be much better than
3:37
B. And let me explain what context
3:39
means, right? So a lot of times.
3:41
So imagine a food delivery chat bot.
3:44
Aside from people asking for food, when you put a chat bot in
3:48
wild, people ask about many different things, right?
3:52
Imagine you know you want to order food.
3:54
Free food. That's what I would instantly
3:56
do. Exactly.
3:56
Exactly. So people care, different people
3:58
care about different things. So there are people who care a
4:00
lot about free food, cheap food, promotions.
4:03
There are people who care about maybe you have a particular
4:05
coupon or your payment card. So a lot of time people come to
4:09
a chat bot, they say that, OK, do you accept this payment card?
4:12
Show me, show me sushi on promotions right now.
4:16
Or maybe it's lunchtime and or maybe it's breakfast time and I
4:22
am asking for pizza and none of the pizza places are open.
4:25
And you know, you don't know about my use case as a user.
4:27
Maybe I'm someone I'm an executive assistant, I'm
4:30
planning already for lunch or maybe I want to have pizza.
4:34
So in this case, for example, the opening and closing hour of
4:36
a restaurant becomes the context.
4:39
So if your LLM or if your agent doesn't know about that, then
4:43
you know, we have tried this before where if your LL, if your
4:46
agent is just good at searching for for pizza and it doesn't
4:50
know about the opening closing hour, it doesn't know about the
4:52
promotion, it doesn't know about the payment information.
4:56
If you put it in real world, it's not useful for people
4:59
because you know, I care about a lot of these things.
5:01
Users care about a lot of these things.
5:03
So one of our hard earned lessons is I spent so much time
5:06
on context engineering with the team.
5:08
So you know, I cannot stop talking about.
5:10
Yeah. And I don't want to brush over,
5:11
there's a few different places where it's a very difficult
5:15
problem that you were just saying when it comes to ordering
5:19
the food and the promotions, for example.
5:23
And you were saying that it's not like there's a database that
5:26
you can query that has all the most up to date promotions.
5:31
Yeah. So the data challenge in real
5:35
time, knowing what promotions are live from what companies and
5:40
then feeding that into the context window is where the real
5:43
challenges. And it's almost, I look at it a
5:45
little bit like when you're crafting that prompt you put in
5:50
variables and getting the proper data into those variables is the
5:57
unsexy job of the data engineer. Or it's, it's more data
6:00
engineering like than anything. But now I guess we're calling it
6:04
context engineering. I think we all agree that, you
6:07
know, most of the enterprise data, you know, it might look
6:09
good from outside, but it's messy.
6:11
So unfortunately, you know, if you go in an enterprise, it's
6:15
it's, it's not like there's a database where you can say just
6:17
just give me all everything on promotion.
6:19
So a lot of data is real time. Maybe a restaurant is running a
6:22
real time promotion during lunch from 12:00 to 3:00 on sushi,
6:25
right? It's very difficult for a
6:27
database to have that. So a lot of that information is
6:29
in real time databases. It's scattered over different
6:32
databases. So the the unsexy work here is
6:36
that, you know, I have data engineers in the team, engineers
6:38
in the team who spend a lot of time kind of connecting all
6:40
this. So imagine now a user request
6:43
comes in and they say show me sushi on promotion.
6:47
You kick in these data pipelines, you bring the right
6:50
context, you make it a part of the prompt, and then the
6:53
response that you get is what the response that the user wants
6:56
to. And there's not a way to do this
6:59
with search that is more simple. No.
7:03
So search is the next topic I want to talk about, but search
7:05
solves a different problem. Search is more about, for me,
7:09
search is about when you're when you're searching for dishes or
7:12
when you're searching for restaurants.
7:14
That's what that's a burger, vegetarian burger, pizza or
7:17
McDonald's. But when you're searching for
7:19
does McDonald's have promotion? Does McDonald's accept payment
7:24
card? Does McDonald?
7:25
Is McDonald open right now? That is context.
7:28
So that's the difference for me between search and context.
7:31
Context is made-up of four things in my opinions.
7:34
I spoke about one of them. There's another ones which I
7:37
want to cover. One of them being The Dirty
7:39
data. Getting it the right context
7:41
when? Yeah.
7:42
It's composed of four things. Two of them are simple.
7:44
Everyone talks about IT system prompt user message and I think
7:48
we have heard enough about, you know, you need to have the best
7:50
system prompt user messages, the dynamic message the user sends.
7:54
So you put that in the prompt as well.
7:57
We spoke already spoke about, you know, bringing the
7:59
enterprise context, The Dirty data pipeline.
8:02
And the 4th context, which is very important is the user
8:04
history. And this is where, for example,
8:07
it's connected to memory. So, you know, there's been a lot
8:09
of discussion on long term memory, short term memory.
8:13
The way I think about is there are so many Jenny I agents
8:18
products out there. Imagine you know, everyone is
8:20
building a shopping assistant, everyone is building a food
8:22
ordering assistant. If even I think about me as a
8:26
user, I don't have an A loyalty for a particular product.
8:29
I'm using ChatGPT today. I'm using another product next
8:32
day. For me, something that creates
8:35
stickiness to a product is if that product knows about me.
8:39
So if I'm using ChatGPT as an example for last 15 days and now
8:44
you give me a new tool, I would use it unless ChatGPT knows so
8:48
much about me that OK, I the switching cost is high.
8:51
Yeah, the switching cost is high.
8:53
And that comes from memory. And that is also part of context
8:56
where you put the user history as part of context.
9:00
Yes, it's funny you mention that because somebody said when they
9:02
were joining the community, they were very interested in the
9:05
problem set of being able to port that memory and that
9:10
context from one provider to the next.
9:12
Yeah. That's amazing.
9:15
If you do that, then then you could switch, right?
9:17
Because if you do that but. It's not like you can just hit
9:20
like download CSV and then upload it to the next.
9:23
Provider, I think there's a lot of data privacy issue with comes
9:26
with that because it contains a lot of information about you
9:29
which you might want to share or not.
9:33
And again, one last point there is, so I'm sure you've seen
9:38
these diagrams that people share about memory, long term memory,
9:41
short term memory, you know, episodic memory and so on.
9:44
So memory can be handled in many different ways.
9:46
It's almost like, you know, when you're having a conversation,
9:50
maybe there's a model which is running behind the scene, it's
9:52
noting facts and that's a part of memory.
9:55
But to be honest, one of the things that we tried, which is
9:57
much simpler than any of this and it works, is let's say
10:01
you're building a shopping assistant or a food ordering
10:03
board and you already have a product, right?
10:06
So there's already OLX or ifood. So there's already users who are
10:13
on OLX who did not have conversation yet who are on I
10:16
Food. You've got that data.
10:18
You got the data, you already know what food they ordered, you
10:22
already know what items they browsed.
10:25
You can easily use that as a simple cold start context.
10:31
And I have seen that create wonders where you know when you
10:35
launch your product. You don't need to be you know.
10:37
My advice here would be when you launch your product, because
10:40
there's already so many things to solve and you know, your
10:42
obsession should be about product market fit and, you
10:44
know, not having the best technical solution out there.
10:47
You should just use the context that you have from your existing
10:50
app and put that in the memory. Put that in, you know, the
10:53
context of the system prompt. And that already does wonder
10:57
that will give you a runtime of three months.
10:59
And then you start, you know, when people start having
11:02
conversation more, they start using your product more than you
11:06
get some dynamic memory. But I think that's one of the
11:09
lesson that I've learnt where I think the way people think about
11:12
it already from day one, it's complicated.
11:15
There's a simple solution for to cold start.
11:18
OK, so that is context engineering.
11:20
Yeah, that is context engineering.
11:22
That's it. Search, search, search is such
11:27
a, you know, so my experience has been building AI agents in
11:32
e-commerce shopping, food and search is a fundamental tool
11:38
when it comes to that. It doesn't apply for all the AI
11:41
agents out there. Let's say if you're building an
11:43
AI agent for supplier side, you know, for, for a car dealer or
11:48
you know, for a restaurant, search is not that important.
11:51
But when you're building an an e-commerce agent, search is the
11:54
most fundamental thing. It's actually the start of the
11:56
user journey. If you don't get the search
11:58
right, Let's say if I search for burger and if I get the pizza,
12:03
my eye drop at that point, you know, it breaks the trust.
12:06
I will maybe you have other tools to manage my cart, you to
12:09
do other things, but I will never go further in the journey.
12:11
So search is very important and I spend a lot of time with the
12:15
engineering team on, you know, fixing search.
12:20
Let me talk about few things that you know, maybe we've
12:23
learnt. So most of the enterprise search
12:25
is still keyword based and you know not to put it down, it's
12:29
for the right reason because keyword based works.
12:32
If I say burger, there's a taxonomy defined for burger and
12:36
you know, show me burger, you don't need fancy semantic search
12:40
to do that. So keyword based works.
12:43
But if you imagine people searching on a search bar versus
12:48
people talking to an agent, the chat bot you put out there, the
12:51
kind of conversation and especially when you have voice
12:53
enabled, the way people express themselves to an agent, to a
12:57
chat bot is so different. The kind of queries are so
12:59
different. I would go and I would say that,
13:01
you know, I already give these examples, right?
13:03
You know, I want to have a romantic dinner with my wife,
13:05
press enter, very, very broad. Or if I'm building a shopping
13:09
assistant, we have seen people just go and say that I'm going
13:12
for a hiking trip. I'm a beginner.
13:14
I don't know what to buy, help me or help me furnish my house.
13:18
These are the kind of queries that you get and you cannot
13:22
deliver these queries with keyword search.
13:25
So we spend a lot of time thinking about first semantic
13:28
search. And semantic search is not new.
13:30
It has been around. It's a search based on
13:32
embeddings. So you, you know, you know for
13:35
the search engineers out there. So you there are, there are
13:38
already standard way of doing these things.
13:39
Semantic search is hard. It's still hard to crack and
13:43
keyword search is still important.
13:45
So most of the time the solution is something hybrid where you
13:48
know, you get a query, you look if a keyword search can answer,
13:51
can answer it. If not, you go to semantic
13:53
search. So it's not new, but it's
13:56
difficult and we spend a lot of time talking about it.
13:59
So let me explain that with an example.
14:02
When I say so, I'm vegetarian. When I say vegetarian pizza, if
14:06
I have keyword search, then I would get items that have
14:10
vegetarian pizza mentioned either in their title or
14:12
description. But pizza Margherita that's
14:15
vegetarian. Maybe you know, it's not obvious
14:18
to mention vegetarian in the title of pizza.
14:20
Maybe the the restaurant person did not mention it.
14:22
And then all of a sudden pizza Margarita will not.
14:25
So this is keyword search. This is the limitation of
14:27
keywords. Now let's say you move to
14:28
semantic search. This can be solved with semantic
14:31
search because then you have embeddings and then pizza
14:33
Margarita embedding is very close to the embedding of
14:36
vegetarian. It's solved.
14:38
Now let's say, come to the example that I say romantic
14:41
dinner, that itself, that cannot be solved.
14:44
So romantic, you know, what is the embedding for romantic?
14:47
Romantic is different for you, Romantic is different for me.
14:49
That cannot be solved by semantic.
14:51
And we see a lot of these queries which are broad, which
14:54
are ambiguous, very fuzzy. So this is where again, you
14:59
know, semantic search is the first layer.
15:01
But we spend a lot of time building a pipeline of search
15:04
where when these queries come in, you have a step before
15:09
search and you have a step after search.
15:11
The step before search, we call it query understanding, query
15:14
personalisation, query expansion, call it whatever,
15:17
where again, you use an LLM. So you already have an LLM for
15:20
the agent. But let's say now you have
15:21
search as one of the tools. Within that tool you have a
15:24
pipeline where the first step of the pipeline is already using an
15:27
LLM to understand the query. So Romantic dinner, it
15:30
understands. And maybe if it has my user
15:32
profile, it says that this is what romantic means for Nishi or
15:35
Romantic in general. And LLM says that, OK, let's
15:37
break it down into, I don't know, cupcake is romantic and so
15:42
on. Candle it.
15:43
Yeah. Come on, cupcake.
15:46
What kind of romance are? You talking about we have
15:49
different definitions of romance apparently.
15:54
I mean. Out of all the ways that I would
15:58
describe romantic dinner, Cupcake was not one of them.
16:02
I hope my wife is not listening to this, but yeah, guilty as
16:05
charged. So anyway, yeah, so there's a
16:09
query, there's a query understanding step which breaks
16:12
down your query, maybe into multiple queries.
16:15
Then you run it through your search pipeline, keyword search,
16:17
semantic search. Then there's a re ranking step
16:21
where again you use an LLM where you know, initially you get a
16:25
lot of candidates. Then this is, you know, re
16:28
ranking is again re ranking is an old concept from the machine
16:31
learning world where we had these algorithms, LTR learning
16:34
to rank and so on. But in the new world of LLM.
16:38
So those algorithms are still important, but they also in my
16:44
experience, I've seen them fail when it comes to these new kind
16:46
of queries where you have a lot of context from the user.
16:49
And there you, you can have another re ranking layer where
16:52
you again, you use an LLM. You say that, OK, this was the
16:54
query from the user. These are the 1000 responses
16:57
from the 1st 2 steps. This is the context about the
16:59
user re rank. You know what is and then you
17:02
get these three or these ten options that you present to the
17:05
user. So that's what our typical
17:07
pipeline looks like. And you know, again, I can can't
17:10
stress it enough. It sounds simple, but search is
17:13
something that has, you know, haunted me in each of my
17:15
projects. It's difficult, it's difficult,
17:18
it's messy. It feels like a new paradigm of
17:20
search too, or just like a building on what was already
17:24
there and trying to leverage the new technology of.
17:28
LLMS yeah, totally, totally. Now with the all the
17:32
advancements in LLM generative AI agents like agentic search
17:37
and LLM helping in search and LLM augmenting search.
17:41
So you're not not replacing, but semantic search is still
17:43
important, but you know, doing some steps things before things
17:46
after. So that's the new paradigm.
17:48
And so you're right, like search is really being reinvented with
17:53
LLMS. And I think, I think very few people talk about it.
17:57
But if you're building an e-commerce agent, search is
17:59
fundamental and it's very important to crack that.
18:02
Yeah, let's talk to UI. Let's talk UI first.
18:07
Let's talk UI. Let's talk UI.
18:08
OK, so again, I have launched many agents, many Gen.
18:13
EI experiences and typically the way we launch them is we AB test
18:19
and I have burned my hands so many times.
18:23
You know, it's again, it's it's there's a lot of pain there that
18:26
I'll give you an example. You know, we want to build
18:28
online shopping assistant. The first thing we do is, you
18:31
know, let's build a ChatGPT for everything, right?
18:34
So let's build a conversational experience.
18:36
We build that. We test it internally.
18:40
I tested it. I spent a lot of time, you know,
18:42
long hours at night. It's super interesting.
18:44
It knows you, it's connected to the catalog.
18:46
You can you say that you know I want to furnish my house it
18:49
shows you furniture it shows you you know the chair so far
18:52
separate sections and so on it's beautiful it works you're so
18:55
excited that you're. Also so benevolent when you're
18:57
using it right, You, you aren't thinking like, how can I prompt
19:01
this to give me free furniture? You're thinking, how can I
19:03
prompt this to get the right furniture.
19:05
You also know what prompting is, so you probably, yeah, explain
19:09
it differently. Totally.
19:10
So I'm. Already seeing how this is going
19:12
to fail. Miserably.
19:13
Yeah, totally. So we are so excited.
19:15
I'm so excited we launch it. We think we we build the best
19:18
product out there. We launch it, we AB test it.
19:21
Our test like it really falls flat on our face.
19:24
We are like really like I'm refreshing the results and it's
19:29
it's it's terrible. It's not even you know, it's not
19:31
even bad. It's terrible.
19:33
Then you think you know what's? Terrible because of nobody's
19:36
using it. Or no people are using it but
19:38
they don't like it. And they're giving you that
19:40
feedback right away or? You don't immediately get
19:44
feedback, but you you can bounce off, you can measure the
19:46
conversion numbers. So A&B itself does the magic of
19:49
AB testing, right. So you have a conversion number
19:51
on A, you have a conversion number of on B, and you can see
19:54
the conversion numbers. Broken on this data.
19:57
You look at the yeah, you look at the agent and everything is
20:00
amazing. It's still amazing.
20:01
So why does it not work so? And this has happened, you know,
20:06
again, you know, I'm, you know, talking about failures.
20:08
It's also important to, you know, embrace failures.
20:10
And this has happened many times.
20:12
And you know, my learning here and you know, again, I'm a
20:15
technologist. You know, I'm not a designer,
20:17
I'm not a user researcher. So I also learned it the hard
20:19
way is the way I think about is it is so there are two
20:24
dimensions. One is technology.
20:25
Second is user adoption technology.
20:28
Like I feel that technology today is more advanced than the
20:33
use cases. It was not always like that.
20:35
So four years, four years back, I tried to do a project where I
20:38
felt that I'm so ambitious with my idea, but the technology is
20:41
not ready. These days.
20:42
Whenever I try to do a project, I feel our technology is, you
20:45
know, few steps ahead, how can I use it?
20:47
That's the problem. So that's technology, but user
20:50
adoption is not going at the same pace as technology.
20:54
Think about it. We all use ChatGPT.
20:56
You know, ChatGPT, it's been three years now and I think it's
21:00
been internalized. People are now understanding.
21:02
We're learning new ways to use it every day.
21:04
We're practicing. Oh, this.
21:05
Actually, I should ask ChatGPT instead of just doing what I
21:09
normally would do, like ask a friend or ask a doctor.
21:12
Yeah. And you and me, we are probably
21:14
maybe buy a sample that, you know, we are both in this field
21:17
and you know, maybe we are in a bubble, but you know, even my
21:21
daughter uses it. I see my my mom does something
21:23
with it when we go in, you know, social circle, people who are
21:26
not in technology, I see. So it has already kind of
21:29
penetrated outside. So I won't say that, you know,
21:31
it's a hype or it's in a Vienna bubble like it's going.
21:34
So chat bots for productivity, chat bots for these kind of use
21:38
cases, general use cases. It's it's a thing in the world
21:42
where we live in today and people, more people know about
21:44
it. But think about it.
21:46
Do you use a chat bot to buy stuff?
21:49
I don't. Do you use a chat board to order
21:52
food? I don't.
21:53
And imagine it's been three years.
21:56
It's it's surprising. It's surprising, you know that
22:00
and we all there's so many demos.
22:01
People are talking about it. We are at Ray Summit, the
22:04
hackathon. People build this stuff.
22:06
booking.com. There are these assistants,
22:07
travel planner, all these ideas in our our head.
22:11
But as users, I eat food, I shop for things, I go for, I book
22:16
flights, I book hotel. But I so far and I'm in this
22:20
field, I'm obsessed with this technology, but I haven't made a
22:23
single of this. You think that's AUI problem?
22:26
I think it's a user adopt. It's AUI and user adoption
22:29
problem. It feels like the UI
22:32
inadvertently introduces so much friction, and it's not the way
22:36
that we're used to doing things on the Internet for our shopping
22:41
experience that we say, you know what, it's easier to do it the
22:46
way that I know how to do it, Yeah.
22:48
Exactly, exactly. So so after we failed, we did a
22:51
bunch of user research. You know, I have a by the way, I
22:55
have a new found respect for designers and user researchers,
22:58
I think, and now I appreciate. So I work closely with
23:00
designers, user researchers in last few years and I think I
23:04
understand this feel much better now and a lot of respect.
23:08
So in fact, these days, you know, when we put together a
23:11
project, you need a designer and user researcher because
23:14
technology is one side you need to understand.
23:16
In the end, you want to solve user problem, right?
23:18
So after the test failed, we did a user research, we spoke to, we
23:22
did surveys, we, you know, we, we called in some users and some
23:26
of our learnings. 1 So if you're the user and if I, if I give you
23:31
OK, now I give you a new way of doing so.
23:34
You're familiar with the UI. You, you go use, you use it
23:37
every day. And now I give you a new UI.
23:40
It's friction. You will not use it naturally.
23:45
You will use it if it's really solving A fundamental problem
23:50
for you. It's really, it's something
23:52
fundamentally different. Maybe something you used to
23:55
struggle a lot that I don't know.
23:58
Maybe you're looking for house and you had a lot of constraints
24:01
and a search bar was, you know, you were not able to define with
24:04
the filters and search bar. And now if you go, I give you a
24:07
voice experience and you just spoke to it and it just
24:09
understands you, then you use it.
24:12
But if it's just incremental, if it's a better way of doing
24:16
search. Yeah, and it's a whole new
24:18
interface. Yeah, that's a pain in the ass.
24:20
Yes, I as a. You would not use it.
24:22
Yeah, I would be pretty pissed too that you changed the
24:24
interface on me. Yeah, exactly.
24:26
So that's one of our first learning is that when you give
24:29
people something new, when you change the UI, it has to be
24:34
something you, the value for them has to be very and they
24:37
should know it immediately in the 1st 30 seconds that OK, this
24:40
is the value for me. Because if they feel that OK,
24:42
why am I even doing this? Is it you know?
24:44
Why don't I just go to search bar and use my?
24:47
Especially when you're trying. To introduce AI into it, because
24:50
it feels like, oh, these guys just want to be able to say to
24:54
their stakeholders or their stock, yeah, what do they call
24:59
it? The shareholders?
25:00
These guys just want to be able to say to their share
25:03
shareholders they're using AI so the stock price goes up.
25:06
It's so important to hand hold the users.
25:09
Often we build tool and we just say, you know, there's the same,
25:11
right? So you build, they will come,
25:14
doesn't happen. So you build then you need to
25:16
hand hold. So on boarding, guiding the
25:18
users. A great example that I always
25:20
like is, you know, I have Alexa in my house.
25:25
It's a black box that sits there.
25:27
It's so inviting. It says talk to me about
25:29
anything. Then you talk to Alexa and out
25:32
of 10 things, I, I talked to Alexa about 88, it fails, 2
25:37
works. But then that's a design
25:40
problem. So if, and that's the case also
25:43
with many conversational chat bots, we say it's a plain
25:45
screen, right? And plain screen is nice, neat.
25:47
I like it. But at the same time, if I'm the
25:49
user and you now give me this thing and behind the scene
25:52
there's an agent which can do a lot of things.
25:54
Maybe there are 20 tools connected to that.
25:56
I don't know. I didn't build it.
25:58
So you need to onboard me, you need to guide me.
26:01
And over the last few years I've been working with designers and
26:04
there are some kind of excellent ways to, you know, some and
26:08
these are not new in the world of design.
26:10
There are you can you can, you know, maybe when I enter, you
26:13
can already have some boxes that I can interact with.
26:16
You can have tool tips as I go. So it can be a guided, it should
26:19
be a guided journey for the user.
26:21
So that's a second learning. So I've tried both a chat bot.
26:25
So, you know, immediately go for chat bot, it fails.
26:28
Then you try UI, you say that, OK, this is the UI.
26:32
Then let me try a new UI powered by Gen.
26:34
EI. Make it different.
26:36
You know, there are a few things happening on the screen.
26:37
It's much more dynamic. That has better results than
26:41
chatbot. But you know, that's also that
26:43
that is constrained, that's limiting.
26:45
Chatbot has more flexibility, right?
26:47
So OK, that also doesn't work. So I'm kind of coming to
26:51
conclusion that the best interface is a mix of UI and
26:55
chat. So these days, so there's this
26:58
word that, you know, we discuss a lot in our, you know, in with
27:01
my colleagues is the concept of generative UI.
27:04
Even the UI is generative. You know, let me first tell you
27:07
what this means. It's like I'm talking to an
27:10
agent. I'm having a conversation.
27:13
Instead of just replying to the conversation, you always present
27:16
me some UI elements. Sometimes it might be, you know,
27:19
show me some items, some Carusos, sometimes, you know,
27:23
related item and so on. And these UI components can be
27:27
dynamic. So the agent needs to decide
27:30
based on my user, based on my previous message.
27:33
Maybe, you know, with the design team, I built 10 UI components.
27:36
And then based on the user request, sometimes you get
27:38
similar dishes or you know, similar items or you know,
27:42
sometimes item Caruso, sometime something else.
27:45
And what we are seeing, we still need to trust it more.
27:48
But what we are seeing is that this experience, because you
27:51
know, we buying is a very visual experience.
27:54
You know, conversation is very limiting.
27:56
You know, I just want to sometimes, you know, even me as
27:58
a user, I want to scroll, I want to click, I want to swipe left,
28:02
right. I make decision about what I
28:05
want to eat based on the image. You know, this image makes me
28:07
feel hungry. So I wanted so.
28:11
But you're dynamically creating these different widgets on the
28:17
fly only depending on the input that I give you from chat or
28:23
also if I click on something then you can show me more like
28:27
that both both. That's a great point it.
28:29
Feels like that is very mix of traditional machine learning and
28:34
new agents in a way because if I'm clicking on something that's
28:38
a recommender system problem. Yep, Yep, totally.
28:42
If you're clicking on something that's a recommended system
28:45
problem, you know, the traditional world is still
28:47
applies. And I think that helps in the,
28:49
you know, that's how TikTok, for example, you know, you Tik Toks
28:53
you, you're looking at these feeds, you'll swipe left, right,
28:55
and the recommendation algorithm gets better.
28:58
But if you think about, you know, again, if you think about
29:00
this movie Iron Man Jarvis, Jarvis is an assistant who's
29:04
watching you. What you're doing in that
29:06
environment, it feels so natural.
29:08
So we have built, you know, it's still not live, but you know,
29:11
it's the proof of concept where we build an agent who who's
29:16
watching the screen. O it's not just answering you,
29:19
but it's also watching your actions on the screen.
29:21
And then it talks back to you depending on those actions.
29:24
So it whenever you add an item to the cart, it says so a good
29:27
choice or you know, I knew that you would like this item or
29:31
something like that. And that interaction feels so
29:33
much more natural than you saying everything based on, you
29:37
know, typing chat where you know it's just watching you.
29:40
So that's. It's almost like you having to
29:42
suck that idea out of your mind and put it into the interface is
29:48
a lot of friction. Yep and if the agent can just
29:51
watch you scroll and click then it can be there with you and
29:55
it's much more of a Co piloted experience.
29:57
Yep, and I've seen that in action.
29:59
We don't have a product live yet, but this is something, this
30:01
is an idea we are playing with, right?
30:02
Now, but also the user adoption problem there is, I imagine
30:05
you're going to get a lot of folks that are like, I don't
30:08
want you watching everything that I look at.
30:10
Yep, Yep, Yep. So there's that trade.
30:13
Off, yeah. So we have, we have, we have not
30:15
cracked it yet totally. You're right.
30:17
So this is something which we are exploring and every user is
30:19
different. So you need to.
30:21
I don't think there's a silver bullet that I can share.
30:23
You need to. There are people that are fully
30:25
OK with that. It's like.
30:26
I'm fully OK with that. My, you know, the way I think
30:29
about, you know, data privacy, of course it's important, but
30:33
you know, my mentality is that take my data, if you can give me
30:37
value. So if I see value in return, I'm
30:40
happy to give my data. So and different people think
30:43
differently. So last learning on UI before we
30:49
move to next topic is contextual.
30:52
So being contextual, so often we we try to build a chat board,
30:57
which is like the solution for everything, right?
31:00
So we give the data as an interface.
31:02
But one thing that we have seen much more useful is, and this
31:06
creates a lot of friction because it's a new interface,
31:08
you don't know the capabilities and so on.
31:10
You're used to regular interface.
31:12
So one thing that we have worked better is that imagine your
31:15
regular UI and imagine you know, there's a, maybe there's a
31:18
floating button there. And depending on what you're
31:22
doing, maybe you spend 5 minutes looking for things and you know,
31:25
looking for items. It pops up at the right time,
31:28
contextual, and it helps you with a very narrow task, very
31:32
micro task. I'll give you an example.
31:34
Maybe I want to buy a headphone and I'm looking at a headphone
31:38
and it pops up, it says, do you want to compare this headphone
31:42
with the latest headphone from Apple?
31:44
It's amazing. That's an aha moment for me.
31:46
If that happens, I would click on that and you you don't need
31:50
an entire conversation. This opens and the comparison
31:53
happens using an LLM. And by the way, that's a very
31:56
simple it's you don't need agent for that.
31:57
It's a simple LLM call connected with tool.
31:59
Well, I guess the hard part is if you want to put it into a
32:02
table, dynamically creating that table and what?
32:05
This checks this box. This doesn't check this box.
32:08
Especially on a mobile screen, that doesn't look good.
32:10
But you know, you know, we have designers and, you know, there
32:13
are different design ways to solve it.
32:15
Yeah, there are different design ways to solve it.
32:17
But the point is, instead of having you know just a chat bot
32:21
that does everything and user has no idea what about its
32:24
capability. If you define some micro job to
32:27
be done and you help to the help the users at the right time, we
32:32
in our experiment we find that much more effective.
32:35
Maybe real fast we can detour into how you make sure it's
32:40
coming up at the right time and giving you the right
32:42
suggestions. This is where evals comes in.
32:46
Perfect segue. Yeah, this is where evals come
32:48
in. This, this is again, this you
32:51
can compare it with the traditional world of push
32:53
notification. So all of us get these
32:55
notifications from different apps at different times, you
32:58
know, You know, I use Domino's app, I like pizza and Domino
33:03
sends me notification. And a lot of times the
33:05
notifications are bad. So, and if the first few
33:08
notifications are bad, then you stop either either you silence
33:11
them or you know, even if you receive it, your brain doesn't
33:15
process it because you know it's bad.
33:17
If you get a good notification, if you start with on a good, if
33:22
you have a good start where the notification actually, maybe
33:25
it's 1:30 PM and I'm in a meeting and I'm really hungry.
33:28
And, you know, the app knows that I'm vegetarian and it pops
33:33
up that you know this. It knows that I like falafel.
33:35
That's the context. So if it pops up at 1:30 PM that
33:39
you know, Nishi, you, you haven't had your lunch.
33:40
I noticed that you haven't had your lunch.
33:42
I don't know how it will know it but.
33:43
You haven't. You didn't order anything from
33:45
our app, so we're assuming you haven't had your lunch.
33:47
Yeah. So, but that's, that's the
33:49
thing, that's the notification, which I would love.
33:52
So again, I think this has been, this is an old problem.
33:55
It applies to push notifications, it applies to
33:57
other things. And now it's still true in the
34:00
world of LLM where if we pop up contextually, and again, the
34:05
idea here is to AB test and try. So the idea again, the idea is
34:09
that don't do too much, don't overdo it, don't send 10
34:12
messages, but you can. So if we take food as a context,
34:17
so there's lunch, dinner. So breakfast, lunch and dinner,
34:21
maybe you optimize, you say that you know at the right time when
34:25
it's lunchtime, you pop up something and then you use the
34:29
profile of the person to whatever information you have
34:32
about the person to write something which the person can
34:35
relate to. And again, this is where LLM,
34:38
you know, we have done some experiments with the LLM and it
34:41
does well. But again, you know, no silver
34:43
bullet. These are few ideas which we are
34:45
trying that don't be, don't overdo it.
34:48
And whenever you do it, make it personal.
34:51
You know, people say don't make it personal, I say make it
34:54
personal. I could see how you get people
34:59
that are in my case, for example, I'm checking out on one
35:04
of the apps, looking at some food.
35:07
And at this particular restaurant, I've already added a
35:10
few things to my basket and it says, oh, hey, I know you like
35:13
vanilla milkshakes. These guys make an incredible
35:17
vanilla milkshake and that pops up at just the right time.
35:19
And do you want to add that to your basket?
35:22
So again, it just reminds me of like traditional recommender
35:25
system in a new way. And so it's almost like both
35:29
with search and with this totally you're doing these old
35:34
tasks that we did in Predictive ML, you're just adding an extra
35:39
layer on top of it to make it even more personalized and even
35:44
better ideally. We talk a lot about this
35:46
internally that the problems are still the same, right.
35:49
So we are still recommend Netflix organize this
35:51
recommendation system competition, I don't know 20
35:53
years from now, 10 years from now.
35:55
So we still talk about recommendation, we still talk
35:57
about search, and recommendation is not completely solved.
36:00
Search is not completely solved sending notification.
36:03
So it's the same problems, but now the toys are different, the
36:06
tools are different, and those tools are helping solve the same
36:10
problem in a better way. OK.
36:12
Going back to evals, what is your take on those?
36:16
So when I think about evals, so we we talk, you know, the
36:21
process is also an investment firm.
36:22
So I speak to a lot of founders and we talk a lot about mode,
36:26
you know, what is the mode of the product?
36:27
How do you differentiate your product?
36:29
And you know, a lot of time, you know, you see people say that,
36:32
you know, people don't share their system prompt.
36:34
You know, system prompt is mode. And you know, I understand that
36:36
you spend a lot of time to engineer that, but I believe in
36:41
eval so much that I think, you know, and I'm not a founder, but
36:45
if I would be a founder and if my system prompt leaks, I would
36:48
not be worried. But if my evals leak, I believe
36:51
so much in evals, then I if I would be a founder someday, I
36:54
would say that evals is the real mode of your product and not
36:58
your system prompt. A lot of times when we launch
37:00
these products before we launch them, how do you know if it's
37:04
good enough, right. And again, this is the same
37:07
problem that you can think of in a regular software engineering
37:10
world where you build a software and software development life
37:15
cycle is much mature. Now where you know, there are,
37:18
there's quality assurance, there's a field, an entire field
37:21
quality assurance testing, testing in production,
37:24
regression test and so on. So there's this entire thing and
37:27
now with AI applications and now when the entire product is about
37:31
AI, we come to this new world where and especially now LLMS
37:35
are non deterministic. So it's not your regular AI.
37:38
So how do you, how do you know if something is good enough?
37:41
And we spend a lot of time, we debate a lot.
37:43
So there are, you know, people in the team who disagree and,
37:46
you know, everyone is passionate and, you know, someone says
37:48
that, OK, we did not think about this case.
37:51
It doesn't work. Other people say that, OK, maybe
37:53
you're being too particular. It works.
37:55
It generally works 80% of the time.
37:57
So the answer to that is eval. So you can do that in a
38:00
systematic way through evals. Where the idea is that you build
38:05
a system to check evals is nothing for me, but a system to
38:08
check if what you build is good enough that the first case do
38:13
you launch it and second once you launch it, once you start
38:17
getting real user queries because it can also degrade.
38:21
Users can also ask different things which you did.
38:24
We are not prepared for evals. Is that the traffic light that
38:27
keeps you informed so that you can?
38:30
So there's a part of it which is pre development during
38:32
development and there's a part of it which happens like in
38:35
production. Yeah, I've heard it explained as
38:38
again going back to like traditional predictive ML, you
38:41
have offline, online, there's like the offline training batch
38:45
jobs type thing and then there's the online like boom, real time
38:50
type of predictions. So few mistakes which I have
38:54
made myself and I see a lot of people make, so I'll talk about
38:56
that in eval 1. I think it's a mistake to wait
39:02
for your product to be launched to write your eval's because you
39:07
know you want real user data, right?
39:08
So eval's need data. So imagine you have a chat bot
39:11
and you need to see what how people are using it in the wild.
39:15
But it's too late already because you launch it maybe your
39:19
chat bot sucks and you will know it much later.
39:22
So, So this is where we do a lot of simulation synthetic data
39:27
generation where before it goes live and it doesn't have to be
39:32
again, it can be simple. It's like, you know, you get
39:34
together in the team and everyone is playing with the
39:36
chatbot and you create evals based on that.
39:39
Or you use another LLM where you know, maybe you are this the
39:43
lone engineer in the team. You give it 10 queries and then
39:46
you use an LLM that you know, generate 100 more queries and
39:49
that's your synthetic data. Or you give it different
39:50
personas. So a lot of people, they wait
39:52
for, they wait a lot before evals.
39:54
But I think first step of evals is simple.
39:56
It's even a person in the room. Just, you know, firing queries.
40:00
You create a data set of 20-50 hundred and then that's a data
40:05
set. And then you see how your agent
40:07
responds and maybe you manually go through it.
40:10
And if you go through it manually, you know, in half an
40:12
hour you'll realise that OK, these are the scenarios where it
40:14
does OK. These are the scenarios where it
40:15
doesn't do good. That influences your thinking
40:18
that you know, what metrics should I, what metrics are
40:22
important for me? How should I use an LLM as a
40:25
judge? LLM as a judge is a popular
40:27
concept, right? For evals because human labels
40:30
are expensive, I'll assume the audience knows about it.
40:33
Another thing that you know, which I've noticed is that
40:37
people immediately run to LLM as a judge.
40:39
But again, there are a lot of low hanging fruits.
40:41
I'll give you an example. It's so LLM.
40:45
So these days, you know, we are a bit spoiled.
40:47
So LLM as a judge, let's just, you know, users.
40:49
And so you, you get the input for synthetic data, you get the
40:53
output from the agent. Now you ask an LLM, you prompted
40:56
a bit that OK, did this input satisfy the user intent?
40:59
And it's easy, you can do that. And immediately people run to
41:02
that because it's easy. But there's something which is
41:04
even easier, which a lot of people miss is a lot of time you
41:08
have more deterministic because you never know that maybe the
41:12
LLM itself is making mistake, right?
41:14
And then you have the final label, which is a mistake.
41:17
But a lot of time there are more deterministic metrics.
41:20
So I'll give you an example from food ordering.
41:22
So imagine a food ordering experience, user comes in, you
41:25
have the agent conversation. In the end, if the user
41:29
generates a cart and usually we send a link to the cart, then
41:34
that's that means that conversation was amazing, it was
41:37
positive. Right, they bought something.
41:39
They bought something. So the final metric that you're
41:42
looking at is did they convert or not?
41:44
Did they convert? But even, you know, it's getting
41:47
a little bit into the details, but you know, sometimes people
41:49
come to the cart stage but they still don't order.
41:51
So there's a so so. Not.
41:53
Always, yeah. But hey, but I'm saying that was
41:55
there a car? If you link, if you think about
41:57
a funnel that you know a user making search, a user going
42:00
couple of steps and if you'll think about a later stage in the
42:03
funnel, could be about making order, could be about adding to
42:06
the cart, could be about something else depending on what
42:08
the use case is. But those are positive signals
42:11
that this conversation actually went through different stages of
42:14
funnel. And it's a deterministic metric
42:16
because I can see that, OK, I have these hundred
42:19
conversations, which of these conversations went to through
42:21
this stage of funnel and that's a positive conversation.
42:24
So grab that for the evals. Grab that for the evals and that
42:28
is so much better than using because LLM as a judge will make
42:31
mistakes. Look for those metrics, whatever
42:34
those metrics suffer and don't don't run immediately to LLM as
42:37
a judge. There's a step before that.
42:39
Another thing I want to talk about is so if you again, if you
42:42
look on Internet and if you see advice from people out there on
42:45
evals, lot of good advice, but it very soon it gets complex
42:49
where people talk about, you know, conversation level
42:54
analysis, single turn, multiple turn analysis.
42:59
The silent failures on the. Silent failure.
43:01
There are eval frameworks tool calling.
43:04
You know, did we call the right tool?
43:07
Did we call the? Did we call the tool with the
43:09
right parameters? Did if you have multiple tools,
43:13
did you go, if you can even do a state management?
43:16
Because maybe a user query calls 5 tools so you can even map that
43:20
OK from tool one to tool 2 did the maximum mistakes.
43:23
So it can very easily blow up and you'll be kind of paralysis
43:28
analysis. You'll be stretching your hair
43:30
often. You know, we, we present these
43:32
evals to business folks and imagine, you know, you have a
43:34
meeting and you know, you have these 12 different people get
43:37
lost and you know, you lose the audience, you lose everything.
43:40
So for me, you know, you know, for us what we have seen when we
43:43
build product set process, it's like the first level of eval is
43:46
very simple. It's like this is the query from
43:49
the user. This is the entire conversation.
43:52
You know, multiple queries, maybe there are 10 messages
43:54
exchanged. You use and maybe you tried the
43:57
usual metrics. Now you come to LLM as a judge
44:00
before you go to turn by turn analysis, before you even look
44:03
at what tools are called, just give the entire conversation to
44:06
an LLM and ask some basic questions.
44:09
Ask it that did it satisfy the user intent?
44:13
Did the conversation went in a direction where you were trying
44:17
to the agent was trying to close the order because you can in the
44:20
end, you care about making a sale.
44:23
And these are so simple and this business people, they relate to
44:26
it. And then an LLM will make a
44:28
judgement, you know, true, not true, partially true and so on.
44:31
And there's already so much information there.
44:34
If you now see that, OK, these are the things that work.
44:36
These are the things that don't work.
44:38
To be honest, a lot of time our eval stop there.
44:41
We don't because there's already so much information that there's
44:44
already so many things for us to fix based on what doesn't work
44:47
that we never go to level 2 because level 2 is, you know,
44:50
tools. Did you call?
44:51
It's? So there's, there's almost like
44:52
a hierarchy. Yep.
44:54
And in your eyes, the hierarchy is first and foremost there's
44:58
like looking at the metrics of did they try and actually make
45:04
the sale? Yeah.
45:05
Because that's our top level metric that we're trying to
45:08
affect the needle on. Yeah.
45:09
That's really all that matters at the end of the day.
45:12
So if the agent isn't closing, Yeah, then Yeah, totally.
45:16
You got to be who was it in the color of money?
45:19
What was that? Who's the actor that was in it?
45:22
You got to always be closing and you got to teach.
45:25
Yeah. Teach your agent.
45:26
Put that in the prompt. Yeah.
45:27
So. Yeah.
45:29
That's one way to look at it, but of course you know on a
45:31
lighter side. Does the agent follow answer
45:35
you? So if you were the user on the
45:37
other side, did the conversation went in a positive direction?
45:40
Or you know the agent asked for burger and you give them pizza.
45:44
So did it satisfy the user intent?
45:46
Did it go in the right direction?
45:48
So, so there's, those are like very strong evals that you need
45:52
to be focusing on 1st and then if you need to, then you can
45:56
start peeling back the layers like did it call the right
45:58
tools? Did it use the right parameters
46:00
when it called the right tools? Yep, Yep, I can see that.
46:04
Yep. It's almost like there's an 8020
46:07
here. Yep, that Breo's principle, Yep,
46:10
on we've got these emails, they're going to give us 80% of
46:14
the important stuff. Yep.
46:16
And it's only these two evals that we got to look at.
46:19
Yep. Totally.
46:20
So that has been our experience. So we also made this mistake
46:23
where we already got into complicated evals.
46:26
It doesn't resonate with business.
46:28
It leads to analysis paralysis. So these days, you know,
46:31
whenever we do evals, we ask it practical business questions 1st
46:35
and then there's already a lot of insights there to act on and
46:38
only then there is value in going deeper.
46:41
Yeah, but it's not as actionable, I imagine.
46:44
Yeah. And like you say, analysis
46:46
paralysis where you're like, there's so much that we need to
46:48
do here. So coming back and saying, is it
46:52
answering the user's question? Is the user experience nice?
46:55
Yeah. Is the metric being moved like
46:59
the metric we care about? Is it trying to move that?
47:02
Yep, Yep, those are very, very basic and very useful.
47:08
Yep, there are two more lessons when it comes to evals. 1 is
47:14
about so at process something we do, this is a recipe that we
47:17
have tried many times and it has been very successful is
47:20
labelling party. So we call it labelling party
47:23
where you know we invite, you invite your team, you invite
47:27
some stakeholders, it's important to invite you know
47:29
business folks as well. You get 15 people in the room,
47:33
you order some pizza. That's why it's a party.
47:35
It's important you book 1 1/2 hours, you spend time with these
47:39
people to actually go through real conversation.
47:43
And then this is what the agent responded.
47:45
Now if I'm imagine and then the question you ask them is that
47:49
imagine you are the user on the other side.
47:52
And then you ask a couple of these questions that we did the
47:54
conversation go in the right direction?
47:57
Did it try to close the order and a couple of other things.
48:02
Did it kind of break any guard rails or whatever?
48:05
It depends on, you know, what the agent is trying to do.
48:07
So you define a couple of questions, easy questions.
48:10
So don't talk. Don't talk about technical stuff
48:12
like tool calling and so on, business related questions.
48:16
And what what would come from that is that you spend 1 1/2
48:20
hours and at the end of 1 1/2 hour you'll have, I don't know,
48:23
every person will label 10 data points.
48:26
You'll have 150 data points, and when you look at that 150 data
48:29
points and now run your LLM as a judge the prompt that you wrote
48:32
and ask it to label these 150 data points, you'll see that
48:35
there's a difference between what LLM says as human.
48:38
Obviously there'll always be a. Difference human label data and
48:40
the LLM is a judge. And then you will see that
48:44
depending on your use case, sometimes the LLM will be more
48:46
lenient. Maybe the LLM always said that
48:48
your agent is perfect, or maybe the sometimes the LLM would be
48:52
stricter. So depending on the use case,
48:54
I've seen both. But now when you see the
48:58
difference where the discrepancy is between human and LLM label,
49:01
you look at it again manually. All this is manual.
49:04
And then the way to use this data is you use this discrepancy
49:08
to inform yourself that OK, this is where the LLM is making
49:11
mistake. You know, human is right here.
49:13
Now go back to your LLM as a judge prompt.
49:16
Give it examples. So few short labeling, give it
49:18
example that when this question comes in, this is what you do.
49:22
So I think often we also use, we take think LLM as a judge very
49:27
easy. We just write a prompt, we
49:29
expect magic to happen and get something, but our experiences
49:32
you need to iterate on that prompt.
49:34
And we're going full circle. It's context engineering.
49:37
Yeah, yeah, it's. Context, engineering, all these
49:39
things are related, but you need to improve your LLM as a judge.
49:42
So we spend a lot of, so we do these labeling parties, we
49:45
improve the prompt, we give it few short examples based on
49:47
whether this and now you're LLM as a judge, you run it again.
49:51
We see that OK, now it's closer. Earlier it they mismatched, I
49:54
don't know, 30% of times. Now the mismatch is 15%.
49:58
And again you need to keep doing it every two weeks do a labeling
50:00
party. So that is a recipe that we have
50:02
tried and it's been very successful, so.
50:04
Yeah, that's an actionable insight, Yeah.
50:06
And it's a manual process and, but team building it's yeah,
50:10
it's, it's rewarding. It's it's nice as as the
50:13
builders of the system to, you know, sometimes times look into
50:15
the data yourself and. And hear from the business side
50:18
of the house, I imagine, Hey, this makes no sense.
50:22
Yep. Why would we hear about this or
50:24
why? Why would the LLM say this?
50:26
Yeah, you discover a lot of other things which you know you
50:29
were not even maybe hoping to hear, but that happens.
50:32
Last point there, it's actually kind of it's related with
50:38
labeling party. So when you do a labelling
50:42
party, how do you offer? It's very important how you
50:45
offer people, how they can look at the conversation.
50:48
And this is where, for example, we use a lot of these external
50:51
tools, Langsmith Lang Fuse. So there are these observability
50:54
tools where it captures the interaction.
50:56
But at the same time, if you're looking at Langsmith Trace, it's
51:00
very user unfriendly. It's like, you know, bunch of
51:03
Jason's. If you have multiple tool calls,
51:05
it's like, you know, I love looking at it.
51:07
You know, I'm a technical person, but the business person
51:09
gets lost. So one thing again which we do
51:12
which works very well is with now with very low effort by
51:15
using V-0, Versal and so on, you create custom annotation tool.
51:20
There are standard tools as well in market for labeling data like
51:23
Label Studio. There are the tools out there,
51:25
they're good. But you know, our learning is
51:27
that every use case is different.
51:29
You need to visualize different things for the user.
51:32
I'll give you an example. If I'm looking at food
51:35
conversation or shopping assistant conversation, if I'm
51:39
the user who's evaluating it, aside from the conversation, I
51:41
also need to see that, OK, these are the items which were shown
51:44
to the user at that point, these items and I need to show it, see
51:48
it visually. Otherwise, how will I judge
51:50
these items are from these restaurants?
51:52
Maybe these restaurants are open or closed at that time.
51:55
Because if you don't give me that information.
51:57
So annotation tool plays a big role.
52:00
And what we find ourselves doing a lot is, you know, with a
52:03
couple of it's like hacking. We just quickly put together a
52:06
tool in half a day using V-0 Versal, and people love using
52:10
it. And then you get the audience is
52:13
much more involved than going through a trace in Langsmith.
52:15
So again, it's a small point, but it makes a big difference.
52:18
User experience. Yeah, user experience.
52:20
And then you are circle.