Latest / Agentic Conversations (formally mlops.community) / Real LLM Success Stories: How They Actually Work // Alex Strick van Linschoten // #287

Transcript

Real LLM Success Stories: How They Actually Work // Alex Strick van Linschoten // #287 · Agentic Conversations (formally mlops.community)

0:00
My name is Alex Strickland Linskreyton.
0:02
I'm a machine learning engineer at Zanamel and I don't drink any
0:07
coffee. I drink green tea.
0:11
Not always, but yeah, a good cup of Jasmine green tea or.
0:15
Good people of the universe, welcome back to the ML OPS
0:18
Community Podcast Today. We've got a very special episode
0:21
going over the database of real world LLM use cases.
0:28
That's right, there's all kinds of AI use cases that Alex
0:33
consolidated into one place, and I appreciate him doing that for
0:38
us. We get to the conversation about
0:41
what exactly he learned while putting together this massive
0:45
database and how he did it. So without further ado, let's
0:49
get into it and if you are one of those special people that
0:53
listens to this episode on a podcast player, I've got some
1:00
recommendations for your music, listening sessions and the
1:05
algorithm. On YouTube you can find such a
1:09
great gem of Bob Dylan and Van Morrison playing the classic
1:14
song Crazy Love in front of the Acropolis in Athens of 1989.
1:28
I can hear her hard to beat from 1000
2:32
time I've ever seen this well done what you've done is when
2:38
the folks that evidently AI put together a gigantic database of
2:44
different ways that ML and AI is being used.
2:48
And they took a lot of disparate data sources from blog posts
2:54
that have been out there and folks that they've talked to and
2:59
probably some people that are using their open source client.
3:03
You all did something similar at Zen ML and you put together a
3:07
database, but for LLM specifically and, and people
3:11
that are using LLMS in production.
3:14
Can you explain how you went about it and what a huge
3:17
undertaking? Sure, Yeah.
3:19
And I'm so glad you brought up the that evidently databases,
3:22
they have two databases I think. And they were totally, that was
3:26
totally an inspiration for us. I was just like, yeah, there's
3:29
all of this stuff going on and you see like people posting
3:32
these little blogs occasionally or just like random things or
3:35
like obviously like all of the conversations you're hosting
3:38
with ML OPS community as well, just like this rich source of
3:42
data. And we're all sort of trying to
3:45
figure out like, exactly how does this stuff work in
3:48
production? Like what are the, what's the
3:51
spectrum from like mega company down to like 5 or 6 people just
3:56
trying to like start something, start something new.
4:00
So, yeah, I'm my background, I used to be a historian and so
4:05
my, I kind of like, I don't know, a hamster, not a hamster,
4:10
a squirrel, like just like hoarding, like all of these
4:13
things. As I'd have been like keeping a
4:15
list of all of these links as as I went through.
4:18
And at a certain point, like you reach, yeah, the point where
4:23
like it makes sense to put this out there.
4:26
And yeah, obviously, I mean, we can talk about the details, but
4:29
like there's summaries for each of the posts, which would have
4:33
been impossible to do manually for for me, or at least like
4:37
without like a mega budget and like LLMS themselves, like
4:43
helped make all of that stuff stuff easier.
4:47
So yeah, it, it, it was a big into undertaking in the sense
4:50
that it's not like someone else had done the work of collecting
4:55
them together. But in terms of overall time, it
4:57
didn't take that long to to to put together.
5:02
Yeah, and now there's one place I, I know, at least for myself,
5:05
I always try to add any quality blog posts that I'll see out
5:12
there on the Internet to the MLF community newsletter that goes
5:16
out as like a hidden gem. And it's true that it goes out
5:23
as a hidden gem and then maybe if you missed that week, you
5:29
missed it for good. But here you have it and you can
5:34
always go back to it and reference it.
5:36
And I really like what you said is how it tries to cover the
5:39
gambit of small teams, what they're doing to large
5:44
enterprises, what they're doing. And did you notice any types of
5:50
repetitive use cases? Because the other thing that's a
5:55
little bit new here is how varied the use cases are.
6:00
It's not with the traditional ML where we kind of have it figured
6:04
out and we know that there's going to be some fraud
6:07
detection, there's going to be some recommender systems, maybe
6:09
there's some loan scoring or some classification.
6:12
But with LLMS, it's wild, Wild West on how you are getting
6:19
value for your company and there's whole task forces that
6:22
enterprises that are trying to figure that out right now.
6:26
Yeah. I mean, variety is definitely
6:27
the word. There are a tonne of different
6:30
use cases. I think there's kind of 2 broad
6:33
categories and one is probably maybe much bigger.
6:37
The bigger 1 is like let's go with something that we see
6:41
everyone else doing so let's build a chat bot and the chat
6:45
bot is either like customer service or it's like chat with
6:48
your data. These are broadly speaking, like
6:51
the two most and these can come in different flavours.
6:54
Some of them have like an A gentic, you know, colour to it.
7:01
Other ones yeah, are are are like completely internal other
7:06
ones with customer, customer facing and so on.
7:08
Those are kind of the yeah, some CEO has seen like some other
7:13
company has done this or someone internally has built like this
7:18
POC demo on like a Streamlit app or whatever, which looks crazy
7:22
impressive like let's roll it out.
7:24
It's already working right? And so you.
7:27
See. Little did they know.
7:28
Those were kind of the the the the most common ones.
7:33
And then you have like a smaller cluster of, of things which are
7:38
either, yeah, which are just like, I don't know, I didn't
7:41
know whether there's any unifying threat thread apart
7:43
from the fact that it's like, it's like the companies which
7:47
are really like pushing the envelope technically they're
7:50
really driving like innovation. They're figuring out like stuff.
7:54
Think, I don't know, think what Copilot was doing like a few
7:59
years back, like really like driving a path onto themselves
8:03
or, or it's some companies doing some like innovative stuff with
8:09
perhaps with agents now or mixing stuff around content
8:12
generation or whatever, which you can't really categorise.
8:16
And probably maybe there aren't like 1000 other companies who
8:20
would want to do this, but like it really works for them.
8:24
And so, yeah, but I mean, it takes a certain kind of company
8:28
and a certain amount of like, risk appetites for you to get
8:32
into. Like I'm just going to go and do
8:34
my own thing even though everyone else is doing chat
8:37
bots. Yeah, Yeah.
8:40
It's easy to do the chat bots because it's it's working for
8:45
the majority of folks or at least it's better than what we
8:48
are used to as far as chat bots go.
8:50
And so why not? And I like that you show there's
8:55
different flavours of chat bots. Maybe it's agentic, maybe it's
8:59
internal, maybe it is just externally facing and it's folks
9:05
that are hacking around on something or it's a full on
9:10
rolled out project that got buy in from leadership.
9:14
And so you see every type of flavour when you get blog posts
9:20
about it and when people write about it or talk about it or
9:23
come on the podcast and share. And I wonder a bit how there's
9:31
got to be a lot that is still obfuscated because when I think
9:39
about a company like like an Uber, they have so many use
9:44
cases that they aren't able to talk about all of them, right?
9:49
And they're not able to show every single way that just
9:53
random departments are using LMS and, and what they're doing with
9:57
it. And it makes me think a lot
10:00
about how we've had traditionally like the data
10:06
governance role and now the AI governance role is just such a
10:11
beast. Because in, in enterprise with
10:15
1000 plus people or God forbid, like a 10,000 people.
10:18
Imagine how many different instances of AI they're doing
10:24
and how many repetitive licences they're paying for a repetitive
10:28
workflows. And that just, yeah, it gives me
10:32
anxiety just thinking about that.
10:35
Yeah, I mean, certainly you know, the things that the things
10:39
that people are putting out, particularly in Uber or or
10:41
whatever, generally speaking, people are putting things out
10:43
which makes them look good or make their teams look good.
10:46
Even if it's a failure, it's like we caught the failure,
10:48
right? Or we'll had good processes to
10:50
to catch the failure. And yeah, lots of respect for
10:55
companies that like includes details of like where they mess
11:00
something up. 11 which comes to mind is like weights and biases
11:05
who develop their like internal sorry support chatbot.
11:10
And they'd be really great at like building in public and
11:13
sharing stuff on how they build their evals and so on.
11:17
They shared like, Oh yeah, we got something wrong about like
11:20
how we did our evals and we needed to spend like several
11:23
thousand like dollars just like re redoing everything because we
11:26
made a mistake or whatever. So like, yeah, it's nice when
11:29
people like put actual money. But yeah, for it would be great
11:35
if there was a bit more normalisation of like sharing
11:37
failures and pods which didn't work.
11:40
I guess there's a shared internally and maybe for a mega
11:44
corporation that is good that that that's the way, you know,
11:47
that's a good thing for them. And yeah, maybe it's expected.
11:50
But yeah, obviously it would be nice to to to see all the
11:56
remarks which didn't didn't work out along the way.
12:00
Yeah. Were there any other patterns
12:03
that you saw as you were putting this together?
12:05
And whether it goes towards common design patterns or how
12:11
folks or the majority of use cases are doing evaluation this
12:16
way or that type of thing that you noticed after reading so
12:19
many, you're like, well, this seems to be the flavour of the
12:22
day, Maybe it is the most useful.
12:25
Yeah, so, so I mean, yeah, lots of lots of, I guess smaller
12:30
insights rather than big insights.
12:32
I mean, I guess if there was a big insight, it's like all of
12:36
the tried and tested things that we know about like software
12:40
engineering, DevOps, all of these kinds of principles, all
12:42
of that stuff is super important.
12:45
And you better get all of that stuff right.
12:48
Otherwise this magic that you're building on top is, is not going
12:52
to work or it's not going to work reliably.
12:55
And you know, we thought a lot, a lot about like exactly what to
13:00
call this database the moment we settled for LLM OPS because this
13:03
is kind of what the community is seems to be settling for.
13:08
Microsoft is, we're trying really hard to push this term
13:11
Gen AI OPS, which is just like too many syllables and no one
13:16
else is using it so. Somebody told me I may get
13:20
cancelled for saying this, but somebody came on here the other
13:23
day and said, Oh yeah, we're doing Gen OPS.
13:26
And I was like, that sounds like you're missing you're, you're
13:30
pretty progressive there, huh? Like is there some?
13:36
It just made me be like Gen OPS. Doesn't sound like what I think
13:40
you want it to mean. We have this term LLM OPS, which
13:50
is, is sort of, I guess like the, the, the ways to think
13:54
about and the ways to do all of this stuff we're doing around
13:57
Gen AI. And to be honest, like most of
13:59
it is around LLMS. It's not still not in the video domain.
14:03
It's they're starting to do people trying to do things with
14:06
multi modality, but like that's still a little bit like future
14:11
facing and and same with same with image generation.
14:19
But, but really like a lot of the stuff which underpins this
14:22
is like is ML OPS And, and even, you know, as I'm sure you're
14:27
very well aware, there's like still a lot of people who say
14:29
like ML OPS is not a thing. It's just dev OPS and you know,
14:34
you can go all the way back down.
14:35
It's why I kind of wanted to say like software engineering best
14:38
practises, which a lot of this is.
14:42
So that's, that's, that's kind of one thing like the, the, the
14:45
fundamentals still really matter.
14:49
And what were some of those smaller insights?
14:52
Kind of I'm, I'm very interested in tracking like the extent to
14:55
which people are actually using this in production every year.
14:59
It's like this is going to be the year of the agent.
15:01
I hear this now about 2025 and you put on a great conference
15:08
recently around this and some of the use cases and it's still
15:14
relatively, I mean, there are some success stories of like
15:16
companies who are doing things, but quite often there's not
15:20
enough technical detail to know. It's not like you're just like
15:24
unleashing customers workflows just like completely unbounded,
15:28
like let the agents. It's like, no, everything has
15:31
been super constrained down like as much as possible.
15:35
And so, yeah, it, it does seem that we're, we're not quite
15:41
there with making agents work reliably.
15:45
And it's, yeah, it's unclear to me not working on a, on a tonne
15:50
of these projects, like exactly where, where the bottleneck is
15:54
in terms of in terms of that. But the places where people are
15:57
managing to, to get this kind of to work is as I said, like
16:03
really, really constraining down exactly the specific tasks or
16:07
specifications for, for agents. Klarna had a kind of huge, huge
16:12
win there where they, they amplified or like supported
16:16
their like customer service agents.
16:18
I think they calculated that they they would like gain, what
16:22
was it like thirty, $40 million in profit on the basis of this
16:26
deployment. They reduce the time of, of, of
16:29
customers, whatever in in waiting lines and people didn't
16:32
come back and all of these kinds of things.
16:35
It's a mega company be from from what you could tell from what
16:40
they released, it's like, yeah, it was a very narrow realm and
16:45
they could kind of control it. And see, they are filing to go
16:49
public so that can boost their stock price when they IPOI think
16:57
that that felt to me like one of the most outlandish claims of
17:01
2024 was when the CEO came out and said like, we're we don't
17:07
need to hire sevens of them because our AI or something like
17:10
that. And it was one of those where
17:14
you read between the lines and you recognise.
17:16
Oh yeah, he's just getting ready to go public.
17:20
He's probably on a roadshow right now.
17:21
And good on them. Let that stock.
17:25
Pop. On a one, but there is it.
17:30
It's funny you mentioned Klarna specifically because of that and
17:34
the support use case, because that feels like the one that is
17:39
most defined when it comes to agents and all the other ones.
17:44
What I've seen is, yeah, we're really trying to figure it out
17:48
still. And you can't just say go be my
17:53
marketing team, right? Or go do my marketing for my
17:57
startup. What you have to do is go deep,
18:01
deep, deep down into one specific task and then try and
18:06
automate it in a way that is possible that you know the steps
18:10
for. And so you can say, all right,
18:13
go and collect all of the keywords that my competitors are
18:19
using a whole pay per click campaign around and then analyse
18:24
which ones I am interested in. Also bidding on that type of
18:29
thing is great for an agent. It's not going do marketing for
18:33
me, right? Because the more vague you are,
18:37
the less that you're going to get the outcome that you want.
18:40
And so that probably is the hardest thing right now with
18:44
agents. And then when you see them
18:48
trying to be used like the the OLX magic use case that we had
18:52
at the agents in production where they're trying to reinvent
18:56
this way that we're doing search inside of their app.
18:59
And so you just, you don't necessarily need to search for
19:04
anything specific. You just say, oh, I'm looking
19:06
for a baby stroller. And it will give you a few
19:09
options, but it will try to be a bit more agentic in the way that
19:15
it presents you these options instead of just giving you the
19:21
the ads or the classifieds of people that are advertising
19:26
their strollers. You can get the information and
19:30
then start honing in on it. I still am not clear though.
19:36
Do we want to be using chat for this and the interface of me
19:43
having to explain exactly what I'm looking for verse me being
19:49
able to click around in a recommender system.
19:53
And so I think what OLX magic is doing that's cool is they're
19:57
trying to combine both and say, all right, we're bringing you
20:01
these first searches or these first hits, but then we're also
20:07
adding a recommender system on top of that so that we can learn
20:12
from where you're clicking and where you're going off of it.
20:15
And so it's. Not throwing out the old just
20:18
because there's new, and thinking really creatively at
20:22
how to layer the two on top of each other.
20:25
Yeah, I think, I think in the database there's also something
20:29
which is somewhat underrepresented maybe because
20:32
the kinds of people who write these blogs are like the
20:34
technical team or like the the software engineers kind of the
20:39
more a little bit more on the back end side.
20:41
But like, you're totally right, like UX innovation is, is also
20:46
super neat and people to like experiment round and and a lot
20:50
of things which are often presented as chat, chat
20:54
interface don't need to be, but they can still, you know, use
20:58
alarms under the hood. It just can be like a button.
21:01
Why make me like type out all of this stuff or or interact
21:06
through voice. So, yeah, that, that, that's
21:09
totally something which yeah, I think even if all of the
21:15
innovation in in models like stops right now, we still have
21:19
like, yeah, a bunch of years like to figure out like new ways
21:24
of, of, of doing this stuff. Yeah, the interface is, is a
21:29
really fascinating one for me because we've got the pointer
21:35
and we've got or the cursor and we are used to clicking around
21:39
or We've also got different commands.
21:42
And so thinking about hotkeys is fascinating to me.
21:47
And then I am constantly referencing a talk from Linus
21:54
Lee from like almost a year ago, probably when he it was before
21:58
Notion AI had implemented their just five suggestions of what
22:05
you can do with Notion AI inside of Notion as you're writing.
22:10
And it's all click based, right click OPS in a way.
22:13
And that's really cool. But then, yeah, maybe there's
22:17
just select all and then you have some kind of a hot key that
22:23
you can add your voice to the writing or just clean up typos
22:28
or whatever it may be. Rewrite in a shorter way or
22:31
condense this thought or, or to bring up the box where you can
22:37
go back and forth with a chat. And so, yeah, all that is it
22:43
feels like we're still in the first inning on it.
22:45
Yeah, yeah. And I mean, that maybe takes to,
22:48
to another kind of lesson, which which came out of the database
22:51
was there's a, again, perhaps not surprisingly, but there's a
22:57
lot of people who are deferring to pre made.
23:03
I almost said pre made necessarily, but like safe
23:06
frameworks and like platforms around Gen AI, whether it's
23:13
bedrock or it's more specific stuff where, you know, AWS has
23:20
has done the the work of like making it super easy for someone
23:27
to create a chat bot based off company or enterprise data or
23:30
something like that. So yeah, I was, I was kind of
23:33
surprised to see how many people, I guess no one, as the
23:38
saying goes, no one got fired for like buying AWS or whatever
23:42
for for your enterprise company. But I worry in the in the lights
23:46
of what we were just talking about with the UX stuff.
23:48
If we go too quickly into the world of like a pre made like
23:54
framework with like relatively little flexibility that maybe we
23:57
don't get to discover like all of these.
24:00
Like. Different ways that like
24:02
customers could, could interact with our stuff.
24:04
Then we're just getting the, the, whatever, the five or 10
24:07
things that that you can get with Bedrock out-of-the-box.
24:11
That's such a great point. And then it all looks the same
24:14
and it's not like we were really excited about that whole
24:19
experience in the 1st place. So now we get just more chat
24:23
bots that we don't enjoy interacting with.
24:26
And yeah, I feel like I've seen that before, but the pattern has
24:31
happened already and so. But it's nice that like the the
24:36
open source side of that doesn't seem to like people don't seem
24:43
to be resting on their laurels too much.
24:45
I think of, I don't know, Lang chain or Lama index where
24:50
probably by this point they could just like stop like
24:55
breaking new ground as new technology come out and just
24:58
like, hey, we're just going to become like the Super stable,
25:00
like chat book guys. And so, yeah, to their credit,
25:06
they're like, they're still discovering things still, yeah,
25:11
they're still adding new, new ways of, of, of thinking about
25:14
like LLMS and, and, and Jenny, I, you know, even acknowledging
25:19
the, the, the many criticisms that people have of particularly
25:24
those two like it, it, it's still, yeah, that they haven't
25:28
fallen into the same trap, I feel.
25:31
One use case that I saw that was fascinating was when Philip from
25:36
Honeycomb came on here and he was talking about how they were
25:43
plugging in LLMS to their product itself and trying to
25:49
have the LLM almost be like a shepherd to help folks get to
25:54
doing something inside of the product that they knew would
26:00
convert them to a paying user. And I think that use case is
26:06
incredibly awesome, but very under seen.
26:11
Or I, I don't know many other companies that are trying to
26:18
innovate in that way. Maybe you saw others that said,
26:21
Oh yeah, we're going to plug in LMS as this guide or as this
26:26
shepherd or really as like our sales agent inside of the
26:30
product or a sales engineer, more like, more like it.
26:34
And from there, they're going to help the user become more
26:40
proficient at the product faster, so that they become a
26:44
power user and inevitably buy the product.
26:48
I mean, I think for the most part, most companies seemed to
26:57
like a little bit wary of like entrusting too much agency like
27:02
to that level to LMSI mean somehow chat bots are that
27:06
thing, right? It's like we can't give you like
27:09
full access to our support team, but like, hey, here's this like
27:13
robot which is there like 24/7 and you can knock yourself out.
27:18
The problem comes is like it's often seen as kind of like a
27:22
panacea or, or or like people don't like stick the details and
27:27
they're like then then you see like people getting frustrated
27:31
with a product. I mean, yeah, certainly.
27:33
I'm sure you you've tried out like random demos and random
27:37
things on a whole bunch of different places and like quite
27:40
quickly, like you realise, oh, it's not actually like doing,
27:46
doing what this thing is intended to be.
27:49
So either, yeah, either people are releasing these things and
27:54
those kind of middling, middling results.
27:56
And that's why you don't see in the logs that people write
27:59
about. It's not so much about like,
28:00
yeah, we made tonne of money out of this, but it's more focused
28:05
on the technical challenges. And then yeah, there's a tonne
28:08
where it's just like we, we built something internal because
28:13
it's way easier. Like we're way, way more
28:15
comfortable with the risk there where the people can find it
28:19
useful and they can kind of take it or leave it.
28:21
But in the end, they, they have to work for us or whatever, you
28:26
know, it's not like they can. And complain to everybody, yeah,
28:33
they're not going to make the stock price go down by writing a
28:37
horrible Reddit thread. So that you know what, as you
28:41
were saying that, one thing that jumped into my mind is how with
28:47
these support bots that we get, you have to inevitably think the
28:54
majority of them are powered by rag systems in some way, shape
28:58
or form. And I'm sure you have read more
29:01
rag blogs than you would like to admit when folks are setting up
29:08
their rags and they're giving context to.
29:12
So you've got the the chatbot that I as an end user am
29:15
interacting with. And then that goes to some kind
29:20
of a search system or it's retrieving stuff that you're
29:23
asking for. And it's also maybe trying to
29:28
come up with a solution. I wonder if people have
29:31
experimented with adding different signals of what I've
29:37
been doing inside of the app. So this may get a little, dare I
29:44
say the buzzword multimodal on us.
29:47
But if I'm clicking around on something and I have I love,
29:51
some people call it like rage clicks because something isn't
29:55
working or I'm trying to find some something.
29:58
The last thing I want is for me to talk to the support bot and
30:03
then it suggests that I do exactly what I've been doing.
30:07
Right. Well, then I have to explain
30:08
like the last 10 minutes of yeah.
30:10
Yeah, exactly. I don't want to have to tell you
30:12
that I just did these five steps.
30:15
I would like that you knew that and took that into account in
30:18
the answer. I don't know if you saw anybody
30:20
doing that because that feels like very cool, but it also
30:23
feels like it might just not be possible or invaluable.
30:27
I mean, for sure it's possible, but I, I didn't see any
30:32
specifically around like what were you doing in the website?
30:36
But certainly customer support lots were like enriched with
30:40
like customer data and the customer's previous context that
30:45
that would for sure was, was something that I could see.
30:47
And so I forget the precise examples, but like, just like
30:51
e-commerce that that was a really common, common thing.
30:55
And that like, yeah, the customer's profile, recent
30:58
things that they bought their preferences, but all of that
31:01
stuff was, was pretty regular to, to pass in.
31:05
But it does go a little bit again to what we were talking
31:08
about with with UX there where for RAG systems, yeah, it's like
31:16
you want a little bit more power and flexibility as you would
31:20
maybe with like a person to like either Fast forward certain bits
31:24
of the conversation or slow down at this bit or.
31:28
Like. Hey, let me send you a photo at
31:30
this moment. Don't make me pick from like 6
31:34
bits of text that like apply to my my thing.
31:38
So yeah. And that that's that kind of
31:42
thing is still, people are still figuring out like the bigger
31:48
picture and stuff. I, I imagine like once that's a
31:50
bit more solidified, then you start getting people thinking a
31:53
bit more about the UX. I'm not sure.
31:56
I'm not, yeah. I'm not sure how, how long this
32:01
stuff takes to like percolate through.
32:03
Yeah, and a lot of times I can imagine that the end user
32:07
doesn't realise that they have the option of sending a
32:11
screenshot or something. Because if they're not, if the
32:14
end user isn't prompt with do you prefer to send me a
32:17
screenshot and explaining what you're doing, then you end up
32:22
trying to explain it in text. And it's a little clunky.
32:25
And so, yeah, maybe it's just as easy as one of those six follow
32:30
up questions that you're giving is send a screenshot with your
32:35
problem. And so they go from there.
32:37
But it's fascinating to think about that I always intrigued by
32:44
the product journeys that people take, and especially around
32:49
friction that they may have that you as the creator of the
32:53
product never think about, right?
32:55
Because you know it in and out and you know, Oh yeah, if you
32:58
want to do that, you just click on this button and then there's
33:00
the little drop down and you get exactly what you want.
33:03
But the new user is just clicking around everywhere
33:06
trying to find what they're trying to do.
33:08
And they may not even really know exactly what they want to
33:11
do. And so they're exploring in one
33:13
way, but they're also trying to figure out if this tool is going
33:17
to be useful for them. And that's where it would be
33:22
awesome to have that little buddy that pops up, but it's
33:25
like, I see you're just randomly clicking around.
33:28
Can I help you find what you need?
33:29
And or I've seen you've done these five actions.
33:33
You know what else is really cool?
33:35
Here's a hidden trick. And so it suggests like ways to
33:40
become a better user of the product.
33:42
And that is, yeah, that's, yeah, that's very few and far between.
33:47
Yeah, and you see people experimenting with this.
33:49
I mean, like open AI and who else has this Microsoft with
33:56
their like we watch everything that you do all day.
34:00
I'm like, you see, like people trying to make that kind of
34:05
experience work. But yeah, it's clear it's it's
34:07
very early days. And yeah, just also just
34:11
thinking about like one thing we haven't talked about yet is like
34:15
evaluation and like how you then like create your data flywheel
34:19
and all of these kinds of things.
34:21
But you can imagine that like once you get to like 24 hours,
34:26
like monitoring the screen and interactions and stuff like
34:29
evaluations becomes, yeah, you need to really, really think
34:34
about how you do that. And you probably need like LLMS
34:37
or multi or LLMS, like in the loop, they're picking out
34:41
interesting examples or whatever.
34:44
And but yeah, at that point you're like evaluating like the
34:48
sum total of human behaviour. Yeah, it's hard.
34:52
Exactly. Was there Speaking of
34:55
evaluation, did you see common patterns that were arising
34:58
there? I'm always interested because
35:01
there's a lot of hype around LLMS as a judge, but actual
35:05
practise of using LLMS as a judge I'm I'm not sure how many
35:09
people are actually doing it. I mean, you, there are certain
35:13
people who tried it and they had mixed results and they found,
35:19
you know, they ran into a lot of the, the common failure patterns
35:22
of like using LLMS and like getting them to output like
35:26
number scores. And they found this was super
35:29
unreliable. So then they had LLMS, you get
35:32
like qualitative responses or you have LLMS which are like
35:38
flagging certain examples as. And you see obviously LLMS being
35:45
used in like synthetic data maybe to get you get you over a
35:48
certain a hump that you have in terms of like building out your
35:53
your functionality. But yeah, probably a lot less of
35:59
that. And yeah, and the, the common
36:03
pattern is like someone builds the POC demo, whatever at that
36:07
point, like evals are not really part of anything.
36:10
Then they start thinking about like, well, how do we present
36:13
this like to a wider audience, whether it's a the public or
36:18
internal facing for the company. And then that point hopefully
36:22
people start like thinking about evals and like, what are the
36:27
real, real like failure scenarios that we really can't
36:30
afford to happen here? So you have some very basic
36:32
stuff come in. And then, yeah, depending on the
36:37
size of the company, maybe they even stop there.
36:40
But then yeah, the bigger ones, ones with a bit more money to
36:44
invest in a kind of in a project like this, which doesn't
36:47
necessarily impact like the bottom line, then yeah, then
36:53
then they're actually like gathering the data, iterating on
36:56
the process and so on. But yeah, it's, it's more often
37:01
than not, like people are, people are often building these
37:05
LLM projects, particularly the internal facing ones, so they
37:08
don't fall behind on the technology almost they want
37:11
exposure to it. And it's kind of their way of
37:18
yeah, just being exposed to what's happening with with LLMS.
37:21
And yeah, we did this as NML as well.
37:22
We built or I built a Slack chat support box when a year, a year
37:29
and a half ago or something like that.
37:31
And that was mainly just like, yeah, we want to understand how
37:34
people are using this stuff. How hard is it?
37:36
What are the failure patterns? And and so on.
37:39
And so there's maybe different incentives played there versus
37:44
when you're doing something which is a little bit more like
37:46
profit results driven, you're yeah, you're more willing to to
37:51
accept certain risks or to do certain things in a certain way.
37:55
Well, you said one of my favourite words, which is ROI
38:03
and profit. Did you get a lot of insights
38:08
from all these blogs? Did many of them talk about it?
38:11
Because as you mentioned before, it was primarily engineers were
38:17
writing about the engineers problems that they're solving.
38:21
So I would assume that they're not necessarily saying this
38:28
netted us a huge ROI, but maybe there are some that you saw that
38:34
actually took that into account or they talked about how it was
38:40
or wasn't viable depending on the scale that they're looking
38:43
at. Not really apart from like the
38:46
whatever a few like Klarna or whatever where they actually did
38:49
did, did, did put like an actual number, big, big number on it.
38:56
And I think it's also a bit the ones that did talk about like,
39:02
you know, a big spike in users or we, you know, a renewed
39:07
interest or a product which was, you know, dying or whatever that
39:12
was revived by, by the introduction of RAG or like a
39:17
better, I don't know, LLM based search system or, or something
39:22
like this. It's really unclear to me
39:25
whether this is like something that would apply for like the
39:29
long term because yeah, a lot of people go to places to try stuff
39:36
out or to try out some new functionality or new technology.
39:42
And yeah, for, for a lot of stuff, it's clear like this,
39:47
this is in flux and it'll be replaced by something later on.
39:52
My favourite thing at the moment is in this, in this kind of
39:57
areas like notebook LM, like very popular.
40:00
It's kind of a cool use case. You know, they just launched
40:05
this like thing where you can like participate in the podcast
40:08
yourself. And if you saw this and tried it
40:10
out, it's kind of cool. It's fun to use.
40:14
Will be, will we still be using notebook LM or like playing
40:17
around with this in three years? I don't know.
40:19
Probably not. Hopefully.
40:21
Maybe that's like. A better kind of like meta tool
40:24
around like yeah, podcasting or discussions or study depending
40:29
on what your angle is in in that in that stuff.
40:33
So a lot of the use cases where people get like really good
40:36
results and like can give specific numbers about like the
40:40
number of customers they have or how people's user journeys were
40:46
improved by this. I don't know whether this stuff
40:48
is for the long term. It's just like we release
40:51
something and we have a lot of users.
40:55
Yeah. The other thing I think about in
40:56
this in this space is like all of these agent platforms, yeah,
41:01
probably like half or most of them won't exist in five years
41:04
from from now. But like some of them have like
41:07
a tonne of users and people playing around with them.
41:10
And yeah, particularly for the smaller use cases, like people
41:16
do like interesting things which are really useful for them.
41:20
But I, I, yeah, I have to think that like a lot of this stuff
41:23
will, yeah, will will be merged or amalgamated or, or turned
41:31
into something else like over time.
41:35
And when you say agent platforms, you mean to help
41:38
folks build agents like a, a framework that can help you
41:42
building agents or it's actual agents that you can go and use?
41:47
Well, some of them have the marketplaces for for agents,
41:51
which seems to be the thing which which makes them money.
41:54
But yeah, it's like kind of these like GUI interfaces, web
41:57
interfaces where you can like connect lines of like you do
42:01
this and if not go to this agent and that kind of stuff.
42:05
They're quite popular and some of them are making like real
42:08
money. And maybe it's a little bit
42:11
like, what's that kind of create your own avatar replica this
42:18
this kind of service where you can create like an AI avatar and
42:22
like it, it's personalised and so on, which is not like a mass
42:27
used thing, but like it, it's popular, it makes a lot of
42:30
money. Maybe two or three of those will
42:33
continue to exist like for for a long time.
42:35
But yeah, right, have to feel like a lot of these will just
42:39
drop off or like people will move on to the next thing.
42:44
Yeah, that's fascinating because the one thing that is clear is
42:49
that you have to have a lot of patience to get something
42:52
working on these. Build your own agents and you.
42:56
The debugging is hard because you don't know if you are not
43:02
prompting it correctly, or if you're not asking it to do
43:06
narrow enough of a task, or if the flow that you've set up is
43:11
is the problem. So if you're not willing to
43:15
spend the time to create that flow, then it's quite difficult.
43:21
However, I have seen there's a lot of common use cases that are
43:26
coming out-of-the-box with those little build your own agent
43:29
things. And so I I think I signed up for
43:32
one the other day and it said, what are you, what do you like
43:36
identify as? And so I said more, let's see
43:41
what they have for product marketing or SEOI think there
43:46
was just all different marketing use cases.
43:50
And then it gave me a lot of flows that they had set up.
43:55
And you just add in simple things like, oh, here's my
43:58
website, here's the keywords that I'm trying to rank for or
44:01
whatever. And and they make that easier on
44:04
you to try and reduce that friction.
44:07
Yeah. And I mean that that works great
44:09
for like a well defined domain where we already know like the
44:13
the things which are important for SEO and the tasks or
44:16
whatever. But when you you know, a lot of
44:18
the use cases where people talk about agents like it's in the
44:21
area of like research where, yeah, that's a little bit
44:25
harder. Like if you knew what the
44:27
problem was and how to get to it, you wouldn't need the agent
44:29
probably. Yeah, yeah.
44:33
Yeah, fascinating man. So as you were putting together
44:38
all these different blogs and different sources into the
44:41
database, did you find any sources that were consistently
44:47
just giving top notch material and publishing absolute quality
44:52
material? I mean some of the the ones
44:56
which are well known. The Netflix tech blog DoorDash
45:01
has a really good one. Honeycomb produced consistently
45:05
stuffed weights and biases since they started their kind of in
45:09
support chatbot. I think they've done 10
45:11
different like deep dive blogs technical styles.
45:15
Yeah, there were a lot some of them, yeah.
45:21
And then the rest were just like a tonne of like random, random
45:25
blogs on, on companies where, yeah, maybe they haven't written
45:30
anything previously or the new companies or yeah, that was that
45:35
was a that was a bit. Finding them was hard.
45:40
I used this really great search, embedding space search engine
45:44
called Exa dot AI, which yeah, you put in like some other blog
45:50
and you say find other blogs like this.
45:53
And because it's embedding space, you get really great
45:55
results. Would recommend that yeah.
46:00
So, so they were, they were all of the ones that you'd like, you
46:04
know, already for like having great technical teams and then a
46:07
tonne of like just, yeah, ones you have to like hope that
46:13
they're posting somewhere in your social network or whatever.
46:15
Someone really shares them because it's hard to find, yeah,
46:20
hard, hard to find these the case studies.
46:23
And yeah, obviously like envelopes community, like
46:27
definitely love all the videos. Like I hope we've done a service
46:31
of like liberating some of the content out in text form out of
46:34
the videos by like summarising from, from the transcripts and
46:37
stuff. Because yeah, there's like, I
46:41
think there was something like 100 or so that can't be true.
46:47
100, yeah, 100 or so videos referenced in the database.
46:51
And like, yeah, maybe you don't have 100 hours to to watch all
46:54
of the videos. So yeah, or at least you can
46:57
decide whether whether you want to like watch it based off of
47:01
the summary. Is there any cool stuff that you
47:04
want to do with the data visualisation on this?
47:08
Because it feels like with all the topics or with all the
47:14
different philtres, you could create some fun data
47:19
visualisation, whether it is OK chat bots and you have a whole
47:24
embedding space the way that you're looking at it, or you're
47:27
looking at the different use cases and, and that type of
47:31
thing. Or are you done with it?
47:33
You're like, all right, I put it out there.
47:35
Now I'm going to get back to work and keep rocking at NML.
47:39
I mean, we're, we're continuing to maintain it and people,
47:42
people submit use cases and articles and so on.
47:44
So, so that's really great. That will only grow.
47:48
The one thing I. Really.
47:50
Where do we submit? Where do we?
47:51
There's a link at the top of the database.
47:53
It's just a form. OK, Fill in.
47:55
Yeah. And you know, we put out the
47:58
data set also as a Hugging Face data set.
48:01
So if people want to like don't want to scrape our website in
48:04
order to get all of the data, like we've done that for you,
48:07
just go to the Hugging Face. But something I really wanted to
48:12
do and I didn't have the the time to implement it basically
48:15
is like to allow people to search tool based by all of
48:19
these use cases. You want to see like all of the
48:22
companies that are using like Llama index for embeddings or
48:27
all the people who are using like quadrant or pine cone, like
48:31
vector databases or whatever. And then see like common, common
48:36
use cases or a common like whether it's failure or success
48:40
patterns around like particular tools and use cases.
48:43
So yeah, it's a bit harder to to implement like the extraction of
48:48
the tools or at least to automate it in a reliable way.
48:53
So, but that that that would be a, a useful thing.
48:59
What I can promise is we're not going to have like a chat with
49:02
your LLM OPS database like functionality on top.
49:05
Or you can build it yourself if you want to download the Hugging
49:08
Face too. That's it.
49:09
Next to Envelopes Community hack.
49:11
It's on. Yeah, we're going to do that one
49:13
that. That's so good.
49:15
Oh, man. Well, yeah, you've done some
49:18
awesome stuff with it and I really appreciate you putting it
49:20
together because it, like I said, is this one resource that
49:24
I can come back to and continue to learn from.
49:28
And so I hope that you keep updating it And anybody out
49:32
there that is doing anything cool, if you write about it,
49:36
make sure to submit it to Alex and the Zen ML team.
49:39
This has been awesome. Thank you.