Latest / Agentic Conversations (formally mlops.community) / LLM Use Cases in Production // AI in Production Conference // Panel 1

Transcript

LLM Use Cases in Production // AI in Production Conference // Panel 1 · Agentic Conversations (formally mlops.community)

0:00
This panel is going to be absolutely fascinating.
0:03
It's going to be a discussion about use cases for LM and
0:06
production now is absolutely stellar lineup for this
0:09
conversation. Gregory, I will let you probably
0:13
do the introduction is everybody here.
0:15
We got Greg here. Hello, Greg.
0:17
Hello. OK, Arjan, Can I call Arjun?
0:21
Is Arjun here? See, we got Arjun here and
0:28
Jason. Good to see you, Jason.
0:32
Yeah. How are you?
0:34
OK, guys, I'm stoked for this. You have 30 minutes for this
0:38
panel. Greg, if I understand it right,
0:40
you're going to be moderating it.
0:42
And Agnieszka, I don't think you see your video yet, but I will
0:47
be coming back in about 2526 minutes and to bother you and
0:53
until then? Beautiful.
0:57
I think that's, I think that's the exit to kick us off.
1:00
Well, either way. Hello everybody.
1:02
My name is Greg and I'm going to be leading a panel today on LM
1:05
use cases in production. Now, this is a use case that is
1:08
near and dear to my heart because I am obsessed with
1:10
knowing how is AI actually going to turn into customer value?
1:14
How is it going to make all of our lives a little bit better?
1:17
And we have an awesome panel today with three folks.
1:19
I will let the introductions be by each person, but either way,
1:23
let's just jump into it because we just have 30 minutes today.
1:25
So Jason, can you start us off? A quick little introduction.
1:28
Hey guys, I'm Jason. I'm right now the creator of a
1:31
library called Instructor that helps use language models, do
1:34
structured outputs and validation.
1:36
And then I also work as an independent consultant for a
1:39
couple of companies such like Trunk Tools, Narrow and Rewind
1:42
AI, and they mostly focus on things around like
1:46
transcription, summarization and also a lot of rag.
1:49
That's super cool and Jason is being a little bit modest cause
1:52
he shares a lot of really awesome content on Twitter as
1:54
well, so make sure to go follow him there And then Arjun, do you
1:57
want to do a quick intro for us? Yeah, UH and margin co-founder
2:02
of a company called Resides. We do customer service for
2:06
rental buildings. So if you ever talk to your
2:08
landlord, we are the person on the other side, but we do that
2:11
at scale. And yeah, as you can probably
2:15
imagine where customer service business, so LM form a very
2:18
large part of a very large portion of everything we do.
2:21
So excited to get into it. Wonderful.
2:24
And I've actually had the chance to talk.
2:25
With Arjun offline about? Some of these use cases and he
2:28
has some awesome. Ones I think he's going to share
2:29
with us today. And then finally, Agnesia can.
2:32
You do an intro for us. Hi hello I'm glad you can
2:36
finally see me and my name is Agnieszka.
2:38
I did my pH. D in machine learning regarding
2:42
the bias in machine learning models and data.
2:45
And also I worked a couple of years in voice AI company when I
2:48
worked with voice spots and also large language models, training
2:53
some of them, fine tuning some of them and now I'm working on
2:57
the company called Chapter when we are working on LM Assistant.
3:03
That's awesome. Thank you very much for.
3:05
Joining us today. So the first question I want to
3:07
ask is gonna be around. The very general.
3:09
Topic of LM use cases. Now, when I hear these, I
3:12
usually don't like to talk to AI.
3:14
Companies because they'll just. Tell you a bunch of product
3:16
marketing. They won't tell you the real
3:17
stories about people. Actually getting value.
3:19
From these things, so I'd love to pose a question to all three
3:22
of you. Arjun, I'm going to start with
3:23
you just to put you on the spot, and I would love to hear about
3:27
an actual LM use case that you're using right now in
3:30
production and you can see the exact value that is giving you.
3:35
Yep, absolutely. So I mean there's a hundred that
3:40
I could go into, but the one that I really like that we're
3:43
working on right now is the idea of being unstructured property
3:48
documents and actually making it so that we can answer, answer
3:50
residents resident questions. So the practical sort of impact
3:54
of this is that our business is built on being a frontline
3:59
customer service for residents, which means if you live in a
4:02
property and you have a question about the gym, about where your
4:05
emergency hotline is, how to book an entity, blah, blah,
4:08
blah. Usually that means you go to
4:10
somebody at your front desk or you e-mail somebody and you just
4:12
never hear the answer back. We are in the middle of that.
4:15
And so that means we're in the business of answering as many
4:17
questions as we possibly can to help our customers, the owners
4:20
and operators, save time and money and actually answer
4:24
questions for our residents. Problem with that is that most
4:27
of the knowledge about a property is kind of offline and
4:30
really bespoke. It's scribbled down on random
4:32
notepads. It's in some PDF that was once a
4:35
poster on somebody's wall or worse.
4:38
It's kind of not written down yet until you ask somebody about
4:41
it. And so we were kind of
4:44
struggling with this for a very long time.
4:46
And by a very long time I mean pre GPT which is any, we would
4:52
have all of these like manuals. But anytime a resident asks a
4:54
question that we didn't know the answer to, it was a very manual
4:57
process of going back to the property, getting the answer,
5:00
coming back and then never ever matching it actually on the next
5:03
use case. Fast forward to today, like
5:06
instead. So that meant like we were
5:08
asking maybe top of our residents question in any in any
5:10
given month, which isn't really saving anybody a lot of time
5:14
even though it's decent. Today we're answering about 95
5:17
to 99% of resident questions. And the reason it's happening is
5:20
because we can take the very same random posters, random
5:24
replies, random questions, throw them all into a vector database.
5:30
We've cut down training time for all of our customers to
5:32
basically nothing, where earlier they had to solve answer like
5:36
questionnaires blah blah. And now that means, like most of
5:39
our customers now use us instead of needing to set up a resident
5:43
call centre or whatever else. And there's a lot of work flows
5:47
in the middle of all of this too.
5:48
But my favourite use case right now is just this idea that you
5:51
can take all this like really weird, really unstructured,
5:54
bespoke knowledge that was just in people's heads and actually
5:57
make a useful repository of it in a way that just wasn't
6:01
possible before. That that's super cool.
6:03
I love that. And if if you were to put an
6:07
impact metric on that, say it saves us 20 hours a month or
6:10
whatever, maybe, how would you quantify it?
6:13
I mean I would say it's cut down our expected head count on the
6:19
number of associates we need by like half.
6:21
So to like we have used, we used to have one person supporting
6:28
about 7 to 8000 apartments at a time before this.
6:34
Now they support maybe closer to 20,000.
6:37
Wow, that's that's awesome. I want to move on to some other
6:41
panellists here. So, Jason, I know that.
6:42
You're working with a lot. Of companies and you have a
6:44
unique perspective because you're seeing a tonne of
6:46
different problems and solutions out there.
6:47
So what's one of the ones that you like?
6:49
That you want to bring up today? Yeah.
6:51
I guess the biggest point that I've been sort of seeing as a
6:54
consultant has been the fact that like there's a separation
6:57
between what is a capability versus the value, right.
7:00
The same kind of, you know if we were to build like a
7:02
summarization prompt for example, the value is kind of
7:05
very unknown. But if we apply the same
7:08
summarization tactics to something like a sales call,
7:11
then if you were to be able to help you convert a sale, you
7:14
know like with like 5% more likelihood that value is becomes
7:17
like dramatically higher, right. If we just capture the right
7:20
pain points or we make sure we're capturing the right, you
7:23
know, poor building opportunities, then you just
7:25
become a lot more impressive to the client that you're working
7:28
with. And so, you know within within
7:30
things like narrow and rewind, a lot of the time it's really been
7:33
around, you know, augmenting the human and making them seem much
7:37
more impressive as a result of these capabilities, right.
7:39
Whether it's, you know, closing consulting interviews like doing
7:43
research and building out reports for folks, people
7:46
getting their time back is only one part of that value equation.
7:49
Yeah, I love that and I love how you positioned it as there's a
7:52
discrepancy between capability and value because often on
7:55
social media people will correlate.
7:57
Ohh, there's a huge capability here, so that must mean there's
8:00
a huge. Value, but that's not always.
8:02
The case So I would love to hear an example of where there's
8:05
actually a small capability, but you've seen a huge.
8:08
Value so like the summarization topic you brought up.
8:11
Yeah. So one of the examples is
8:13
actually from a startup that I was working with earlier this
8:15
year where we were just doing extraction out of a transcript,
8:18
right. But if you were to sell
8:20
extraction of a transcript, that could be a free Tubidy app, but
8:24
the the audience that they captured was executive, not
8:27
exactly the systems executive coaches or these are people that
8:31
will charge, you know 800 to $1000 an hour to sort of help a
8:34
board of directors manage this CEO.
8:36
And they would take about two hours to just extract quotes to
8:38
prepare the presentation. And so the API call for 25 ends
8:43
up being $0.50. The actual upside for the
8:46
downstream user becomes thousands of dollars because of
8:48
exactly what they're trying to do.
8:50
And now because we have very good metrics on OK what is or is
8:54
not a good quote, we can fine tune against that.
8:58
You end up charging you know maybe like 100 dollars, $200 for
9:01
a single API call just because the value is there, right.
9:04
You would have self summarization as a service, it's
9:07
very cheap. But if you niche down and and
9:09
capture a very specific target audience, maybe failed people or
9:12
you know people who go door to door and sell solar panels, like
9:17
the the more you niche down the bigger value you can capture
9:19
because they we know exactly what they're trying to solve.
9:22
Yeah, I, I, I absolutely love that.
9:24
That's that's really awesome. So then Agnieszka, I want to
9:27
move over to you and hear about a use case that you're excited.
9:29
About within your line of work right now.
9:34
Yeah. So when we usually talk about
9:36
the in business or how it affects the business, we usually
9:40
talk either about lowering the costs or you know, getting
9:45
higher sales, right. So we already talked about that
9:48
also about higher efficiency, but one of the very nice
9:52
projects I had in the past was was detecting arguments within
10:00
the sales calls. So we had transcripts like
10:06
thousands of hours of transcripts which we used to
10:09
detect arguments and objections towards those sales calls.
10:14
So what actually happened is we could discover what arguments
10:19
worked better for each objection.
10:21
So for example, if a customer didn't want to buy something, we
10:26
could say for example that, but we are the leader in the market.
10:30
But maybe it doesn't work, maybe the better way would be saying,
10:35
yeah, but actually our price is the best on the market, maybe
10:40
this will work better. And we were able to find the
10:44
best arguments for each objection and our customer used
10:49
that argument and increased the sales by 30%.
10:53
So we thought it was really amazing.
10:56
Equal 30% is a tonne on the sales side.
10:59
Do you know would they incorporate this into their?
11:01
Training or how would they incorporate this data into their
11:04
workflow? Yeah, so actually they used this
11:06
in the training they may, they talked to their workers, show
11:14
them how they can talk with clients and they increase the
11:17
sales. It was easy as that.
11:18
So just investigating the data, so maybe they had some better
11:24
colours that used those better arguments.
11:27
But there was no knowledge sharing sharing along the team,
11:30
so this helped a lot. That's awesome.
11:33
That's that's very cool. And so I want to move to another
11:36
question here and this is almost around the anti value meaning
11:40
paths that folks may have gone on that haven't been as fruitful
11:43
as they wanted. So Arjun, I want to start with
11:45
you because I know at Resi desk you're running a lot of
11:47
experience experiments, a lot of mini.
11:49
Workflows. Which workflows have you
11:52
started, maybe implemented but then have cut because the value
11:55
wasn't? There for the overhead it took
11:56
to maintain. I think to start when we started
12:02
we approached it. The mistake was I think overall
12:04
optimising and over engineering the so broadly the mistake was
12:10
taking solving this like an engineering problem instead of a
12:12
product problem. And what I mean by that is when
12:15
when the LMS came out we spent way too much time optimising for
12:18
the best possible prompt the best possible latency.
12:22
Which meant that I think our lead time to ship something
12:24
within LM was just like not bad, but it was still on the order of
12:28
weeks instead of on the order of hours or days.
12:31
And the biggest shift for us was just changing that to be like,
12:36
no, just ship what you can and then evaluate it with an LM
12:40
continuously in production. And that changed quite a bit of
12:45
quite a bit of our entire workflow and ended up being
12:49
valuable. So create like concrete example
12:52
is we want an LM that scores every conversation that we have,
12:56
much like what Ignacio is talking about with a resident,
12:59
right. And like we spun our wheels for
13:02
way too long trying to figure out what are the exact segments
13:05
that we want to score along etcetera etcetera.
13:07
And that took us like months and was a waste of time.
13:10
And instead what we should have been doing from the beginning.
13:13
So what we do now is just like score it, tell me anything
13:17
that's worth noting and then collect 1000 examples, use that
13:20
to develop your your buckets, use that to score your next 1000
13:25
examples and then go from there. Like it's the same lesson I
13:28
think that you learn with every new tech paradigm, which is
13:31
think about it from the product angle first, even though the
13:33
engineering angle is so much more interesting or it seems so
13:36
much more interesting at the beginning.
13:38
So we over optimised at the beginning and didn't really
13:41
monitor continuously and that was a massive pattern that we
13:45
needed to change. Yeah, that's very cool.
13:47
It's. It just reminds me of the old.
13:49
The outage around just. Get the MVP.
13:51
Out and start there and then iterate on performance after you
13:54
optimise later. Yeah, cool Jason.
13:58
I'm. Very curious to hear your
13:59
answer. To this question too, because I
14:00
know you engaged in a lot of. Projects with a lot of.
14:02
Different companies what? Has started, but it wasn't worth
14:06
the overhead to keep on maintaining or it didn't work
14:08
out. I guess a lot of it has almost
14:11
been an an organisational level. I think just in the same sense
14:16
of urgent talked about how you know it's an engineering we
14:18
should focus on product. One of the big things people
14:21
started trying to do is sort of plan too much ahead of time,
14:24
right. I think in engineering it's very
14:26
easy to think of, OK, what are the exact deterministic results
14:29
we're looking for like what are the like 20 tickets that we can
14:32
put this problem down into and then what are the edge cases we
14:35
expect to bump into, right. The issue here is like just just
14:39
that it's it's it's product about engineering.
14:41
A lot of the machine learning optimization pipeline is very
14:44
much science and not engineering, right?
14:47
And so I find that a lot of these junior engineers or
14:49
engineers transitioning into doing more AI related work if I
14:53
just spent too much time guessing the outcomes rather
14:55
than optimising a test pipeline, right.
14:58
You know, we could spend the week writing at this
15:01
documentation proposing what could be fixed or what needs to
15:03
be improved. But instead of doing that, as we
15:06
focus on metrics, hypotheses, and experiments, we end up being
15:10
able to iterate a lot quickly, a lot more quickly with that sort
15:13
of the overhead and the complexity of sort of trying to
15:17
guess everything. Yeah, absolutely.
15:19
And you brought it up just a little bit, But I like the
15:21
advice for. Junior engineers trying to get
15:23
into more. AI related work, so less on the
15:26
technical side, but what advice do you give to them?
15:28
More on the meta level about their learning path in which
15:31
direction they should be pointing.
15:33
Yeah, understand. Like how we should be focused on
15:37
evaluations, you know, be educated on what kind of metrics
15:40
and evaluation tools we have at our disposal and really focus on
15:44
building experiments that are very fast to run, right.
15:47
When a test takes one minute to run versus a test that takes an
15:49
hour to run, you know, it really changes the way that we do the
15:51
work that we want to do. And in particular, it gives us
15:54
the ability to build it intuition as to how this problem
15:57
is going about. And on top of that just you
16:00
know, making sure we look at the data, if we have evaluations, we
16:03
can now look at things that are performing well in performing
16:06
poorly and build this internal intuition that will effectively,
16:10
you know that will be something that you take with you for the
16:12
rest of your career. Nice.
16:14
I love it. A test that takes a minute to
16:16
run versus an hour or a test that takes a week to set up and
16:19
then you can finally run. It is also a pain in the butt.
16:22
So I want to switch gears to you Agnieska, about understanding
16:27
how you prioritise your energy. So there's a lot of things we
16:30
can all work on, but because the tools maybe aren't as mature as
16:33
some other tech stacks, well, it might take a while to go
16:36
implement those and we need we need to prioritise.
16:38
Our energy, very well. How do you think about the
16:40
projects that? You're taking on in your career
16:42
or your work and I guess yeah. Could you step us through?
16:46
How you pick your projects. Yeah, this is very interesting,
16:50
very interesting problem. I myself often have problem with
16:55
managing my time, but what I usually do is listening to
16:59
podcasts, so learning from others, listening to podcasts,
17:03
reading blogs, reading papers and just being up-to-date of
17:10
what's happening. So because I'm learning and I'm
17:13
learning from others failures, other successes, I can avoid my
17:19
failures by myself. So I think this is very
17:22
important to just be up to date with everything.
17:26
I know it's very difficult and it's sometimes almost impossible
17:30
to catch up on everything by. But I myself go into archive
17:36
website every three days and check every paper and paper.
17:41
I don't read all of them. I read the title, I can see if
17:44
it's worth checking and I read it.
17:47
Yeah, and I think this is very important to survive in this
17:52
field. Yeah, absolutely.
17:55
And so when you check archive and you have this idea.
17:57
Around Hey, we're going. To eventually need to.
17:59
Be building products that provide value to customers.
18:02
What sort of things catch your eye as you scan through?
18:04
Those. OK, so when I read the, because
18:07
I have a PhD background, I can quickly search for those papers
18:12
read and I know what's happening there.
18:14
So first I read an abstract and I can see if it's a nice paper.
18:19
I can see if it's written nice and this will probably mean that
18:24
it's it has a good value. So I go through the methodology,
18:28
usually through the results and when I see that results are good
18:31
enough for me to read, I read the whole paper.
18:34
If it will bring value to our product you're developing or if
18:39
we have an an idea based on that paper for a new product, then we
18:42
can move with it. And also it's pretty nice to use
18:46
assistance to summarise those papers.
18:48
So you can always use either GPT or Gemini or other help you
18:55
summarise the paper and find the key points you need.
18:59
Yeah, I I love it. I love it.
19:00
Arjun, I want to move over to you.
19:02
So at Resi Desk, you're running the company again.
19:05
You only have a finite amount of energy to pick your projects.
19:08
What's your evaluation criteria? How do you think?
19:10
About what you take on. So I'm going to give you the
19:14
boring answer, which is it goes back to where the company is and
19:19
what our customers want we happen to be.
19:22
You know, before our customers started telling us what they
19:25
needed, it was much more of a discovery tool.
19:27
So like it was just, it was closer to what Agnieszka was
19:30
saying, which is discovering you know all the capabilities that
19:33
we have with this technology and surfacing new features in the
19:36
hope that one of them turns into a hook that captures user value.
19:40
But now nowadays it's much more of we have they pretty strongly
19:45
validated pipeline in terms of customer problems we need to
19:48
work on for the next, for the next moment almost too validated
19:52
I would say. So now it's really about like
19:56
where you constantly having to make trade-offs between where do
19:59
we choose to expand our margins AKA like automate our humans
20:03
versus expand our product and increase prices AKA make our
20:06
human team much more awesome. Right now my general framework
20:10
is much more weighted towards focusing on use cases.
20:15
That make our human team much more awesome.
20:18
So that's why I go back to you know improving our understanding
20:22
of documentation, improving our understanding of our customers
20:25
like investor playbook. So that anytime a great
20:28
conversation goes through our system that matches something
20:32
that makes them look good to their investors like we surface
20:35
that things like that. So that's sort of our, our
20:39
general outlook, it's elements haven't fundamentally changed
20:44
the way that we prioritise things with the business, but
20:46
they have changed the cost of experimenting within those
20:49
priorities. So business priorities are
20:51
always like make more, grow your top line, grow your bottom line.
20:55
LM have just made it far easier. Run 10 things that help you do
21:01
that instead of, you know, do two things at a time with a
21:03
small. Team Totally, totally.
21:05
And I know there's. Popular frameworks out there to
21:07
do prioritisation like the Impact.
21:09
And effort? Do you follow that line of
21:11
thought or how do you prioritise?
21:12
What you work on internally. We do a very rough impact and
21:17
effort prioritisation or at least we used to the the
21:22
denominator or the effort just ended up becoming so small that
21:26
everything started to feel like it was a good priority.
21:30
And so now instead we have, we're basically run a lot of
21:36
tests around what a customer might pay for something that we
21:39
do. And that has replaced the effort
21:42
piece for us because a lot of projects in the mean in the
21:47
medium term for us are all roughly equivalent in terms of
21:49
how much time they're going to take.
21:51
But they're vastly different in terms of how much a customer
21:53
would pay for them. Yeah, yeah, yeah.
21:56
That's a whole nother path because.
21:57
There's the internal operations, there's building our product,
22:00
but then there's also how much of the costs are going to pay
22:02
for it. So that's a lot of.
22:04
Factors to tackle there, Jason, I want to move to you and the
22:07
lens I want to look at this question through.
22:09
Specifically, is around. The the user research side, and
22:13
that's a kind of a lame name. For how do you identify projects
22:16
that are going? To make the most impact value
22:19
wise to an end user, not. Necessarily from the technical
22:21
side. Yeah, I guess here I'm really
22:26
borrowing from a framework of like Alex, from Ozzy where you
22:28
talked about like the value equation, right.
22:30
The idea is that the value derived is a function of the
22:34
dream outcome times the likelihood of success divided by
22:37
the, you know, time it takes to get there and the the sacrifice
22:40
that we are willing to get right.
22:42
And so it's an amazing thing is, and ultimately that's has that
22:49
has been an incredibly simple framework to apply, right?
22:52
It's like, OK, are we saving them time, right, by giving them
22:56
two hours, by doing something in 10 minutes, are we giving them
22:59
the likelihood of success because there's some kind of
23:01
coaching aspect or some kind of, you know, making sure that if a
23:05
human is 95% accurate, are we getting them to 99% accurate?
23:09
Or is it purely an effort play where reading legal text is just
23:13
very, very difficult. And so I think that for the most
23:17
part has been really helpful to to apply and at least prioritise
23:21
prioritising some of these projects.
23:22
And ultimately the the the most exciting thing really is around,
23:25
you know, what does that dream outcome right.
23:27
I think right now we think of simple things like
23:29
summarization, but really we want to sell is becoming a
23:32
better salesperson or becoming like a better partner by having
23:35
perfect memory. These are kind of things that
23:37
I'm really excited. About That's cool.
23:39
When you say become a better partner, do you mean romantic
23:42
partner here or which were you talking about?
23:45
Yeah, like even even in this example, I think about like what
23:48
is the, you know, for example, with rewind, do you think a lot
23:50
about, you know, giving you perfect memory?
23:52
But what is that perfect memory in service of, right.
23:55
If you were a salesperson remembering that thing about you
23:57
from 2 weeks ago or the reason you had to reschedule, those are
24:01
touch points that are very useful.
24:02
But in the same sense, you know, remembering like what kind of
24:05
like flowers someone likes or you know, whether or not you go
24:08
travelling, you see a postcard. Oh, this person thinks about
24:11
that thing. Again, like the idea of search
24:14
and memory are not applied to this much more, these much more
24:17
human outcomes. And that is kind of the the
24:20
grand, the grand vision. Totally.
24:22
Yeah. If something cost $30.00 a
24:24
month, but it makes you a better person, it's basically free, but
24:27
if it makes it cost $30.00 a month.
24:29
But it just helps you answer a question every once in a while.
24:32
That same system positioned differently, has a different
24:34
kind of, you know. Value equation.
24:36
Yeah, I think that's that's great and I want to be mindful
24:39
time here, so we just have 5 minutes left.
24:40
It's amazing how quickly a session like this can go.
24:43
I'm going to finish up with the fun question, which is talking
24:46
to. Your prior self.
24:47
And so we've all taken on. Projects We've all.
24:49
Had a couple of lessons learned as agnese gonna start.
24:52
With you 1st. What would you tell yourself,
24:55
say, I don't know, call it 18 months ago about?
24:59
What you've learned over. These past years.
25:01
With regards to picking projects that are going to create impact.
25:08
Yeah. So definitely, I would say focus
25:10
more, focus more on user value than on exciting research
25:14
because it's really, you know, easy to get lost in exciting AI
25:18
research in developing the most recent technology.
25:23
But as Jason said, we are selling the user value, we are
25:27
selling selling use cases, not the exact AI script or
25:32
programme. That's wonderful.
25:35
Arjun, I will pass the mic over to you.
25:37
What? Have you learned over these past
25:39
couple of years that you would tell yourself?
25:43
I think the the main thing for us has been don't follow the
25:50
don't follow the flash. A lot of a lot of our AI
25:55
intuition comes from seeing great products like rewind or
25:58
whatever that are magic. But in the context of what we've
26:02
built for our company, our customers actually like it when
26:04
it feels more human and less like AI.
26:07
And so to us it was don't focus too much on the technology.
26:11
In fact, the best way to have an AI product go get out there in
26:14
the wild is to make it feel human has been the lesson that
26:17
we've had to learn. Cool.
26:20
I love it. And then?
26:21
Jason Wolfe finished. Off with you.
26:22
What have you learned? Over the past couple years that
26:24
you tell yourself. I mean for the most part this is
26:28
kind of very selfish, but I've just been really enjoying this
26:30
sort of understanding of the the long tail of issues of you know
26:34
what happened in production, right.
26:36
I think very many people can get that to 80% result, make the
26:39
tweet, but then when you actually launch the product, you
26:42
end up just running until things OK.
26:43
Now we have a turn problem because we we got the virality,
26:46
but now we can actually deliver on these promises and just
26:49
getting exposure to many different industries and
26:52
different verticals and seeing what that long term looks like,
26:54
whether it's the idiosyncrasies of the questions and the
26:56
Interreg application or you know how difficult it is it is to
27:00
even sort of parse specific kinds of data.
27:03
Those are the things I find really enjoyable.
27:05
That's cool. I love it.
27:06
And yeah, I'm sorry, you're going to say.
27:09
I just said how? I'll finish it off here, just
27:13
with one lesson learned that I've learned myself over here is
27:15
I've really come to learn the value of a headline.
27:17
Before you start building. So if you aren't sure what
27:20
you're building and you can't succinctly, put it into a.
27:23
Headline. That's not a great place to.
27:24
Start and also when you're communicating it to users.
27:27
If you can't put it in a simple headline, that somebody.
27:29
Can get right of. Way.
27:30
Well, it's not going to get easier after you've built it.
27:33
So I think headlines. Matter for my own clarity and
27:35
we're also communication other users here.
27:37
But either way I want to say thank you to the three panels
27:40
say that was a wonderful. Again, it's amazing how quickly
27:42
these things go by. And Adam, we just had a great.
27:45
Session here, so we're ready to pass it up.
27:46
Back off to you. Thank you very much.
27:49
Greg, just along that line of thinking about the headline,
27:52
there's a really good book that I read called Working Backwards
27:55
about how Amazon product teams start building their products.
28:00
The first thing they do is come up with a press release.
28:03
Is the press release itself even attractive enough to get
28:05
anybody's attention? If so, then you can go build it.
28:08
You can't even do this. Forget about it.
28:09
I want to ask you guys. There's been a couple of
28:11
questions in the chat and we didn't really leave time for
28:15
Q&A, but so perhaps like rapid fire like one sentence answer if
28:19
you can. Uh, OK Smriti is asking.
28:22
Could you guide on some references for on how to
28:25
validate quality workflow for LLM outputs?
28:31
I think there are lots of open frameworks.
28:34
For example Lang Smith is one of them.
28:39
So just use open frameworks. Uh, OK.
28:45
My answer is revenue. Staying answer for me.
28:50
Whatever gets people paying. Nice.
28:56
OK. Last one is Smitty's asking
28:59
where can you source these papers from against.
29:01
I think that one was for you. Yeah, so it's a a website called
29:07
Archive ARIX. Then search it and it's a
29:14
website with a preprints so papers before the review.
29:20
Awesome. And then last one, maybe any
29:22
trick for recurring value from Pablo?
29:31
So interesting one. Recurring value.
29:35
Connection lost. Can you guys hear me now was I
29:44
got OK, yeah. So I think the question was any
29:46
any trick for recurring value? Like I should take this one.
29:56
I'm thinking about it. It's like, I mean ideally if you
29:59
have L in your workflow, they're constantly adding value.
30:02
So in in a sense that is recurring.
30:04
But maybe I'll I'll answer a slightly different question
30:07
here, which is how do you continuously get more value out
30:10
of a workflow that you've put in production.
30:11
And I think the answer is use LM's to evaluate your LML
30:17
workflow. The meta aspect of using LM is
30:21
highly underrated. They can do 100 IQ tests for you
30:25
every every hour. So you can continuously improve
30:27
your workflows and that's how you get better efficiency,
30:30
better output and ideally more revenue.
30:35
Wonderful. Thank you very much guys.
30:37
This was a sick panel. I appreciate all of you for
30:40
showing up and sharing these insights with us.