Latest / Agentic Conversations (formally mlops.community) / All the Hard Stuff with LLMs in Product Development // Phillip Carter // #170

Transcript

All the Hard Stuff with LLMs in Product Development // Phillip Carter // #170 · Agentic Conversations (formally mlops.community)

0:00
My name's Philip Carter. I am a principal product manager
0:03
at Honeycomb and I take my coffee home roasted because I
0:07
buy beans and then I roast them at home and I make espresso with
0:10
it. OK, we are back for another ML
0:15
OPS community podcast. I am your host Dimitri Owes.
0:18
Today I'm flying solo and we're talking with Philip Carter of
0:22
Honeycomb. As you saw, he got on my radar
0:26
because of the incredible blog post that he wrote on the
0:31
Honeycomb blog. Talking about the hard parts or
0:34
the parts that people don't usually highlight, especially
0:38
not on Twitter, when you're talking about your AI isn't
0:41
demos and how it's. This revolutionary new AI
0:44
product, well, he is actually using Ellms in production and we
0:51
go over all of that in the conversation today, we talk
0:54
about some of the challenges that they encountered and I
0:58
absolutely loved the conversation that we had around
1:02
how they measure the ROI of their open API calls.
1:07
It was nice to see that he had tangible numbers and he also had
1:12
leading metrics. That they looked at.
1:14
And so they saw that if we're able to affect these metrics in
1:20
a tangible way, then it makes all of the Open AI API calls
1:25
that we're doing worth it because we know that.
1:29
In. Theory More people that are able
1:33
to activate by taking these actions in our app, they become
1:38
paying customers and they're able to subsidise the cost of
1:43
these open AI costs. So enough of me talking about
1:46
this, let's just get right into it.
1:48
I hope you all enjoy the conversation that I had with
1:51
Philip Carter. And I will mention that if you
1:55
have not given us a review or at least liked this podcast yet,
2:00
what are you doing? It means the world to us.
2:03
If you share this with one person, one person only that you
2:06
think would find value in it, it would be a huge help to help us
2:11
grow and get the word out about the ML OPS community or these
2:16
days. Might as well call it the LM OPS
2:18
community. No, I'm joking.
2:21
We are not going to change it because let's be honest, it's
2:25
all ML OPS. At the end of the day, we're
2:28
just trying to navigate this unchartered territory together
2:32
and there's a lot to learn. I hope you enjoy the rest of
2:36
this podcast. to be on here with us today.
2:48
This is, it's been some time in the making.
2:52
I know we had to reschedule and I was having problems and then
2:55
it was 4th of July and things happened.
2:57
So we had the persistence though to make this happen and I
3:02
already know it's going to be great.
3:04
And so thank you for coming on here.
3:05
Yeah. Thanks for having me.
3:06
So you said you wanted a insightful conversation from
3:10
this and you want me to? Ask you some hard questions.
3:13
Now, when it comes to the blog that you wrote, you threw out a
3:18
lot of things that I think many of us were thinking, but you
3:21
articulated it in a way that really rallied the Internet and
3:27
everyone who has been playing around with AI.
3:31
Behind you. And so it struck A chord it and
3:34
I I was rereading it just again today and there's so many pieces
3:38
that I want to pick out and mention and talk about.
3:42
But before we do that, I think it's probably right to start
3:46
with your position and your company and how you're using AI.
3:50
Yeah, absolutely. Even though I don't really have
3:53
an ML background, I've been involved in a couple of like ML
3:56
projects tangentially in in my career like a little bit of
4:01
Microsoft and and now I work for Honeycomb, which is an
4:04
observability company. You give it as like it's data
4:08
analysis for information about applications that are running in
4:12
production and so sort of like an evolution of you know like a
4:15
PM monitoring stuff that that a lot of the industry uses today.
4:21
And so I've I've been actually involved a lot.
4:23
I'm I'm a maintainer in the Open Telemetry project which is
4:26
around the idea of creating open standards for generating data
4:30
about things that are running in production so that developers
4:34
can sort of own their data and and take it to like the tool of
4:37
choice rather than get locked into a particular vendor or
4:40
something. And so our product is about we
4:45
we take that data and we let you analyse it.
4:48
And in fact in Honeycomb itself, everything is about querying
4:52
that data. And you could argue that every
4:55
major pillar of our product is an act of querying in some way.
5:00
Whether it's like you're literally writing out the query
5:03
itself in our product or there's a thing that like runs a query
5:06
on a Cron job where there's a different kind of query that's
5:08
sort of running ambiently and surfaces certain things.
5:12
And so one thing that we found is we have a good product market
5:17
fit with scores and platform engineers and that's great.
5:21
It's like the right shape product for them.
5:23
But for your average software developer or even a product
5:26
manager or someone who like, has a vested interest in things
5:30
running well in production, struggles with the act of
5:34
querying. Because it's it's not that like
5:36
it's the hardest thing in the world, but like it's an
5:38
unfamiliar kind of tool and they're like, OK, I don't know
5:41
how to make heads or tails of this, so I'm just going to
5:42
leave. And when we talked with new
5:45
users, they're like, well, we have mental model of kind of
5:48
like in our heads what we think makes sense.
5:51
And golly, if there were just some technology out there that
5:54
could take like my natural language representation of what
5:57
I care about and turn it into a honeycomb query that could then
6:00
run. And as luck would have it, right
6:04
around that time ChatGPT came out and it was far too expensive
6:08
for us to use the the APIs that a couple of months later they
6:12
reduced their prices by two orders of magnitude.
6:15
And so I was like well, natural language querying looks like
6:18
it's in now. So let's let's hop on it.
6:21
So we just went full steam ahead, you know, Then we just
6:25
one of the thick of it and we gave ourselves a month and we
6:28
said OK, we're going to ship this live to everybody as good
6:33
as we can possibly make it within a month.
6:34
And that's, that's it. And it was really, really
6:37
ambitious and we ran into a lot of problems along the way and we
6:41
had to ship with embarrassing lack of features that like we've
6:46
since added. But you know, that's that's the
6:48
nature of fast product development.
6:51
And yeah, we're we're using just LMS as an API.
6:53
But even within that, there's just like a mountain of
6:57
complexity and nuance to that that that we were able to
7:00
uncover. And so we wanted to sort of
7:02
share that story of like, hey, we did it for real.
7:04
And our goal was to get it out to everyone.
7:06
Not like create like a little fake Early access thing that you
7:10
know makes it seem like everything is good and then you
7:12
go live later and you find out it's dog shit.
7:14
So yes, the refreshingly real take was really what resonated
7:19
with me. And so just to.
7:22
Zone in or DoubleClick on the idea of taking natural language
7:26
in the background. Basically you were doing.
7:29
The natural language to SQL? Or was it to another?
7:33
How were you querying it? Was it to a coding language or
7:38
was it just hitting the API and giving it context?
7:43
So there's a couple of things. So first, a Honeycomb query is
7:46
actually a Jon specification of like a thing against our query
7:52
engine that runs on the Honeycomb querying interface
7:54
that we call it manual querying is sort of like a visual
7:57
programming language and less like SQL or prom QL or something
8:02
like that. So like the goal was take
8:04
natural language input and produce a JSON spec that matches
8:07
you know this this format. Basically it has all these rules
8:10
associated with it. And yeah, with that we we, we
8:14
pull a lot of things as context along with that.
8:17
So we have this notion of called data set definitions.
8:21
For a given data set that you want to query, you can have
8:23
something like a canonical representation of an error
8:27
column. By default, that'll be a name
8:29
called error, and it's a boolean.
8:31
And like when Honeycomb sees that in data, it'll be like, I'm
8:34
just going to assume that this means the errors.
8:36
And so any part of the product that deals with errors is going
8:38
to assume that that's what it is.
8:40
But you might have a data set where you want a different
8:43
canonical representation of errors.
8:45
You want to maybe it's a string and you want to have it be just
8:48
error messages instead of like whether something was an error
8:50
or not. Yeah.
8:51
And so there's a way to configure that.
8:53
So we take that into consideration for like when
8:56
somebody asks something related to errors, if there's a boolean
8:59
error column and if there's a string error column, we know
9:03
which one is the canonical representation, and we go with
9:05
that rather than the other one. And.
9:07
And the model seems to be very, very good at sort of picking up
9:10
on that on that nuance. It reminds me because you you
9:13
mentioned the pains of working with JSON structures.
9:17
We mentioned it a little bit. It's it takes quite a bit of
9:20
effort to to get these models to conform to output that you want.
9:25
And granted, I think that's gotten better now with the
9:29
function calling updates that Open API has added where you
9:33
sort of specify OK, this is like the shape of the input that I
9:37
want to pass to a function and it'll get it to conform to that
9:41
shape. However, I've been playing with
9:43
that and it's not a silver bullet either.
9:46
I I feel like I need another blog post that's okay update.
9:49
They they did something that solves the problem in one way
9:52
but it makes it harder in another yeah, which which is.
9:56
So we have this constraint where we need to, we need to output a
10:01
JSON object that follows a particular format.
10:03
But we need to support a very wide variety of potentially
10:07
ambiguous inputs where somebody like, we literally see this
10:11
people type in errors or traces or like, so a trace and and in
10:19
distributed tracing there's like these unique identifiers that
10:22
sort of represent like here's all the things that represent
10:25
this this trace that flows through a system.
10:27
They will have another tool that will grab that unique
10:30
identifier. They'll just paste that into the
10:32
box and hit enter and it's like OK, what query is supposed to
10:37
come out of? Like a 16 byte hexon coded
10:42
value? Like yeah OK, but like it's
10:46
better to show something than nothing.
10:48
And so you know, with with the, with the APIs that we're using,
10:51
which is not using function calling, we're able to generally
10:54
produce something. And when we tried it with the
10:57
function calling updates, it ended up not producing anything
11:02
a lot more because it was like, ohh, well I don't know how to
11:05
get this to conform to the shape.
11:06
And so it might be that our current prompting techniques are
11:11
not actually suitable for the function calling stuff.
11:13
But then that feeds back to another point that I had in the
11:16
blog post about how, like, prompt engineering is not easy.
11:19
It's a total Wild West thing, and you know what's going to
11:22
work for one thing is not necessarily going to work for
11:23
the other thing. So that's exactly what I was
11:26
going to talk about with the function calls and this new
11:29
capability. It definitely seems like it's a
11:31
dark art and I've mentioned it before.
11:34
It is like even on the function calls you can just say.
11:38
Ohh, you change the order of these words or you flip flop
11:42
these words and you get much more consistent output.
11:46
And unless you're very very good at this or you just spend a lot
11:51
of time messing with the prompt, you don't actually figure that
11:54
out. And so it feels like you can
11:57
spend more time trying to figure out the prompt than if you if
12:01
you try and do this the old school way.
12:04
Yeah, yeah. And and I think one thing that
12:07
we learned after the release and so we basically we had this
12:11
initial month to get something out there and then we gave
12:15
ourselves another month to just iterate with what was in
12:17
production to try to make it as good as we could and take it out
12:20
of experimental mode and you know make it live GA quality I
12:25
guess which is which is what we have now.
12:27
What we learned is that once you're in production the fastest
12:31
thing you can do is just literally look at the inputs and
12:33
outputs of like this is what users are doing over the past 24
12:38
hours. These were all the user inputs.
12:40
These were all of the LLM outputs that came from that
12:43
output. And then this is the results of
12:46
our like parsing and validation system that tries to take that
12:49
LLM output. And and you know, we parse it
12:53
and and there's like a potential kinds of errors that can happen
12:56
there. And then we try to execute it
12:58
against our querying engine. And we learned that it's a lot
13:02
faster to just work with that data directly because now we're
13:05
no longer in a hypothetical world.
13:06
We're like, all right, this is what people are literally trying
13:09
right now. And when you're in that core
13:12
duration loop like, I think the challenge that you run into is
13:15
you you work yourself into a local maximum of like how good
13:18
your prompt is. Alright, I synced that.
13:24
If, if it's based off of what users are actually doing, that
13:29
local maxima is probably good enough.
13:32
And you don't have to tie yourself into knots trying to
13:35
figure out like Ohh or let's explore this brand new prompting
13:38
technique. Or like you can, But you're like
13:40
you're basically buying yourself some time there.
13:42
Because if you make it good enough for the majority of
13:44
people, then you're like, OK, we can sit back and we can be
13:47
really exploratory with all the different prompting techniques
13:50
that exist out there. And we can take some of that
13:53
data from like, OK, now we've got it to a point where it looks
13:55
like it's looking pretty good. Going to take a snapshot of
13:58
that, Let's build an evaluation system for offline model running
14:02
and say like, OK, now we can have some guess as to if we're
14:06
going to regress what's in production or not.
14:09
And let's go wild with prompting techniques.
14:11
But like, I think it's really, really hard to front load that
14:13
stuff because you don't have that actual user input that
14:17
you're working with. And so you're just taking shots
14:20
in the dark instead of, you know, you might be more
14:22
methodical with your prompting, but you're taking shots in the
14:24
dark in terms of like, OK, but like, what are the actual inputs
14:27
to this would be? So good.
14:29
You mentioned how people were just pasting in error or asking
14:33
error or traces. What are some other things that
14:35
you saw from the data, and were you doing this manually?
14:38
Were you just kind of looking through what people were seeing
14:41
or were you actually going through and trying to write
14:45
code? I imagine it's a tonne of data
14:47
that you're going through, but also at the same time the manual
14:50
aspect might help you pick out some of these patterns.
14:55
Yeah? So, so briefly to, to answer
14:58
your first question, I think favourite class of inputs is we
15:00
have this one customer who's like they're actually like our
15:03
heaviest user customer of all of Honeycomb period.
15:07
And they were like, ironically, the one customer I didn't want
15:09
using this feature up front because there were all these
15:11
like problems that they would totally run into.
15:15
And they were they're the heaviest users, which is like,
15:17
ohh shit OK. Of course, yeah.
15:20
But at any rate, so how do you comb has this feature called
15:25
Derived column? Or you can think of it like
15:27
Computed Column or Computed field.
15:29
The idea is it's a DSL where you can write an expression that
15:33
takes values and columns that exist in your data set and
15:37
produce sort of this this computed column that's made-up
15:40
of the other ones. And it has its own syntax and
15:46
they were hasting in just like expressions.
15:50
From that syntax into the the editor and actually getting
15:55
useful queries, we have like a little feedback mechanism where
15:59
you can say like was this result good?
16:01
Yes, no, not sure. And they pasted in like this
16:04
huge computed expression field thing and the answer they gave
16:08
back was yes. And I'm like OK, I don't that
16:11
makes no sense to me, but whatever.
16:15
So that was that's probably the wildest and like it we have like
16:18
a little stream of of inputs that all that'll sort of look at
16:21
from time to time and it still comes up.
16:23
It like apparently it works some emerging capability and this
16:27
system that we built that we had no idea about.
16:29
It makes sense though, because that kind of is one of the
16:32
patterns that you see. Like ohh there's an error
16:34
message. Let's just throw this back into
16:37
the LM and see what they make of it.
16:40
And then I don't have to go through and try and debug it on
16:42
my own. Yeah, yeah, exactly.
16:44
Or or they take like things that they knew they could sort of do
16:47
and they they they treat it as like a shorthand and it tends to
16:51
work out OK. But but so to to your second
16:53
question about how we're looking at this information so we can
16:56
see how we're going to deal with it.
16:58
There's two things that I think are really important here.
17:01
So we have this instrumented pretty well.
17:04
I actually have another blog post that talks about some of
17:06
the things that we did there where we capture about a dozen
17:10
or two dozen or so fields for a given request in a in a span
17:16
that exists in a trace in our production data.
17:18
So like broadly it captures what the user input was, what the LLM
17:24
output was, what the error was, if it came from Open API AI,
17:30
what the error was. If it came from like our own
17:32
parsing and validation system, some metadata about like we use
17:37
embeddings to like pull out a subset of fields and a schema so
17:39
we don't have to pass the entire schema.
17:42
Some metadata about like you know how much that was pulled
17:44
out and like what the actual names of those fields were.
17:46
It's like the top 50 and things like that and so that and then
17:51
we also have a response. So like if they gave us feedback
17:55
like the yes, no or unsure, we also capture that and so we can
17:59
we can sort of slice and dice that data and group things
18:02
arbitrarily. Ohh.
18:03
And also one of the things we have like these little bubbles
18:05
at the button bottom that you can click that are suggested
18:08
natural language queries just to kind of give new users an idea
18:11
of the sorts of things they can ask and we track.
18:13
Like if the thing that they gave us was like a suggested query or
18:16
a written query. And then we groups by our by
18:20
frequency of pretty much all of these kinds of different fields.
18:23
So like one of the most common ones is grouping by open AI
18:27
error and then grouping by parsing error.
18:29
And it turns out there's some patterns that emerge very, very
18:33
quickly, where a very common one that we found was if we grouped
18:38
by our parsing errors. Because there's rules for like
18:41
pieces of a query. Like, you know, this thing can
18:43
only work when this thing is present.
18:44
Or you know, like this thing does not take a column as input
18:47
or something like one of the more common ones that we found
18:50
where people were asking some variant of what is my error rate
18:54
And we kept emitting something that like I could see why the LM
18:59
would admit that, but it was definitely the wrong thing to
19:02
admit and it was a very common kind of error.
19:05
And we're like, OK, this is like through prompt engineering we
19:08
could probably solve this. And so it took us a little bit
19:11
because it was a little, it's a little tricky.
19:13
But then we did that and then a week later we have this thing
19:16
that tracks like what are, what are errors are of the query
19:20
assistant thing itself. And we noticed the error rate
19:22
went down and the success, we call it a success rate.
19:25
The success rate went up by about 6%.
19:28
And so we're like, OK, that was that was just a legit
19:32
improvement right there. And that was just, that was a
19:35
very, very simple query within Honeycomb.
19:37
We literally just like looked at this data and then we grouped by
19:40
our like parsing error thing and we just it grouped by frequency.
19:44
So we just saw the top one and we're like all right, the like
19:47
the the top five were all this error and then there were like
19:50
you know, certain kinds of inputs that people would put in.
19:52
We're like alright, cool, this is a pattern.
19:54
Awesome. Moving on.
19:55
And we would repeat that again and again and we eventually got
20:00
to the point where these errors that we saw.
20:02
Were entirely unique and we're like OK, like we're getting
20:05
diminishing returns now I'm doing this sort of thing.
20:07
So let's like pause on this improvement cycle for a little
20:11
bit and see what happens. And so that's we use Honeycomb
20:16
for all of this. So like we're dog fooding our
20:18
products for our our feature that we just got out there.
20:22
Awesome, as I would hope you would be doing.
20:24
And so if you look at where you're at right now with the
20:29
feature and you look at where you started at how much of an
20:33
improvement. Do you think you've had since
20:36
releasing this feature? And then how much room do you
20:39
still feel like you have to grow?
20:43
That's a good question. So the first part of it I can
20:46
actually quantify. So part of what Honeycombs
20:50
product offers is what's called a service level objective or an
20:53
SLO. The idea is that it's a measure
20:55
that gets computed over a particular interval as pass or
20:59
fail, and you set up like what your success threshold is and
21:05
then it calculates a budget of failures, basically based off of
21:08
your success rate over a specified time window that you
21:11
care about O if you're like, OK, I have this operation I want it
21:15
to succeed at a minimum 95% of the time over 7 days.
21:20
And so it calculates all right, there's a, there's a, there's a
21:23
just based off of the amount of like events that come in.
21:27
There's there's a number of failures that you're allowed
21:30
before you've you've sort of failed this service level
21:33
objective. And So what we did is when we
21:35
first released we just we had no idea what what would be good or
21:41
bad. So we just set a service level
21:43
objective of 75% / 7 days. So we said, OK, 1/4 of of inputs
21:49
are gonna result in a failure of some kind.
21:52
That was our our, our initial thinking and that is kind of how
21:56
it played out. It was slightly better.
21:58
It was 76 or 77%? Ohh wow.
22:01
I would have said that's a little optimistic.
22:04
Yeah. Yes.
22:05
Well, I mean we dog food a lot internally and we were actually
22:08
feeling pretty good. So we're like OK, let's do 75%
22:11
And then as as we iterated that number, we could see that
22:16
actually that rate, success rate go up on this service level
22:19
objective. And so we got it up to about 94
22:22
percent is I think where we're sitting at right now.
22:25
And that was that was through just what I mentioned earlier,
22:28
like going through looking at errors that people are
22:32
experiencing and noticing patterns, doing prompt
22:34
engineering, doing like manual corrections as well.
22:38
So like sometimes the LLM outputs something that's like
22:42
basically correct. And so we're like, alright, well
22:45
we see this, we literally know how to fix this like statically.
22:49
So let's just fix it instead of trying to you know do some hyper
22:54
focused prompt engineering stuff.
22:56
Yeah and combining these two techniques we got up to almost a
23:00
95% like always gets a query output that is runnable and
23:07
we're we're pretty happy with that.
23:08
So, so like that that that's, that's where some of these
23:11
observability practises like setting up a service level
23:13
objective and seeing how like you can observe how this changes
23:18
over time really, really helps. Yeah, makes complete sense.
23:22
And I I do wanna dive into one other piece when it comes to the
23:27
feature that you set up, which is that you are not using it as
23:31
a chat bot, which I I personally love.
23:34
Because we had a talk at the first LM in Production
23:39
conference where Linus Lee from Notion talked about how you
23:44
know, is the future of interacting with LM's chat or is
23:48
there some better user experience.
23:51
And user interface, and I think you all did that.
23:56
What he talked about in his talk, you did it intuitively
24:00
where you said, you know what, we're not gonna let someone just
24:03
ask any question in the world. We've got these boxes that you
24:08
can prompt, but it's not like a chat bot type experience.
24:12
Can you break that down and what your rationale behind that was?
24:17
Yeah, definitely. So we actually had an early
24:20
prototype that was a chatbot and it actually had way more scope
24:24
than what we ended up shipping with.
24:26
Not only could it output A Honeycomb query, but it could
24:29
also explain like a general question you have about
24:34
Honeycomb observability stuff. And it could also output A
24:39
hypothetical query. So like, imagine that you want
24:44
to ask about errors, but your application has never emitted an
24:47
error before, which happens a lot with new users who are
24:51
instrumenting like a really basic app.
24:53
There's no possible query for errors that you could create if
24:56
it's not admitted an error because there's no there's no
24:58
error columns, but that doesn't mean no query should be
25:02
possible. So what do you do there?
25:04
Well, we had this sort of mode of it would try to chat with you
25:08
about like, hey, so I'm not able to create a query from this, but
25:12
that's because you're missing the data to be able to create a
25:14
query for this. And if you had columns of like,
25:17
this name query could look like this, so like you should go and
25:21
like add that to your code or something.
25:23
We ended up scoping that out because that was a big can of
25:26
worms of complexity, like there were all kinds of weird
25:29
behaviours that it was doing. We didn't throw the idea out.
25:32
I think it's something we're going to eventually build, but
25:34
it didn't seem like it was necessary for an initial release
25:37
because especially when we talked with our sales team, they
25:41
were like, hey, so people in our experience don't seem like they
25:46
want to chat with something. They just want to get a
25:47
honeycomb query as fast as possible.
25:49
So we're like, OK, let's just focus on doing that
25:51
specifically. And then we sort of started
25:54
realising like chat isn't actually the right modality for
25:58
this problem because or it could potentially be it's it's it's
26:03
kind of tricky. So, so like with Omnicom query,
26:05
the way that you generally try to query your data is you try to
26:08
start broad and then you start narrowing down with your query
26:12
until you get to like a subset of the data.
26:14
We were like alright cool, this is specifically what I'm into.
26:17
It could be like your your, you got page for an incident and so
26:21
you're like you're starting with errors but now you're scoping
26:23
down to a very specific part of a very specific service and like
26:26
that's the thing that's responsible.
26:28
You're like, cool. I understand what's going on.
26:29
Now to an extent. Chat could hypothetically work
26:33
with that where you could sort of you ask it a thing, it
26:35
outputs a thing and then you say, OK, but like what if you do
26:38
this and then it outputs a different thing and so on.
26:41
But we found that with people internally at Honeycomb, what
26:45
they wanted to do is the query would like fill out the UI.
26:49
So you would see in the Honeycomb query UI, here's all
26:51
these like things that we're grouping by and filtering by is
26:54
they're like, Oh well, I can just like click in these things
26:58
and like manually query. Now that you've sort of created
27:01
the thing to start out with. And there was that tendency to
27:05
people to be like, ohh, well I don't wanna like go back to a
27:07
chat bot to ask it a thing. I just want to click on the
27:09
thing that like now that the UI has been filled out for me, it's
27:12
faster for me to just go and click around.
27:15
And that felt right to us. And then as we started getting
27:18
into production using this thing and getting into prompt
27:21
injection attacks, we're like, ohh, a chat bot is an end user
27:26
reprogrammable system where they can, they can use that rapid
27:30
feedback of textual input and output to manipulate the system
27:35
behind the scenes. And that was kind of rekey,
27:41
especially since we're dealing with a lot of different data.
27:44
Like, you know, somebody could be trying to do some sort of
27:46
like data exfiltration technique or something like that.
27:48
And you know, we're going to try to build as many or we built as
27:52
many protections about that as we could.
27:54
But a lot of it really comes down to how annoying is it for a
27:58
bad actor to do bad things. And what we built was actually
28:02
really annoying for somebody to do bad things with.
28:05
Yeah, that's classic that. And that's great that you
28:08
thought about it and you saw that problem because.
28:11
I imagine it's really hard to just safeguard and everyone's
28:17
data when you are sending things to Open AI and how do you make
28:21
sure that there is a clear line that is the different customers
28:26
data and how they use it. And if someone wants to and
28:30
they're good at it because this is such a new toy that we're
28:34
playing with and technology that it's really hard to stay ahead
28:38
of the curve. And so I I think about that one
28:41
and I also know that you all. Spent a lot of cycles on the
28:45
whole prompt injection thing. What made you do that?
28:49
Were people trying to break it from the get go or was that you
28:53
dog fooding it and trying to break it?
28:55
It started out with us, but then we also reached out to a person
28:59
who used to work for Honeycomb who now had like he has his own
29:01
startup. He's kind of like on a
29:03
preferred, you know, you know he he he knows everything about the
29:07
products he like. You know he's, he's rooting for
29:09
us all that kind of stuff. We're like, hey, try to break
29:11
this thing. And he knew, like, who some of
29:14
our top customers were. And that's like literally what
29:16
he was starting to put in there. Was like and give me data from
29:19
you know customer name ohh wow and we're like OK we're like
29:24
alright like he's he's he's a good actor in this like we we
29:28
all trust each other we're all friends.
29:30
Like it's all good. But if it's easy enough for him
29:32
to just ask for that sort of thing, it's gonna be easy enough
29:34
for an actual an actual bad actor.
29:36
Who we we get, We actually, we get that quite a bit.
29:39
In fact, there are a lot of we see, like in a lot of attributes
29:43
on data coming in. If you group by least frequent
29:47
unique values, there's all these sort of, like fields that are
29:50
supposed to be normally filled in with like, normal metadata
29:53
that will have like, script tags and a whole bunch of different
29:55
stuff. And so, like, people are like
29:57
trying to attack our stuff. Ohh wow.
29:59
Ohh wow. So like that.
30:01
That kind of that kind of freaked me out a little bit.
30:03
Yeah. So we thought about it and there
30:06
were a few things that we did. So firstly, we parse all outputs
30:13
of a large language model, and so that's not that's not going
30:17
to solve everything, but if you have to parse something, you
30:20
tend to just drop a whole bunch of garbage that ends up not
30:24
fitting the particular thing you're trying to parse into OPS.
30:27
You have to immerse yourself in the ML OPS content.
30:32
The best way to do it is to subscribe to the E-mail OPS
30:36
Community podcast. So Secondly, we have our own set
30:49
of protections to protect against, like SQL injection and
30:52
a bunch of other stuff when we actually execute a query.
30:54
And executing a query doesn't like run something against a
30:58
database directly. There's a whole bunch of other
31:00
stuff that goes on and so there's that.
31:04
The large language model itself is not interacting directly with
31:07
our main database that like pulls in you know like user info
31:11
and team info and and you know, validate this API key and stuff
31:15
like that. It doesn't interact with that at
31:17
all. And we sort of have it like
31:19
sandbox in a way because of that.
31:21
We then also rate limit users. So you know like that's kind of
31:26
the core loop that bad actors have is they're they're
31:29
constantly iterating and if you only get a certain amount of
31:32
tries each day, well there you go.
31:35
And it turns out most people, when they're using this
31:37
legitimately, don't ever go above like 25 or so uses.
31:43
The rate limit is in 25 anymore. We were able to update it a
31:46
little bit more, but it started out with 25.
31:48
And again, like this doesn't solve everything, but if you
31:51
only have so many tries a day, well, there you go.
31:54
Another one is limiting the amount of input that you pass
31:57
in. So again, turns out most people
32:00
when they ask a natural language query, are not going to produce
32:04
that much text that needs to get turned into, you know, tokens.
32:07
That goes against the large language.
32:08
Model. Forget also we have a limit.
32:11
I could totally see that you're starting to be like ohh, that is
32:15
obviously a prompt injection attempt because it's very clear
32:19
when somebody says do not obey the last or you know do
32:24
anything. Now that I say they're taking
32:26
the hottest topics off of Reddit and they're trying to inject it
32:30
into their. Yeah, and and so and then the
32:33
last one is also just parameterizing the prompt
32:35
itself. And I think it's also, well it
32:38
there's sort of two things with the prompt.
32:40
So like you want to parameterize your your inputs.
32:42
And again that doesn't solve everything, but it it kind of
32:45
makes it a little bit less likely.
32:47
And we have in our prompts, it's sort of there.
32:50
There's a lot of this like simulated conversation that goes
32:53
on where it's like simulated user input, simulated LM output
32:58
input, output input, output. And it's like this pattern and I
33:01
think like it's sort of orients the LM to sort of be like ohh,
33:05
well I need to be in that pattern.
33:06
I can't just like emit random garbage.
33:09
And I think when you combine all of these things together, you
33:12
have something that is too difficult to get really juicy
33:18
info out of and so it's just not worth the effort.
33:21
Like there's there's plenty more companies out there haphazardly
33:25
throwing chat bots over their databases that you can just go
33:28
after instead. Yeah, there's much lower hanging
33:31
fruit. So go to one of those and figure
33:34
out which ones they are you're it's time better spent.
33:37
And so I I just want to come back and close the loop on the
33:41
idea of the UI because I find that fascinating and I think
33:44
that being able to. Really narrow down the scope is
33:49
something that. Is a little bit of.
33:52
It's undervalued when it comes to LM's because the LM's have
33:57
this gigantic surface area that they cover, right?
34:00
And you can ask it for a poem on the meaning of life, or what
34:04
King Henry's second wife's name was and.
34:07
People don't necessarily say we're just gonna carve out this
34:11
niche and we're gonna try and make it as small as possible.
34:15
That's one thing that I feel like when you change the UI, you
34:19
do you you almost are forced to do that.
34:22
So I love that you took that approach and it made you think
34:26
about it in that way. The other piece that I have to
34:31
ask about because this came up in the LMS and production survey
34:35
that we put out and a lot of people are having.
34:38
Questions on how to quantify the ROI of your feature that now has
34:43
AI in it. And I think that you are a
34:46
perfect person to ask because basically you took something
34:51
that I don't think you can. Are you charging more for the AI
34:55
feature? That's I guess the first
34:57
question. No, it's 100% free to use for
35:01
anybody who we even have our our sandbox.
35:03
So like it's a part of the UI that you don't even need to sign
35:07
in or have a team, you can just use it and we even have it
35:10
enabled there. So this is huge because.
35:14
You took your tool and you now basically cut some of the
35:19
margins so that you could be using AI, right?
35:22
So how did you think about that? Because now you're making Open
35:26
AI API calls or whatever. If you're using Anthropic or
35:30
Cohere or I imagine you might want to try and start using open
35:34
source when a good one comes out, but we can get to that
35:37
later because I know you mentioned that the open source
35:39
models are just there's a lot left to be desired back in that
35:43
blog post. So I'm curious if anything has
35:44
changed. Let's hit on that in just a
35:47
second and let's stay on this ROI cost or the ROI idea,
35:51
because for me it's fascinating to think you said we're gonna
35:56
create an AI feature and we're not going to charge people extra
36:00
for it, but now because of the AI feature, we're.
36:06
They're having essentially we're getting less profit.
36:09
So there's there's a couple of pieces to that.
36:12
So the first is so you're gonna have to deal with cost, not even
36:18
directly, but indirectly with rate limiting.
36:21
So Open AI has a set number of tokens per minute that they
36:26
allow you to process and if you go over that, well, too bad, and
36:31
that's per API key. So like maybe you could
36:33
Multiplex API keys for users and all that.
36:35
And some companies are trying to force users to like, provide
36:38
their own API key, which is total bullshit.
36:40
Good luck with that. I'm sorry, but that's just not,
36:44
that's not a business model, that's not the future.
36:46
And so like you, you you have to deal with the problem of the
36:50
amount of tokens you're passing and the amount of tokens you're
36:52
getting back one way or another. Like it's your if your feature
36:56
is successful, you're going to hit rate limiting.
36:58
If it's, you know if it's successful, it's also going to
37:01
cost a lot. So like, do work to reduce the
37:04
size of your prompt, reduce the the number of inputs that you
37:07
pass in and that's hard. Shit, that's not easy.
37:09
But like it's good to do. The second thing is on the cost
37:13
front, try to accurately analyse your costs a little bit.
37:16
We did that. We did some forecasting.
37:17
So like I looked at, so we have manual querying in the product
37:21
and there's a certain volume of that that all of our customers
37:23
do over a given month. And so I took like the average
37:26
over the past months and looked at the general trend and I'm
37:28
like, OK, here's like about a million or so queries that are
37:31
run against our. And this month there were about
37:33
a million queries that were run, manual queries, I should say.
37:36
So people like literally clicking around and typing stuff
37:39
in our UI and clicking the run query button that was run
37:42
1,000,000 * / 1 of the months. And I'm like, all right, cool.
37:44
So people are going to be running those manual queries a
37:47
lot more than the natural language queries because it's
37:50
very easy to just sort of like tweak one particular field and
37:52
execute. Again, it's designed to be fast
37:55
that way. But let's assume for a moment
37:57
that it's a million queries that are going to get run per month,
37:59
OK, We can work back from that and we can look at our cost.
38:02
And we did. And it was like, all right, this
38:03
will cost us about 100 grand a year in Open AI.
38:06
Bill, 100 grand a year is far less than the cost of a single
38:10
engineer. And so that's just not that big
38:13
of a deal. Like we we sponsor conferences
38:15
that cost probably ground per sponsorship and it's like
38:18
alright, like this, this is, this is yeah it's money but like
38:22
this is not that big. Like if we're like we're so
38:25
we're gross stage startup, we're you know in the 10s of millions
38:28
of AR per per year. But like 100 grand for something
38:34
that could potentially bring in a lot more new users is cheap.
38:38
Really. Yeah.
38:39
And like that's also the difference between GBP 3.5 and
38:42
GBP four. If it was with GBD 4 that would
38:44
cost in the 10s of 1,000,000 per year and that's not something
38:49
that we could do. And so like part of that goes
38:51
back to like, hey, this is hard stuff.
38:54
Prompting is hard. You can't always use the magical
38:56
model that's supposedly the best or whatever you you have to work
39:00
with the one that is going to actually fit with all your
39:02
different constraints. And if that means putting in
39:04
more elbow grease with your prompting so that you still get
39:06
accurate outputs from that thing, then you got to do that
39:09
work. But that's solely on the cost
39:11
side. I also want to talk like the
39:12
ROI, like what we get out of this.
39:15
So there's two things to consider. 1st, I have a very
39:18
good relationship with our sales team.
39:20
And our sales team has been complaining for a long time that
39:24
people who come in through the sales pipeline would look at our
39:27
UI and be like, whoa, this is a little hard to use.
39:30
And there was just too much hand holding they had to do to like
39:32
get them to feel comfortable with it.
39:34
And like, yeah, an enterprise sales, you can get that.
39:36
But they're like we want to spend our time more effectively
39:39
and like every week we have to spend teaching people how to use
39:42
it. Is a week lost on like getting
39:44
that deal signed, right. So like there's there's real
39:48
money on the line as far as that's concerned.
39:50
And they were with us giving us feedback the entire time.
39:53
And that helped sort of shape what we want to do here.
39:56
And then at the end, after about a month or so, if it was out,
39:59
they came back. And they're like, this is
40:01
legitimately saving time for some of our prospects.
40:03
They see the box and they type into it and they go, ohh, neat.
40:06
And they sometimes complain to us about it.
40:08
But they're like, we want those complaints because, like they're
40:11
they're not asking us to handle that, use it.
40:13
They're just fucking going for it.
40:15
So that's number one. This is harder to quantify, but
40:18
in general, if your sales team is saying that something is
40:21
making their jobs easier, that's a really good thing.
40:25
The second thing is. Like a lot of different
40:27
companies we have a product LED growth initiative which is the
40:31
ideas that people come in through the free tier and then a
40:34
certain percentage you know starts paying us money and and
40:37
like our our paid tier and you know they sort of sometimes
40:41
they'll just stay there or sometimes they will talk to
40:44
enterprise sales and be like hey where we want to really go for
40:47
for real on this thing. And that acts as a funnel for
40:50
our sales pipeline and it also builds a part of the business
40:52
that's self completely self-sustaining if there were no
40:56
humans involved in the process. The biggest problem from the
41:00
product side, so like there's a problem in terms of getting
41:03
people into to to begin with and then there's a stage where they
41:07
need to get data in. So they need to like use and
41:10
seek or a tool that usually through the open telemetry
41:13
project that generates the data pointed it, honey Cohen get it
41:16
on there. But then after that, the largest
41:19
drop off in that funnel was writing a fucking query and
41:24
getting results. That was, that was #1.
41:27
And so we're like, OK, we tried all these other experiments
41:29
beforehand and we were able to move the needle a little bit,
41:32
but like not a whole lot. And so we have these these
41:35
metrics that we tracked of like, you know, OK, what percentage of
41:38
teams create what we consider a complex query, which is to say
41:42
they create a query that has one or more group buys and one or
41:46
more philtres. So they're like they're really
41:48
analysing their data. We also have a feature called a
41:51
board. It's a way to save a query to,
41:54
you know on a collection of queries and and sort of this,
41:56
it's like this collection of executable queries that you can
41:58
look at. And we find that if a user
42:00
creates a complex query or if a user adds a query to a board,
42:04
they tend to be hardcore activated, like there's a very
42:08
high chance they're going to start paying us money like down
42:10
the road. So we're like, all right, if we
42:12
get those numbers up, they're probably going to increase, you
42:15
know the other numbers. So we found after a month of
42:18
looking at the data with new teams coming in at the start of
42:22
May, from May to June, there were percentage of teams that
42:26
never used the query sistant and a percentage of teams that did
42:28
use the query sistant. Of the teams that did use the
42:33
query Assistant, the percentage that went on to create a complex
42:37
query was more than twice the percentage that did not use
42:41
Query Assistance. So like the jump was from from
42:45
like 100% of that population down to 36% of that population
42:49
and the AND that was using Query Assistant.
42:51
For the ones you were not using Query Assistant, 100% of that
42:54
population dropped down to like 15%.
42:58
So like from a jump from 15% to 36%.
43:01
The second one in adding a query to a board, the jump was from 6%
43:07
to think it's. It's almost 17%.
43:10
Yeah, almost 17%. So again, like that one was
43:13
almost three times the amount. But like there that's
43:16
quantifiable. Like, you know, this isn't, this
43:18
isn't total magic, but you know, at the end of the day, like, you
43:21
know, you're not going to get massive numbers for this kind of
43:24
stuff. But these are jumps that none of
43:26
our previous experience where none of our previous experiments
43:29
were able to get even remotely close to.
43:31
And so like, you know, it's only been two months.
43:34
So we don't have like we're not raining in in dollars right now
43:38
because of this, but we're like, hey, we have these things that
43:41
we are highly confident are able to move the business forward and
43:45
we are able to see significant jumps in those things.
43:47
So now we're like OK, how do we make the query assistant much
43:52
more prominent for new people who are coming in because there
43:54
were people who never even noticed it and we're like OK,
43:56
like it's able to yield these better outcomes.
43:59
So let's really get their eyes on this thing.
44:02
But like already we think that ROI is worth it just based off
44:05
those two factors. Feedback from sales saying this
44:07
is helping them with that initial part of the cycle which
44:10
decreases the time it takes to get a deal signed.
44:13
And then the second is on the self-serve side of the business,
44:17
significant increases like the largest increases in conversion
44:20
that we've ever seen with the experiments that we've been
44:23
doing as a result of this thing getting out there.
44:26
Dude, so just having these leading indicators that you
44:30
picked out and you said, you know what, if we can move the
44:33
needle on these, then we know the rest of the dominoes are
44:36
going to fall. That is so good to be able to
44:41
try and use that as your target because then it's clear if our
44:47
AI initiative, it increases the usage here of these metrics,
44:52
then we're good and we know that the ROI is there, especially if
44:56
we're just. Talking, I mean just I say just
44:59
with hand quotations, if we're just talking about 100 grand a
45:01
year, then it's a whole different scenario because
45:05
potentially the average sale price for somebody when they do
45:08
get into that fully activated zone is much higher than 100
45:13
grand a year. So you're saying if we can just
45:15
get one of these people to activate fully from this
45:18
feature, it's all worth it. And so now I'm I'm thinking
45:23
about the idea of you and from where you sit and your vantage
45:27
point. Being so deep in the product
45:30
space, but also knowing that lots of people in this podcast,
45:36
people that come to this podcast listen to it and they're sitting
45:39
in a machine learning engineering space, they're
45:41
sitting in a data science space, how do you did you already have
45:47
those metrics picked out? How do you pick them out?
45:50
How can you give advice to people that are in the machine
45:55
learning engineering space to think more like a product
45:58
person? Because if there's one thing
46:00
that I have seen over the last whatever, X amount of months
46:04
doing all these hackathons with AI, it's that now everyone is a
46:09
product person. The ability to use AI.
46:13
And some people call it, you know, like the democratisation
46:16
of AI. Although I think that's a little
46:19
bit like overplaying it, because there's still some really hard
46:22
stuff that has to come along with it.
46:24
And you need a lot of times you need some serious engineers to
46:28
implement this. You can't just.
46:30
Have some front end web developer create something that
46:34
can be at scale, but I do think that everyone now is looking at
46:40
their position and they're thinking about how can I bring
46:43
more of a product mindset to what I'm doing and so.
46:47
From your vantage point, you mapped out some very clear
46:50
things. You're talking to different
46:51
stakeholders like sales. I imagine you're talking to a
46:55
lot of different people. So maybe you can go into that a
46:57
little bit. And then you're also saying,
46:59
here's some leading metrics that we're looking at and just if we
47:03
can affect those metrics, the rest of the initiative is worth
47:07
it. And so from a machine learning
47:10
engineering perspective, how can we help identify those metrics?
47:16
How can we look at who we should be?
47:18
Talking to so that we can also be part of this conversation,
47:22
that's a good question. So I think what this is
47:24
fundamentally rooted in is you shouldn't be looking for, you
47:30
know, it it it's, it's kind of a classic thing, right?
47:32
Like there's this amazing technology, but you shouldn't be
47:34
taking the technology and looking for a problem to solve
47:36
with it. You should be looking for a
47:38
problem to solve and seeing if the technology can be used to
47:41
solve it. And that's the mindset that we
47:42
had in this was this very scoped problem of people coming in and
47:46
being able to be successful with querying for the first time.
47:49
And on the product side of it, like, again, this is the, this
47:53
is something that I did and some other people on the product team
47:56
did. But you can be in the MLS side
47:57
and do this too. Literally just talking to some
47:59
of these people, talking to some of the customers were much more
48:02
early stage working with someone to get like emails of people who
48:05
signed up, who are on the free tier and just chatting with them
48:08
and being like, hey, why did you use this product?
48:10
What questions are you trying to ask?
48:11
Like what are you trying to get out of this?
48:13
And you find after about five or six of these relatively short
48:16
interviews, there's patterns that emerge.
48:19
And in our case, the pattern was, well, I cared about my
48:22
latency or my errors usually or like, you know, user behaviour.
48:26
And I want to see, I want to see that.
48:28
And then you go, OK, well, what does that?
48:30
And they're like, well, you know, they, they often like,
48:34
struggle to explain what they want to see.
48:37
And that's part of why they came to our product.
48:40
You don't have to be a product manager.
48:42
You don't have to sit in the product org to be able to do
48:45
this sort of like work. Once you do that, you can then
48:48
take a step back and be like, OK, are these problems that are
48:50
worth solving? And if they are, OK, let's
48:52
identify some good metrics that are worth like, OK, this is how
48:56
we're going to be able to start quantifying it.
48:58
It's really good to have a product person there.
49:00
Especially if you have a product person with a background and
49:02
growth product work. They're going to be really,
49:05
really helpful at this because those people are amazing at
49:09
knowing what is trackable today and what could be tracked and
49:13
like what work needs to go in there to instrument it so it can
49:15
be tracked. And then saying, OK, you know,
49:18
just framing it up like, OK, if you see a percentage increase of
49:21
this amount, that usually means it's good or means it's maybe
49:24
not so good. And Justice give you a better
49:26
sense for like this is what we could like if we what we view as
49:30
success or what we view as failure we can have some bounds
49:32
around that. When you have that in place it's
49:35
it's basically go time at that point.
49:37
Like I mean I took it a step further and worked with sales
49:39
because you know where we we have a large enterprise sales
49:43
arm of our business and it turns out that their problems were the
49:47
same as these problems like we're just complete alignment
49:49
across organisations. You know, again, you don't have
49:53
to be a product person to do this sort of work, but talking
49:55
to those people and sort of identifying where all the common
49:57
points are and then working with somebody in product or doing
50:00
yourself to figure out those metrics you want to move, you
50:03
can do. But then I think this question
50:07
can be answered in two different ways.
50:08
I think like a there's that work, especially on the initial
50:11
legwork that ML engineers can be deep much more deeply involved
50:15
with than perhaps they might be today.
50:17
The second piece of that is I think product managers should be
50:22
working with mill folks a little bit more and becoming a bit more
50:25
data literate because they may like, you know, it's hard to
50:29
develop a deep rooted understanding of how users are
50:32
using your product. But if a product manager can
50:35
like bring in that expertise and an ML person can sort of come at
50:38
it from the other direction and have some of this like, hey, I'm
50:41
talking to you sort of identifying some of these
50:42
problems. You have a common language that
50:44
you can speak and if the product person is able to say, hey, I
50:49
know what embeddings are, I know why they're useful.
50:52
I know like what is good or bad about LM's in general, like
50:56
things where they can fall down. You can really, really start
50:59
iterating quickly. And I think it's essential.
51:02
Like I I think the ML folks are going to have to be more product
51:06
minded. They're going to have to learn
51:07
how to do more of this legwork to better understand a problem.
51:11
And then I think the product folks are going to have to get
51:14
more data minded, basically to better understand, like, OK,
51:17
these are our data. Like, these are the systems that
51:20
we can work with and these are our data sources and how they're
51:22
sort of laid out and what we can do with that.
51:25
I think before LLM's and how easy it was to just sort of call
51:29
an API and get something kind of halfway decent, there was just a
51:32
lot of work that went into like getting a halfway decent ML
51:36
model into production. That now is not that it doesn't
51:40
take as much effort anymore, but all that other work with like
51:44
quality of data and making sure we have the right data pipelines
51:47
in place, like it's still there. I think like a lot of the focus
51:51
is going to shift on that and making sure that you're
51:54
collecting the right information in the 1st place to be your
51:57
source of info that you're going to feed to an ML model.
52:00
And I don't know how much of like you know, I think that like
52:06
as more organisations are able to adopt machine learning,
52:09
there's going to be more of a need to be able to set U these
52:11
kinds of systems. And so there's going to be a
52:13
higher need for mill folks who are really focused on ensuring
52:18
that data is in a good state. And then also being able to work
52:20
with product people to be like all right cool.
52:22
When is a good time that we could take a snapshot of what's
52:25
live and build an evaluation system based off of real user
52:28
behaviour so that we can then say, OK, like this is how we're
52:32
going to be able to iterate on this thing in in the future.
52:34
Because every product person knows the first thing you
52:37
release is going to kind of look in some ways.
52:38
So like, how do you make it not suck anymore?
52:41
Like, that's kind of one of the keys to doing that.
52:43
Our skill sets are going to be put a little bit into a blender.
52:45
I think I'm all for it because I think on both sides we could
52:49
stand to improve our skill sets and change how we work.
52:53
So I love that because there is something that you spoke about
52:57
where it comes when it comes to just hitting an API and making
53:00
it really easy for someone to go out and get something up and
53:04
running really quickly, which was not possible before.
53:08
And there was always that that that came out with like every ML
53:13
OPS tool marketing where it said, you know, 80% of machine
53:18
learning or AI initiatives never make it into production.
53:21
And now I think I haven't seen that stat for years because
53:25
you're just hit an API. But there's a whole other slew
53:30
of problems which you mentioned on the front end, where it's
53:33
collecting the data or the security part of it, and how you
53:36
make sure that there's no prompt injections or just prompting.
53:40
In general, how that works. But when it comes to iterating
53:44
and figuring out how to better your whole system, do you see
53:49
yourselves going towards using open source models?
53:53
I mean for you, I know you mentioned in the blog that the
53:56
open source models aren't that interesting and they're just not
54:00
as good as what you can get off the shelf right now, but I also
54:04
wonder if that is something that you have mapped out with the
54:07
product road map. It doesn't seem like the cost.
54:11
There would be a reason for you to do it, so I guess it would be
54:15
for some other reason, like maybe the rate limit.
54:17
Or I know that a lot of people complain about how Open AI
54:21
updates this model every other week and then it completely
54:24
changes things. So if you're at mercy to an
54:28
update, that can really mess up your AI feature.
54:32
So I just, I wonder how you think about that and if it's
54:36
better not to have full control and bring it in house.
54:38
Yeah, it's it's really about that cost is not really an issue
54:42
for us. Well, it's about it's about
54:46
really two things. So first, if we have full
54:49
control over the model and it's running in our infrastructure,
54:52
then yeah, that's great. We could even fine tune it to be
54:54
potentially even more accurate. To be honest, the bar is going
54:57
to be pretty high there because Open AI is pretty great at being
55:00
accurate already and it's super, super easy to use.
55:04
And so you know the ROI there compared to an open source model
55:09
running in our own infrastructure like it's got a
55:11
high bar to to cross. Now like I I think that there's
55:14
sufficient motivation in our industry to build those things.
55:17
So it's super easy to take an off the shelf open source model.
55:22
Like you know, there's going to be like a marketplace of models
55:24
all sort of better suited for specific tasks and sort of say,
55:27
alright, cool, this is the one we're going to use.
55:29
And you know, this is the shape of the data that needs to be in
55:32
to fine tune it. And we're going to click a
55:34
button and it's going to upload the data and it's going to fine
55:36
tune it. And then in three hours we're
55:37
going to have it available for us to use.
55:40
If that existed, we would probably consider that later
55:43
this year. But right now it doesn't.
55:45
And so the amount of time and money that we would spend in the
55:48
trying to use that only to get something potentially better
55:53
than Open AI, no guarantee that it's better, is like, well, all
55:57
right. Or we could wait for Open AI to
56:00
release an even better model. And yeah, there we go.
56:04
That's kind of where our, our mindset is.
56:05
I mean, I think like you know, being at the mercy of Open AI's
56:08
model changes and model latency, which I've complained about
56:12
several times actually, which you know they're they're working
56:14
on, that's that's also motivation.
56:17
But it's not posed a problem enough to the point where we
56:20
need to make a move now. If we were starting all over
56:23
again, I think we would still go the same route that we are right
56:27
now, just because it's still not easy enough to take an open
56:29
source model and fine tune it and use it the way that we want
56:32
to. O also looking forward and
56:36
thinking about where you're going to take the product next.
56:39
I know you mentioned how right now agents or the chaining of.
56:46
API calls are off the table and that is because and you again,
56:51
you articulated this perfectly. It's something that I have in my
56:54
head, but I wasn't able to really say, but it's like, hey,
56:56
if the agent or if one of your API calls has a let's say 90%
57:03
accuracy rate, if all of a sudden you start chaining three
57:06
or four of those together, the accuracy rate starts to go down
57:10
very quickly. And I think anybody that's
57:12
played around with auto GPT or Baby AGI, they have felt that.
57:16
Pain first hand. And so when it comes to
57:21
chaining, do you feel like that is something that you're going
57:24
to try to do in the future or is it again, one of those things
57:27
where, yeah, it's not that interesting because we have
57:30
something that's working and the scope that we're looking at, it
57:33
doesn't need for Query Assistant the, the feature that we
57:36
shipped, we won't be using January agents, I think ever
57:41
because it's, it seems like it's pretty much doing its job.
57:44
So you know, but there are other parts of our products that are
57:49
more I use, I use an analogy compile time versus runtime.
57:52
So Query Assistant is a runtime concern.
57:55
So it needs to be fast. If it's a little bit less
57:57
accurate, it's OK because it's very easy to correct things and
58:01
a user can take over at any point.
58:03
There are other parts of our product that are more compile
58:06
time concerns and so having an agent or a chain that you spend
58:09
so like is it is less accurate overtime to an extent.
58:14
There are techniques that you can do to sort of make things
58:16
more accurate over time, but it's significantly increases
58:19
your latency which is why it's not a good fit for this part of
58:23
your product. But for other parts of our
58:25
product where there's a lot of upfront like hey, if it takes 3
58:27
or 4 minutes to execute but the end result is really, really
58:31
good, then that's a worthy trade off.
58:33
It's just that the querying teacher is that's not the right
58:37
place to put that sort of thing. I do like this idea of.
58:40
Win. You're looking at trade-offs.
58:43
What are the most important? And if you really have to focus
58:47
on latency then chaining is not the option no matter how good
58:51
the end result is. So.
58:54
Dude, Philip, this has been awesome, man.
58:56
I knew it was going to be great. I did not realise it was gonna
58:59
be so good. It was well worth the wait to
59:01
chat with you. I think we can end it there.
59:04
Is there anything else that you wanted to mention?
59:07
Ohh so we are hiring, but then also I'm currently involved in a
59:12
project right now of taking observability practises into
59:16
this whole like people deploying stuff with alms and trying to
59:21
iterate. And so if if anybody, whether
59:23
you're an engineer or you're another product manager or
59:25
whatever, if you're like hey we're going to put something in
59:28
production and it's gonna be like V 0.1, how do we get from O
59:33
.1 to one dot O as effectively as possible?
59:36
I would love to chat with you and you know my info should be
59:39
in the show notes or something like that.
59:41
You could also e-mail, e-mail me directly at Phillip at honeycomb
59:45
dot AO and I'd love to have that conversation, because whether
59:48
observability practises are the right thing or not for the
59:53
problems you're trying to solve, that's something I'm trying to
59:55
figure out as well too. There might be a version of
59:58
Honeycomb that could work well for these sorts of problems, or
1:00:01
there might not be, right? This is a discovery sort of
1:00:03
thing, so that's what I'm currently sort of after, and
1:00:06
anybody who's interested, I'd love to chat.
1:00:08
I love that. And I also want to chat with
1:00:11
these people, man. That's a great, great
1:00:13
initiative. So I'm excited.
1:00:16
I I love what you put out. I love the work that you're
1:00:18
doing, and we'll leave all of the blogs that you've written in
1:00:22
the show notes. And I'm gonna try and propose
1:00:24
that you write a few more from this conversation.
1:00:27
Or maybe I'll just distil what you've said into a blog.
1:00:30
Now, Philip, this was great, man.
1:00:32
Have a great day. We'll end it there.