Latest / Agentic Conversations (formally mlops.community) / How Sierra AI Does Context Engineering

Transcript

How Sierra AI Does Context Engineering · Agentic Conversations (formally mlops.community)

0:00
It is useful to understand energy levels and think about
0:03
when light hits atoms, how electrons will change energy
0:07
levels. But it is more accurate to think
0:09
about the electron cloud, and so that what you sort of modeled
0:13
with the web is probably more accurate.
0:15
But decision trees and flow diagrams are still potentially
0:18
useful constructs to think about how an agent should behave.
0:33
The difference about AI is that instead of the software being
0:36
super cheap, super fast, super reliable, it's actually quite
0:40
expensive to run. Kind of slow, gets slower and
0:45
slower the better the reasoning models get, and non
0:48
deterministic. So it can be creative, it can be
0:50
flexible, but it can also hallucinate.
0:53
And so the methodology that you need at each of those different
0:57
junctures in this flywheel that you're trying to build is pretty
1:01
different from with traditional. Software.
1:03
So it's like traditional software does this.
1:05
Let's just take the opposite of all of those.
1:08
A good example is like unit testing, right?
1:10
So with unit tests, you run them once if they pass your good.
1:16
With simulations, which is what we call our flagship testing
1:21
product in Sierra, we might simulate a given conversation
1:25
5/10/15 times and then you'll look at what data changes
1:30
happened as a result of the conversation, which is a similar
1:32
methodology to what we published with Tao Bench, which is the
1:36
open source framework for eval. But we'll also do using an LLM
1:40
as a judge or a third agent. So you can have the user agent,
1:44
the agent agent, and then the evaluator agent.
1:48
And the evaluator agent will say, you know, did this
1:51
conversation meet this checklist?
1:53
And that's pretty different from unit tests or even integration
1:56
tests in traditional. Software.
1:57
Do you even put that in the CICD pipeline?
2:00
You do, yes. So we have the concept of what
2:03
we call critical simulations, which are simulations that run.
2:07
You know when you actually will merge a commit to main or
2:11
schedule a release or what have you.
2:13
Feels like that could be really slow.
2:15
It could be we have a high degree of parallelism, so if you
2:19
have 300 simulations that you need to run, you're not waiting
2:22
for them to run 1 after the other, they're going off and
2:25
being executed in parallel and. When you say simulation, are you
2:29
only talking about it for voice or are you talking about for
2:32
everything? It's interesting you asked that
2:34
question with voice first. I feel like most of the time I
2:37
get that question, it's are you only thinking about it for chat?
2:40
But it is both. So it's actually pretty
2:42
interesting to listen to voice simulations.
2:45
The way that they work is we'll have a library of background
2:50
noise, library of voices, a library of all these sort of
2:54
different and Tropic things that you can introduce to try to
2:57
simulate what it would actually be like to get a phone call.
3:00
And our customers are all over the world in all different
3:03
environments and all different industries.
3:05
Exactly. So you have people calling in
3:07
from the car. You know, SiriusXM is one of our
3:11
larger customers and people will call in because their radio is
3:14
not working and they're in the car or by the, you know, on a
3:18
busy street or something. And so you have the background
3:21
noise, you have different accents, you have different
3:23
microphone quality. Not everyone's talking into
3:25
this, you know, magical podcast mic.
3:28
And so you have to deal with all that.
3:29
And our simulations have kind of gone through that punch list,
3:32
you know, row by row and said, OK, we're adding this to the
3:36
product. And then if you run it 5/10/15
3:38
times over 3400 simulations, you have a pretty good idea that if
3:44
those are all passing, your agent is doing what you would
3:46
expect it to do. Do you also look at simulations
3:49
a little bit like red teaming? Definitely.
3:52
So there's I don't know if I would say there's two classes of
3:55
simulations, but there are definitely.
3:58
Somewhere it's just the happy path, Oh, I, I need to reset my
4:02
radio cause the encryption keys expired.
4:05
And so an API call goes out and a satellite beams down the new
4:10
encryption keys. And you can listen to sports
4:12
talk again or Howard Stern or whoever, whoever you like to
4:14
listen to. But then there's also the more
4:17
adversarial simulations where you're trying to, you know, we
4:21
call it abuse the agent. And so we have a number of
4:25
custom models that are just about detecting abuse.
4:28
And those are often customer specific as well where, you
4:32
know, some customers they might have a younger audience and
4:36
things like self harm and bullying are top of mind.
4:39
Some customers might have different audience and things
4:42
like hacking into the system or top of mind.
4:44
So you have to be pretty customer specific in how you
4:48
think of it. The nice thing is this is a very
4:50
valuable problem for our customers.
4:52
And so they're deeply engaged with us and we've gone over the
4:56
last six months or so making the entire product fully self-serve
5:00
so that it's self-explanatory. You can get up to speed.
5:03
And while it is this deep thought process, it's kind of up
5:06
to you on the spectrum of how much you want to take that into
5:09
your own hands, how much you want to work with Sierra's agent
5:12
development team. How are you creating these
5:15
nefarious simulations? There are three agents involved.
5:19
There's the user agent, there's the agent itself, the AI agent,
5:24
although they're all AI, and then there's the LOM as a judge.
5:27
And so the user agent basically can take a system prompt as part
5:31
of the simulation. But you don't have to massage it
5:34
to get it to be like really bad. So I think to get it to be
5:39
really bad you would and typically we won't have the
5:45
really bad simulations as part of the the the run.
5:50
You can also because. You want to catch that before
5:53
it's out or trying to get to production.
5:55
I think the honest reason is that you typically the list of
5:58
really bad stuff, you just want to make it so not everyone has
6:02
to look at that every couple minutes.
6:05
And also, I don't think that AI models are that good at
6:10
imagining some of the terrible things that people say, probably
6:15
because, you know, the model providers have gotten out in
6:19
front of that. That being said, we also have
6:22
the concept of verbatim simulations where you can just
6:25
hard code a script of what the user agent should say it when
6:30
the AI agent says X. And so for those ones where you
6:35
really want to test against, you know, some of this behavior that
6:38
you definitely want to prevent against or, you know, prompt
6:41
hijacking or these kinds of things, you basically will do a
6:45
verbatim simulation instead of an AI user AI simulation.
6:50
Does that make sense? Yeah, the really bad stuff.
6:53
I'm not necessarily thinking really bad and like what humans
6:56
are capable of. I'm just thinking really bad of
6:58
like as a prompt hijacking type of thing where I'm just trying
7:04
to get some data out of the model.
7:07
If I know that I'm now dealing with an AI agent, let's see what
7:12
I can get from it in that way. Maybe it's naive.
7:16
I feel like the models that are out there today are not that
7:21
good at trying to like hijack themselves.
7:24
There's two things that come into play here as well.
7:27
One is that the actual access to sensitive information is always
7:33
safeguarded in a deterministic fashion show.
7:36
So we'll have more traditional unit tests that will make sure
7:39
that the access controls to a system are no different from
7:44
what you would have if you were on the website, you know, logged
7:46
in as a user as an example. So we don't leave it up to the
7:50
model in any case whether to expose sensitive information, we
7:55
leave that up to deterministic systems.
7:57
How does that work? Like the connection between
8:01
access and model. So often, let's say you go to,
8:07
I'll think like Sonos for example, and you're logged in.
8:11
The fact that you're logged in means that you have access to a
8:14
set of data in Sonos's database, maybe the speakers that you've
8:17
connected to the networks in your home, and the AI model also
8:22
has access to that same information.
8:24
So you would never be able to use the model to get information
8:28
that isn't already available to you.
8:31
You know, as a logged in user on the website.
8:33
Yeah. So it's almost like as you log
8:35
in then the model just has this sandbox of your world.
8:41
Correct. And then if it needs to go and
8:43
do something because you have a complaint, then it like gets out
8:46
of your world or it. Well, if you have a complaint,
8:50
that would be something you could do in your world as well.
8:52
So the models, they're, they're taking action all the time, but
8:54
they're not taking actions that you wouldn't have permission to
8:57
do as a user. Yeah.
8:59
And so then you're just protecting on the permission
9:01
side of things like and that doesn't seem like it's like
9:05
that's a tried and true thing that we've been doing in
9:08
software. And that didn't necessarily flip
9:10
like you were talking about back with that cycle of thing.
9:13
It's not one of those slow, non deterministic.
9:16
If you can make sure that the model doesn't have access to it,
9:19
it just gets access to what it gets, then whether or not it
9:24
gives the information is where things get wonky.
9:27
That's exactly right. And you said earlier like, oh,
9:30
so you're just kind of and doing the exact opposite of the
9:33
software development life cycle. And I kind of just took it in a
9:36
different direction because it's, that's actually not true.
9:39
A lot of the times you need to do the exact same thing that's
9:41
tried and true from traditional software development.
9:44
Sometimes you need to do the opposite and sometimes you need
9:47
more of a hybrid approach. So it's more of a first
9:50
principles analysis of each of the different steps of kind of
9:54
plan, build, test, release, optimize, analyze, plan, build,
10:00
test, release, you know, etcetera, etcetera.
10:02
But just kind of from the first principles of what's changed
10:05
using AI systems along with deterministic systems as opposed
10:10
to only deterministic systems. I wonder how different the
10:15
simulations are for voice and for text, or if they are.
10:19
I know that of course with voice you got to add in all these
10:23
special things like you were saying, like noise, background
10:26
noise. Also, I, I've talked at length
10:29
with people about conversational design.
10:31
You can't let somebody know that the models going off and doing
10:35
something or you don't want to have somebody thinking that
10:40
somebody hung up, the agent hung up because the model is off and
10:44
doing something. So you want to give like cues
10:47
and all that stuff. So I imagine that's one type of
10:50
simulation and that you're only going to find invoice and then
10:54
in text you have different types of simulation.
10:56
And how do you see those two being different?
10:59
It's a really good question and one that's very close to my
11:02
heart. So because my personal story at
11:04
Sierra was I spent the first nine months building agents with
11:07
customers. So I mentioned SiriusXM because
11:10
they're one of our larger customers that we're proud to
11:13
work with, but also because I helped build the very first
11:16
version of their agent. And then SiriusXM, you know, as
11:20
large as they are over chat, they're even larger over voice.
11:23
And so just seeing that kind of having it beat us over the head
11:27
of hey, voice is actually the huge opportunity here, you know,
11:31
LED me and a couple others from the team to start prototyping.
11:35
What would it look like to build out a voice version of their
11:39
agent and. So we iterated on that and
11:42
eventually kind of moved up the hierarchy of needs from just
11:46
being able to go back and forth to actually responding to verbal
11:51
cues and interruptions. One example is an interruption
11:56
is not actually a super objective thing.
11:59
If you start talking right now, you might be agreeing with me,
12:02
encouraging me, interrupting me, sort of telling me to go in a
12:06
little bit of a different direction, but most.
12:09
Automated systems today, if you say something they will just
12:13
stop and start over no matter what you said and so we.
12:16
So painful too as the user. Yes, yes.
12:19
And so it actually feels like they're talking over you because
12:22
you're like, you go, no, I go. And I don't know if you've
12:24
everyone's had this where you're walking down the hall and
12:27
someone's walking towards you and you're like.
12:29
Who's going to move? And that's what it feels like,
12:32
all the. Time dancing in the hallway.
12:34
And like obviously once you get so deep in the matrix you notice
12:38
everything. So as an example, the two of us
12:40
were just talking at the same time, but it wasn't a problem.
12:44
Another example is. If I pause.
12:48
For a very long time, like I just did in the middle of a
12:50
sentence. You know, to wait for me, but if
12:53
I'm done speaking, you know that it's your turn to speak.
12:56
And so context is so important, both was this an interruption
13:00
and is it my turn to speak and it you can't just boil it down
13:04
to a number of milliseconds. You need to actually think about
13:07
what's being said and start planning your response.
13:10
You know, before the other person's done talking.
13:13
Everyone almost has their cadence.
13:15
Their way that they talk is unique to them.
13:17
And so I may give longer pauses and I may speak slower and I may
13:24
not want to finish, but then actually I do finish or things
13:27
like I speak. Weird, right?
13:29
Right. And we all kind of have our own
13:32
quirks when we speak. And you can't have a universal
13:38
setting of all, right, well, the model is going to be like this.
13:41
The model almost has to adapt to each individual way of speaking.
13:45
But I haven't seen anywhere, and maybe you've messed around with
13:48
this, where you can have the model interpret the first,
13:52
whatever, 10 seconds and go, oh, OK, this person takes long
13:56
pauses in the middle of sentences, or this person speaks
13:59
a little bit slower. And so I'm going to give them
14:01
more time between words, right? It's a really good point.
14:05
At the very high level, you can do some of this for customer
14:09
specific. As an example, you know, if a
14:12
customer has mostly older callers, you might wait longer.
14:17
If a, if a customer has a lot of authentication flows, you know
14:21
that it takes a while to, you know, talk through your e-mail
14:24
or your password or these kinds of things.
14:26
So some of it is shared. That being said, I think to do
14:30
it in a personalized manner like humans do, you probably will
14:35
have the most success with some of the speech to speech models
14:38
where the audio tokens themselves are the things that
14:42
are being inferred upon. And so that actually all kind of
14:46
gets boiled down into the training data.
14:49
That being said, right now most of those systems for most
14:53
production environments are just a little bit too hallucinatory.
14:58
Our approach has been we built a modular architecture.
15:00
So on the. Transcription side and on the
15:03
synthesis side. And by the way, even if you're
15:06
using the speech to speech model, you need transcription
15:08
because you can't make an API call with a bunch of voice
15:11
tokens and so we transcriptions part of the pipeline no matter
15:14
what. But we do have support for
15:18
models like, you know, the, the model that underlies the real
15:23
time API, the GPT audio models, as well as the Gemini audio
15:27
models, which are these kind of fully speech to speech models.
15:32
And I think that's going to be the future.
15:34
It's just being honest with you right now.
15:37
The amount of hallucination that we've seen in our production
15:41
tests is such that for the larger customers, the text to
15:45
speech is still the most reliable way.
15:48
Yeah. I've heard that from just about
15:50
everybody I've talked to where it's like we want it with that
15:53
speech language model, but we can't have it because it's just,
15:57
I can't put that into production with any amount of confidence.
16:00
Yeah. And I would I would go step
16:01
further than we want it, which is.
16:03
You know that I believe in it. I think if you and I were
16:06
passing notes across the table right now, it would be a much
16:09
less interesting and productive conversation.
16:12
And that's basically what AI agents are doing.
16:14
So we're willing to make the long term investment in our
16:17
architecture where we support that out-of-the-box today even
16:20
though it's not quite ready and we're following the open AI in
16:24
particular just released a new real time audio model.
16:28
And every time one of those things happens, we, you know,
16:31
test it out, figure out are there.
16:34
Places where we could use this. And obviously there's a risk
16:37
tolerance and risk likelihood spectrum for each of our
16:41
customers. So I think we will find that for
16:44
certain languages, for certain customers, for certain use
16:46
cases. This is going to be an
16:48
absolutely magical experience and then over time it will kind
16:52
of disrupt the text to speech approach.
16:54
At least that's my personal belief.
16:55
Do you have a pipeline that you'll run models through as
16:58
soon as they come out just to see like, because obviously when
17:03
you look at benchmarks, it's just a bunch of bullshit, right?
17:06
We do. Some benchmarks are really cool
17:09
as well, like we love our Tau bench benchmark.
17:11
It's really cool when the Math Olympiad is won by an AI.
17:15
But when it comes to putting systems in production for our
17:18
customers, I think what you said is roughly accurate.
17:22
We do immediately when a model comes out, we have a whole suite
17:26
of evals that will run. But it's usually an iterative
17:29
process because one model might be better if you prompt it in a
17:33
different way or if you if you shot it in a different way or
17:36
even if you fine tune it in a different way.
17:38
So it's not you have a good instinct immediately, but it's
17:42
often a multiple days, multiple weeks process to know if the
17:45
ceiling of the new model is higher than the ceiling.
17:48
You know, the local maximum that you've climbed to on.
17:51
The previous model, so it's it's and then I would say also for
17:55
voice. Language is a huge deal as well
17:59
as like dialect and locale. So Brazilian Portuguese being
18:02
different from Portugal Portuguese.
18:05
And so you have to, we basically have a rubric and we have, you
18:10
know, basically callers that we work with who can call in and
18:13
test out each model. You know, because you need
18:17
people that actually speak the language that can be in
18:19
realistic scenarios or else you're going to most likely
18:23
overfit to some data or, you know, not be representative.
18:26
Yeah, my buddy was telling me how he was doing a bunch of
18:29
simulations on the open AI real time and 1 funny thing that he
18:35
found was when you speak English with an Italian accent or no
18:40
when you spoke in Spanish with an Italian accent, it would
18:45
switch to this like made-up language that was like a mix
18:49
between Spanish and Italian. And when you spoke when you
18:52
spoke English with a German accent, it would just switch to
18:56
German and speak to you in German.
18:58
And so it was like what is going on?
19:01
I speak in English, you know I want it to be in English but it
19:04
keeps switching to German. And at the end of the day,
19:07
you're like, but is it the end of the world?
19:09
Because if the German accent is there, that probably implies
19:14
they speak German, and so why not just speak in German?
19:18
Exactly. It's like, I don't know.
19:20
Do you speak any other languages besides English?
19:22
OK, so if you go to. A restaurant, let's say, and
19:27
someone greets you in English, but they have an accent that
19:30
suggests they speak Spanish or Portuguese.
19:32
You do you kind of feel that urge of like, oh, I want to
19:34
practice my Spanish or like, you know, could we have a discussion
19:38
in another language? Wouldn't that be fun?
19:40
And I guess that's sort of what's going on in the.
19:42
Model. It's by chance, yeah.
19:47
And on the simulation, I know that our mutual friend William
19:52
has told me about how he uses heat maps to see where the
19:57
agents are good and where they're bad.
19:59
And this is like on the task completion side of things.
20:02
Have you tried to do any of that with like, because that feels
20:05
very orthogonal to simulation in a way.
20:08
It's like you, you simulate things and you see where it's
20:11
good, where it's bad. But then if it's an agent, maybe
20:14
it's like, the next step is, can it do this?
20:16
Can I actually like complete the tasks that we ask it to do?
20:20
It's a really important piece of it is not just what did I do
20:23
wrong, but how do I do it right next time.
20:25
And I think without a clear understanding of where you went
20:28
wrong, which I think maybe what you're getting at with the heat
20:31
map, you can't really understand what the fix would be.
20:34
So to give a few examples, sometimes the agent will go in
20:38
the wrong direction because it doesn't have the right answer.
20:41
Sometimes the standard operating procedure is wrong.
20:45
Sometimes it was asked to reason about too many things at the
20:47
same time, and so you want to know those things at the point
20:51
where there's an error so that you can fix it most quickly.
20:54
And in simple cases, the agent can fix it itself.
20:58
So one example is a number of conversations from Sierra.
21:02
It's remarkably low at this point with a lot of our agents,
21:06
but a number of conversations end up getting transferred to a
21:09
human, especially if there's something that you need to do
21:13
that just requires oversight or regulated industries, etcetera.
21:17
And so in those cases though, you.
21:20
Can actually learn from what the human did post transfer and
21:24
detect? Is there anything missing in my
21:26
standard operating procedure? Is there anything missing in my
21:28
knowledge base and have AI author draft article of, hey,
21:34
you should, maybe you should add this knowledge we had, you know,
21:37
150 different agents all mention this thing, but the AI agent
21:40
doesn't seem to know about it yet.
21:42
And sometimes it's like, no, no, I don't want the AI agent to
21:45
know about that. And sometimes it's like, Oh
21:47
yeah, I was missing. And it's remarkable how often
21:50
it's the latter because a lot of the companies we work with, one
21:54
of the things we hear a lot on our discovery calls is like.
21:56
Oh, I don't know if I'm ready because my knowledge base isn't
22:00
in a good place. And so our response to that and
22:04
one of the reasons that we built this product is well, actually
22:08
the best way to get your knowledge base in a good place
22:10
is to launch an AI agent and just make sure that it's scoped
22:14
appropriately. But you'll actually find the
22:16
edges of what it knows and it. Our system, the Sierra platform,
22:20
will create basically a prioritized list of the
22:23
knowledge that your agent should have but doesn't have, or where
22:26
it's a little bit off or implying the wrong thing.
22:29
It reminds me of like the continuous integration where
22:32
you're just, you're seeing where's the edge case?
22:34
OK, cool. Now let's integrate that back
22:36
into the knowledge base so that hopefully it's not an edge case
22:39
next time. Yes.
22:40
And I think there's two other things here.
22:42
I know I'm doing a lot of twos. There's two other things here
22:47
that are very, very important. One is it's a really important
22:51
business problem for customers. So when you create the system
22:55
that allows you to kind of pull in the right direction and just
22:58
create that upward spiral of improvement, they're going to
23:02
pull on it. So we have customers where they
23:04
have, you know, 10 plus full time people evaluating
23:07
conversations and looking for opportunities for improvement
23:10
because it's such an important business problem.
23:13
And then the other piece, I know it's not a business podcast, but
23:16
I think it's kind of geeky too, is we have this outcome based
23:19
pricing model where Sierra only gets paid when the agent just
23:23
does the job well. And so you have the incentives
23:27
on the customer side and you have the incentives on the
23:29
Sierra side. And so as sort of a systems
23:32
thinker, it's this very, very fun environment where if you
23:36
create the system. The business incentives on both
23:40
sides will just pull the agent in an upward spiral, and I think
23:43
that's accountable for a surprising amount of our
23:46
success. It's just that the incentives
23:48
are so aligned and the table is set to improve the agent.
23:53
I always love the idea of outcome based pricing but feel
23:58
like it is insanely dangerous. Why do you say that?
24:03
Because you, it's on you. And if your agent fucks up and
24:09
then you don't get the outcome, you pay money like you eat that
24:13
price of having. To deal with it.
24:15
Yeah, exactly. Like there's a potential world
24:20
where you don't get the outcome, but you end up spending money.
24:22
Obviously. It's like we're talking sense
24:24
here, potential, but at gigantic scale and you don't recognize
24:29
there is a problem. Soon enough, you can burn
24:32
through some cash. I think though, these are two
24:36
correlated events in a way, which is it kind of turns a
24:40
lagging indicator into a leading indicator.
24:42
We'd actually rather know that sooner rather than later that
24:45
our agent isn't performing well. We'd rather feel the pain of
24:49
lighting money on fire so that we have to fix it.
24:51
And then when it comes time to renew a contract a year from
24:54
now, we actually might feel better about our relationship
24:57
with our customers. So I think our NPS with our
25:00
customers is much higher because of this, because we actually
25:03
find out about issues and we have to fix them because we're
25:06
feeling the pain of like a hole in our pocket in the moment
25:10
instead of, you know, 12 months from now where they're like,
25:13
hey, yeah, we, you know, it wasn't performing as well as we
25:15
thought. Yeah, that's a great point.
25:17
You're not going to get that conversation as often because
25:21
you're going to be charging them when they have successful cases.
25:25
I could see that. One thing that I've heard folks
25:29
talk about a lot is that getting the, and especially in the
25:36
customer success lane, getting it working for one or two people
25:44
when they're coming in and complaining or doing something
25:46
is not, I'm not going to say easy because it's, it's not
25:50
easy, but it is doable. And then where it really gets
25:54
hard is when you have to service a company that has millions of
25:59
users and then the customer support is trying to service all
26:03
of these tickets at scale. Have you, how do you think about
26:07
like that? And is that right?
26:09
Because I've heard it from folks.
26:11
I I want to like Fact Check it real fast and then see what how
26:15
you think about it. And you're talking about the AI
26:17
agents or the human agents because we, we definitely see
26:20
that on the human side where, you know, it's not a coincidence
26:23
that so many of our customers are large consumer companies
26:27
where they have millions of customers.
26:29
I think, I think it's true that most of our customers have
26:32
millions of their own customers. Most of our customers have over
26:36
a billion dollars in revenue, so.
26:38
That kind of stands to reason. If they're a consumer customer,
26:40
they probably have millions and they see this scale such that
26:46
the power of bringing in AI just changes the nature of their
26:49
relationship with their customers, where previously
26:52
maybe they had a strong incentive to hide their phone
26:55
number on some backwater page of their.
26:57
Website. Now they can plaster it on the
26:59
front. That's so funny Offer.
27:01
We've all seen that company, yeah.
27:03
Yeah, or you're like trying to click through.
27:06
Yeah. Yeah, yeah, it's like a meme at
27:08
this point. And so I, I think when you think
27:11
about, you know, the, the AI agent as kind of the analogy to
27:17
the Internet website or the mobile app, it's also about the
27:22
actual addressable market of contacts with a business growing
27:27
because now you can provide a great experience to anyone who
27:30
calls in. I think though, what you said
27:34
about, you know, seeing just a different distribution when you
27:37
get scale the stat, that's stuck with me a lot.
27:40
I spent the first seven years of my career at Google and to this
27:44
day something like 20% of all Google searches have never been
27:47
made before. And that's because different
27:51
things are happening everyday. New language changes, you know,
27:55
the things people are curious about change, etcetera.
27:59
And so you need a system that's actually like resilient because
28:02
the same thing is true for our businesses.
28:04
The reasons people are chatting in are going to change each day.
28:08
You know, if someone is doing an interview in the SiriusXM
28:13
studio, you're going to get different questions from a day
28:16
where they had an outage or something like that.
28:19
And so each day there's just different questions and you have
28:22
to build a system that's truly resilient.
28:25
And one of the things that I look back on Google really
28:29
fondly about is just the amount of invention occurring in the
28:33
company. You know, you'd go to the all
28:35
Hands, which is called TGIF, and you would hear, oh, you know, we
28:40
use machine learning to improve the way that cooling works in
28:43
the data center. And it's saving us a billion
28:45
dollars. It's a lot of money for an
28:47
algorithm or like you know, they before the latest iterations of
28:52
large language models, you know we are understanding search
28:55
queries better. We created this Bert classifier
28:59
or analyzer which can tell us, you know a lot more about the
29:02
actual semantic meaning of a search query, not just the
29:05
keywords that are in it. Or, you know, our data centers,
29:09
they run 20° hotter than any other data centers because
29:12
actually machines failing isn't a big deal for us because we
29:15
have fail over different from everyone else.
29:18
And so there's the sense of invention and, you know,
29:20
building things and creating differentiation at the technical
29:23
level, not just at the product and marketing level.
29:26
And Sierra's the first experience I've had since then
29:29
where I feel like because AI is moving so fast and because we
29:33
just have some of the most genius thinkers in the world and
29:37
and builders in the world, you have that feeling of we're
29:40
inventing new things. And I think the, what we call
29:43
our agent SDK, which is the software layer that our agents
29:47
are built on, both the developer ergonomics and tooling and
29:51
release cycle and kind of the what we call the agent
29:53
development life cycle put into a product.
29:56
But also the software constructs and the agent architecture
30:00
itself are just novel and inventive and flexible.
30:04
And I think it's, it's what I said I'm, I'm, I definitely feel
30:07
a lot of just admiration for the team, but it's also being out on
30:12
the experience curve ahead of some other companies.
30:15
I think just building with customers from the beginning,
30:20
starting early in 2023, less than six months after ChatGPT
30:23
came out, and just getting reps with customers means that the
30:28
abstractions that we have are a little bit less leaky, a little
30:32
bit more robust for all the different things that that could
30:35
come in. At least that's my theory.
30:38
And then I think we have the right, sorry to ramble a little
30:41
bit here, but I, I think we have the right layers of the stack
30:44
where you have the platform, but then you also have the
30:47
flexibility of agent development.
30:50
So most of our agents at this point are being built in no
30:53
code. Many of them are being built by
30:55
our customers directly, but you still have that ability to
31:00
express anything you need by dropping down into code and kind
31:04
of just interoperating between the two.
31:07
And I think that's where you want to be working with the
31:09
customers we work with, where their business leaders might
31:12
have the best understanding of the goal we're trying to
31:14
achieve, the metrics we care about, the standard operating
31:17
procedures. But then you still need to
31:19
connect to AP is you need your engineering team to get
31:21
involved. Sometimes you need to drop down
31:23
and do something with, you know, not your chef's knife, but your
31:26
paring knife. And so this platform was just
31:30
thoughtfully architected by being so customer obsessed over
31:33
the last 2 1/2 years that you know what we have as a reward,
31:38
as a. System that can handle scale in
31:39
that way. Thinking about 3 years ago when
31:43
you started versus now and how much you've had to reinvent or
31:48
try and create new or try and like test out and it's perpetual
31:54
R&D in a way. I think that's exactly right.
31:57
And if there's one thing that I feel like has helped me, you
32:02
know, have more good ideas than I otherwise would have, it's
32:05
spending those first nine months at the company, you know,
32:08
working as an agent, product manager and agent engineer and
32:11
just being so close to customers, feeling their pain,
32:15
you know, having to tell them directly face to face when the
32:18
agent did something they weren't expecting.
32:20
Getting to celebrate and go out to dinner when it, you know, hit
32:23
some business target. Those those experiences are kind
32:27
of. You know, in my neurons in a way
32:29
that I feel like it's just, I'm just so grateful for.
32:33
And I think anyone building a startup, the the more you can
32:36
kind of feel what your customers are feeling, the the more
32:39
integrated your decisions will be.
32:41
Huge win. I remember a conversation I was
32:44
having with a buddy of mine who was saying, look man, we have
32:49
found the biggest lift on our AI systems from just gathering the
32:54
whole floor of the company together, no matter what their
32:58
job role is, and having a labeling party and getting a
33:02
bunch of pizzas. And we just get together and we
33:05
look at the conversations that we've been having with the AI
33:09
agent, the agent and the user, like real conversations.
33:13
And everyone just labels across like 5 metrics.
33:16
Did it do this? Did it do this?
33:18
What it, what would you give it as a score?
33:20
And by the end of it, the business side of the house
33:24
understands the AI so much better.
33:27
And they can say like, oh, yeah, OK, I see what's going on here.
33:31
And then you catch all these anomalies that you probably, if
33:34
it was only engineers looking at it, they would have never caught
33:38
because it would seem like, yeah, that seems fine.
33:40
But then you have the subject matter expert looking at it and
33:43
they're going, why? What?
33:45
No, we can't say this. We shouldn't be saying this,
33:47
this doesn't make sense or it, or sometimes it's just like,
33:50
this is stupid. No.
33:52
So I I think about that and enabling the low code no code is
33:58
almost like another way of doing that same thing.
34:02
Yeah, it's exactly right. And I think what we've seen
34:04
working with hundreds of customers now is when we bring
34:07
together the business side and the technical side.
34:10
I wouldn't say it's the only way you can get results, but it's
34:13
certainly a heck of a lot easier.
34:15
And it it's the only way you can get really like the best results
34:18
because these two stakeholders are so involved.
34:21
And we've done that enough Times Now where we have some of those
34:23
repeatable processes to bring everyone in the room and make
34:27
sure that all of the different viewpoints are represented.
34:31
The other thing, this is kind of like a pinch me moment from
34:34
working at Sierra, but I don't know if you're familiar with the
34:36
book Unreasonable Hospitality. Yeah.
34:39
Why have I heard of that though? I think.
34:43
But why? Yeah, why?
34:44
Why do I know that? It's not a, it's not a Greek guy
34:47
that wrote it, is it? His name is Will Gadera.
34:50
I'm not sure. His actual origins, he's from
34:53
New York. He ran a restaurant called 11
34:56
Madison Park for a while, which rose they got, you know, they
35:00
became the number one restaurant in the world is the spoiler
35:03
alert. Like the five Michelin star
35:05
where it's like you could only get 3, how do you have 5?
35:10
Right, right. And it's, you know, this
35:12
beautiful ballroom environment in New York, but was kind of a,
35:17
you know, pretty good middle of the road Bistro for a while.
35:20
And he just, you know, along with the team there, took it to
35:23
the next level in terms of service and hospitality.
35:26
And he's one of our advisors at Sierra and helps a lot with our
35:30
how we think about tone and language.
35:32
And so it, it popped into my head because what you're saying
35:34
is like, get the whole floor together and have an eval party
35:37
or labeling party. We've also actually gotten to do
35:41
that with him where he'll come in to the office and spend an
35:44
hour just looking at conversations with us.
35:46
And we say, you know, what would a three Michelin star restaurant
35:49
do in this scenario? And of course, you know,
35:52
sometimes they would do things that are not within the budget
35:55
of our customers, but sometimes they would do things that are
35:57
totally free. And so it allows us to think
36:00
about aspirationally, how can we make an agent where it feels
36:03
like it understands you, the way the person, the host who greets
36:08
you when you go to 11 Madison Park or one of the best
36:10
restaurants in the world and that kind of feeling of service.
36:14
And I think that's one of the reasons why it's fun to work on
36:16
is yes, there's the part of, you know, companies are hiding their
36:20
phone number and we want to just bring it up to a baseline.
36:23
But then there's also that part of how can we actually make the
36:26
relationship that you have with your businesses that you know
36:29
good and great and, you know, something that feels like you're
36:32
going to a Michelin star restaurant.
36:35
Do you remember any of the things that he said, like, oh,
36:38
we could try this, or maybe they would be that they were the free
36:42
ones, not the. Right expensive ones not going
36:45
out. Let's.
36:45
Fly them buying you dinner. Yeah, I do.
36:49
I do. I remember it very vividly
36:50
because one of the things that stood out kind of to your point
36:53
actually, is that even though he at that point hadn't had as much
36:57
experience with our technology, he had.
37:00
Such an intuition of what a good answer would be.
37:03
And so I, you know, had this picture of, you know, I
37:06
understand each part of the stack and that helps me
37:08
throughout the stack. But then I put him in this other
37:11
category of just like savant at the one thing that matters.
37:15
And so we're reading through a number of voice conversations
37:18
and listening to them and hearing for different tone and
37:21
awkwardness and the ability to kind of just.
37:27
Connect and and drop the barrier.
37:31
And what actually stood out to me is there was one message in a
37:34
long, I think it was like a 45 minute conversation.
37:37
And there was one message where the agent sounded really
37:41
empathetic and like clearly connected with the user.
37:44
And we're going through each and being hyper critical.
37:47
And I think on that one, you know, it was less of the, I'm so
37:52
sorry that this is frustrating and more like men.
37:56
This sounds really tough, you know, like we've been working
37:59
through this and we can't figure it out yet.
38:02
And it was a very difficult technical troubleshooting for
38:05
like a basically banking or like financial services application.
38:11
So not an easy problem to solve. But what really stood out was
38:15
that ability to just connect and be like, I hear you, I see where
38:18
I see you, like let's try and figure this out.
38:22
And the, you know, I would say, I would give us, you know, maybe
38:25
a constructively like AB minus on the particular conversation
38:29
that that we were reviewing. But in that particular message,
38:31
it was like, that's what we want.
38:33
And I think someone like Will and, and some of the people that
38:37
have really good taste, which is a one of the few like
38:40
appreciating assets, you know, in the labor market right now,
38:44
maybe the that really good taste often comes out when you're like
38:50
not necessarily criticizing all the bad things with being like
38:53
that. That's.
38:53
What we want? And so I was really impressed
38:56
with his ability to kind of put his finger on those moments,
39:00
those bright spots that we want to scale.
39:02
So he tells you that and then what do you do?
39:04
How do you incorporate a lot of that in in a way that it's not
39:07
just like you're absolutely right.
39:10
Right. So yes, this is where the full
39:12
stack understanding becomes relevant.
39:13
Again, that's. Classic.
39:16
There's a number of different ways you could do this.
39:20
We've had success or I should say one of the ways we've had
39:25
success actually in this particular case is with fine
39:29
tuning of models to phrase things in a certain way.
39:35
The a lot of the base foundation models have been trained
39:38
primarily for chat achieving AGI doesn't really care if it's
39:43
written or spoken. And so I think where when you
39:46
look at where the Frontier Labs are going, they're actually
39:48
going sometimes in the opposite direction of a low latency
39:52
conversational voice experience. And so having a little bit of
39:56
control at the model level for style can help a lot on that
40:00
particular problem. Now when we do have fine tuning,
40:04
if it's with data, you know, that's in any way derived from
40:08
the particular customer, then it's a customer specific model.
40:12
And if it's more just, you know, general tone and style, it can
40:15
be useful across customers. But you also have to balance the
40:18
fact that anything that is being put in the weights could be
40:22
regurgitated. And so you have to make sure
40:24
that there's nothing in those weights that.
40:27
Any of the users who would interact with those weights
40:30
wouldn't be able to see. So the that process is, as you
40:34
can imagine, much more labor intensive.
40:36
And so it's really those specific circumstances like the
40:40
style of voice where it's very specific, very high leverage,
40:43
very important where it makes sense to kind of drop down to
40:46
the weights. Most of the time I would say the
40:48
solutions come at a higher. Level in the agent architecture
40:51
or in some of the underlying task prompts, or, you know,
40:55
changing the model itself, those kinds of things.
40:57
Yeah, it's only when Will comes in for the day when he's like
41:00
that piece right there, You're like, shit, I guess we're going
41:03
to have to fine tune and you. Want to make sure it's durable
41:06
too, because you know if if like style is something that most
41:10
brands don't want to change too often, too markedly.
41:14
Whereas if you're fine tuning this new promotional offer into
41:18
your weights, then two weeks later when the promotion ends,
41:21
your model is out of date. So the right tool for the job is
41:24
definitely one of those things we think a lot about.
41:27
This is probably a good segue into the ensemble of models and
41:31
how you work with that. So it's actually mind blowing to
41:35
look at one of the conversation traces with the Sierra agent.
41:40
If I, if the CRS agent says hello, how can I help you today?
41:43
And I say something like, oh, you know, my, I'll try not to
41:48
use SiriusXM, you know, my groceries weren't delivered and
41:51
I'm not sure where they were. You know what's going on here?
41:55
Between me finishing talking and the agent responding, there
41:59
might be 10/15/20 different models that get invoked.
42:04
And some of them, you know, might be as as simple as like an
42:08
embedding model to do retrieval, augmented generation.
42:11
Some of them are frontier models, maybe even with some
42:14
reasoning tokens to make sure that we're able to handle a
42:17
complex reasoning space. Some of them are fine-tuned,
42:22
fast but cheap classification models to just, you know,
42:25
understand the task at hand. And it would be very slow to run
42:28
these all in series. So sometimes there are certain
42:33
models that need to return a token before the next model can
42:36
start. But most of the time this is
42:38
happening in parallel. And so I think what's really
42:41
impressive is the ability of the agent architecture to break
42:46
problems down into tasks that can be handled by different
42:49
models and then selecting the right models for each of those
42:51
tasks and doing it super quickly.
42:54
And we call that whole process kind of our our constellation of
42:57
models that we use to respond to a message in a way so feels
43:02
really simple. But I think you know my Google
43:04
background showing underneath the iceberg.
43:07
There's just a lot going on between when you type something
43:10
in or when you say something and when you get a response.
43:13
OK, so this is wild, man. Tell me more about the way that
43:17
you are. So you're doing everything
43:19
async. It's basically you're gathering,
43:22
it's that same input data, but then it just spurs out and
43:27
there's a bunch of different variants of it.
43:29
And you do have dependencies on certain parts of the this
43:35
output, like does it already know the taken path?
43:38
Because I can't imagine that it would know all right.
43:41
Well, I how do I, how does it know that it needs a dependency
43:45
and it needs like, I don't understand this, break it down
43:47
for me more? So this is one of the reasons
43:49
why I think voice is still an area where we're doing extremely
43:54
interesting things from an engineering perspective and
43:56
where we're still, you know, significantly ahead on a product
43:58
perspective. And it's because latency is so
44:01
important. So to give you an example,
44:04
retrieval augmented generation, like if we need to access a
44:07
knowledge base to answer a question, the agent will often
44:10
make a decision to look up the knowledge before it even knows
44:14
or a simultaneously while it's deciding whether it would be a
44:17
good idea to use the knowledge. And so often times if you're
44:21
just building something naively, you would think, oh, should I go
44:23
look that up? OK, I'll go look that up now
44:26
I'll give you the answer. But what an agent is doing,
44:29
because it's tokens are, you know, more parallelizable and,
44:34
you know, less expensive than ours.
44:36
It's saying just in case I need to look something up, I'm going
44:38
to go do that. Meanwhile, I'm going to use a
44:40
smarter model to reason about whether it's an even a good idea
44:43
to have gone and looked that up. And then if I decide yes I
44:47
should have looked it up, then I already have the information.
44:50
But you're using that as like a tool.
44:52
It's a tool call that gets invoked.
44:55
Yes, conceptually, yes. So the the look up tool would be
45:00
running in parallel with a classifier to decide if the look
45:03
up tool was even necessary. And.
45:04
Then you already have the information and that's just very
45:07
small microcosm, but there's tons of speculative execution
45:10
throughout the each agent turn. And then you kind of like, you
45:15
know, and this is one of the reasons why like that was not a
45:18
term I was familiar with when I started working in Sierra.
45:21
But we have a number of people that have, you know, worked on
45:24
building compilers and, you know, very complex systems from
45:29
scratch and kind of understand these concepts of software
45:32
engineering. You know, our Brad, our CEO was
45:35
one of the principal authors of Tornado, which is one of, I
45:38
think the first, maybe still the most popular, like async driven
45:42
Python web server. And so we have this, you know,
45:46
20-30 years of software engineering history and
45:49
conceptual understanding that was very new to me when I
45:52
started, but it's like a kid in a candy shop for someone who's
45:55
just kind of a geek. Yeah, that speculate.
45:58
I haven't heard that, but it makes a lot of sense.
46:00
Like it's better to have it and not need it and to need it and
46:05
not have it. Right, right.
46:06
That's exactly right. And our agents are always
46:09
thinking that exact same thought.
46:11
And the so then this is only for voice or this is for everything?
46:15
Well, I don't think we would have been forced to build it
46:18
this thoughtfully if it weren't for voice.
46:21
But now everything is powered by this.
46:24
And you know, sometimes, like e-mail, you probably don't need
46:28
this, right? And you might just want to save
46:30
the money on your little rag, look up and just wait.
46:34
Because a 5 second e-mail response versus a three second
46:37
e-mail response is not a very big deal.
46:40
But in voice, 5 seconds and you know, 1.5 seconds are very
46:43
different and so you want to make sure that you're doing
46:46
everything in parallel. Yeah, because real time is it's
46:49
always been a hard problem in ML in general.
46:52
I know that when it's whether it's like real time fraud
46:54
detection or real time recommender systems or now real
46:58
time voice, it adds this level of complexity to an already
47:01
complex thing. We were talking about how
47:03
complex the voices and that like rich medium and but how we all
47:07
have our own ways of talking. And now to add on to that, this
47:12
whole piece of we need to make sure that we're getting the
47:16
right information at the right time.
47:18
Am I thinking about this correctly with decision tree
47:21
type of thing like that? Style.
47:24
I think conceptually, if a decision tree would be useful to
47:27
a human, then it might be a useful concept to one of our
47:31
agents. Similarly, if a standard
47:34
operating procedure would be useful to a human, it might be a
47:36
useful concept. But the ways that our agents are
47:40
architected is probably more similar to how you and I would
47:44
learn how to complete a task or how to help someone with a task,
47:48
then sort of a decision tree in the sense of traditional
47:53
software and some of the legacy chatbot systems.
47:56
So it's useful concept in that it models a flow that you might
48:01
want to follow. But a lot of times I think our
48:04
agents are more correctly described as kind of goals and
48:07
guardrails where you say, you know, the goal right now is to
48:12
help the agent reset their radio or help them make a payment.
48:18
And you mentioned fraud detection and recommender
48:21
systems as other things were real time matters because
48:24
there's often money involved and like click through rates matter
48:26
a lot and conversion matters a lot.
48:28
And that's true for a lot of voice conversations as well.
48:32
So for example, if you're, you know, thinking about whether you
48:36
want to maintain your subscription to a streaming
48:38
service or whether you'd like to cancel and the agent can make
48:42
you an offer for a 50% reduction for three months, but they take
48:47
too long thinking it over, it starts to get frustrating.
48:49
And you're like, you know what, I just want to cancel.
48:51
So these things matter a lot. And so the goal might be, hey,
48:54
you want to, you know, save the customer from canceling, but
48:59
some of the guardrails there might be around the types of
49:01
offers that you're allowed to provide and the experience that
49:05
the customer should have and, you know, making sure you're
49:08
telling the truth and only the truth and all that stuff.
49:11
And so, you know, decision tree, flow diagram, etcetera.
49:15
These are all useful concepts for just thinking about
49:17
operations. The one that I think is the most
49:21
aligned with our architecture is probably this Goals and
49:24
Guardrails architecture. And it does make sense now, the
49:28
constellation metaphor. In a way, it's like a web of
49:33
different models in different ways, as opposed to some
49:35
straight path that you take a decision tree on.
49:38
This is going to go, you know, to the side.
49:41
I won't say it'll go over people's head with the side, but
49:43
I think like there's kind of the model of the atom where you have
49:47
energy levels for each electron and then there's the electron
49:51
cloud and it is useful to understand energy levels and
49:55
think about when light hits atoms.
49:57
How? Electrons will change energy
50:00
levels, but it is more accurate to think about the electron
50:03
cloud and so that what you sort of modeled with the web is
50:06
probably more accurate. But decision trees and flow
50:09
diagrams are still potentially useful constructs to think about
50:13
how an agent should behave. And then we go back to the heat
50:16
maps and that that makes a ton more sense of why those are so
50:23
important. You have this web that's so
50:25
intertwined and you want to know where things are going wrong and
50:29
you're trying to debug That could quickly turn into a
50:32
nightmare. That's right.
50:33
And the, you know, ultimate expression of that is just the
50:35
LLM itself, where, you know, there's some interesting
50:40
research around model introspection, but in general,
50:42
most developers don't know how LLM's made the decisions they
50:46
made. And So what we've had to do is
50:50
build a system where you know exactly why a decision was made
50:53
and you can fix it without, you know, you can plug this hole
50:57
without juice squirting out the other side.
50:59
And so that's, I think, you know, the, the sort of special
51:02
sauce, not to extend the juice metaphor too far, is kind of
51:06
being able to have both at the same time.
51:09
So with no code and code, how do those two mix and match?
51:14
So when Sierra started, you're primarily a developer platform
51:18
and most of the agents were built by employees of Sierra in
51:22
collaboration with our customers.
51:25
And as we've scaled, we found that a lot of these concepts,
51:28
because we understand them from doing them hundreds of times,
51:32
can be abstracted and actually implemented through no code
51:38
tools as well. And so, you know, within the
51:40
last six months, I'll call it, most of our agents are now being
51:44
built in no code. But it is, you know, the fancy
51:48
word I've learned in the last few months is isomorphic with
51:52
the code. So you can go.
51:54
It's never a one way door to choose to use code or choose to
51:57
use no code. And what this means is someone
52:01
from a customer service organization at one of our
52:03
customer companies can go and build a journey themselves.
52:08
And then they can say, oh, it would be nice for this
52:10
particular tool to have it implemented in code because it
52:13
actually needs to use an API in the specific way with streaming.
52:17
And I just don't want to go through the process of, you
52:19
know, wiring that up in no code. It will be much more idiomatic
52:22
to just have my engineer write that thing and it will
52:25
integrate. These folks are when you say
52:27
tool, like we're talking a tool call for the agent, or we're
52:31
talking like, oh, this tool, this program that I'm used to
52:34
using as. A tool call for the agent.
52:36
OK. There might also be areas where
52:39
you, you know, most of this you at this point you can set up
52:42
with no code, but a situation where you might have an
52:46
integration and you already have built that integration out in
52:50
code, but you want to store the metadata, you know, through like
52:54
environment variables basically. And so what's happened over the
52:58
last six months is this has gotten gone from being mostly
53:02
something you do with our team to mostly something that our
53:05
customers do on their own with the support of our team.
53:08
And so we still have that kind of high touch consultative,
53:11
highly accountable model because we think it's just effective.
53:16
But we also offer the control and customizability that you
53:21
mentioned at the beginning where it's just like here's the
53:23
product, go for it. And so we've landed on kind of
53:26
the best of both worlds where you can build an agent yourself,
53:29
you can use our no code platform, you can also use our
53:31
code platform as well. And you can use the two
53:33
together. And it's kind of like the
53:35
productized instantiation of what you said, which is when the
53:39
customer experience org or the business org and the technology
53:44
org are working so well together, we just tried to like
53:48
make that a product instead, you know, having seen it work so
53:51
well as a process. And so that's another thing
53:55
where I think the technical depth and understanding of the
53:59
code was necessary to make the no code project or product so
54:04
robust and extensible. Yeah, because if you try to
54:07
start talking to the business side of the house about like, Oh
54:12
yeah, our evals are failing here, or like there's a certain
54:16
level that they just don't care. And so I imagine you're
54:21
abstracting a lot of things away from that type of user with the
54:27
no code part. And you're allowing the software
54:31
developer, if they want to figure out like, oh, let's go
54:34
write a tool or let's figure out how to make this work in
54:38
whatever way with this API, then they can do that.
54:41
But for the most part, you're like, I just care about like,
54:46
can I have it do this? Here's the, what did you call
54:49
it? Goals and guardrails.
54:51
Here's the goal. Here's the guardrails.
54:52
That's all I want to give you. Yeah, it's exactly right.
54:55
And we work with companies that have, you know, very robust
54:58
change management processes and they have their own concept of
55:02
releases or continuous integration and deployment.
55:05
And so being able to fit into that on the code side, while
55:08
also fitting into the quality assurance and review oftentimes,
55:12
which involves, you know, large teams doing review, filing
55:16
issues, making improvements has been a huge accelerator for the
55:20
business. Because oftentimes, you know,
55:23
when we were mostly a developer platform, we had companies we
55:27
would sell to where they're like, hey, we really want our
55:29
operations team to own this and we love your product, but just
55:33
can't get on board with that. Over the last six months, now
55:36
that we have, you know, equally good product for operations and
55:39
CX teams that we have for developers, we've seen it be
55:43
much more of a, you know, 2 thumbs up from all parts of the
55:47
organization. Yeah, the amount of reviews that
55:50
you need to have and the amount of like meetings that you have
55:54
to align on. Somebody told me the other day
55:56
they were like, man, launch a model with one click, like one
56:01
click deploy and like you get laughed out of the room in the
56:05
enterprise if you have that. Because the one click part is
56:09
not the hard part. Like it's all those meetings and
56:11
all that alignment and all of their reviews with the regular
56:14
or the lawyers and the everybody that you have to talk with
56:18
beforehand before you get to the click.
56:21
That's the hard part. Right.
56:23
And the business side as well usually has the outcome in mind
56:28
that we're trying to drive here. And so having them on board,
56:32
integrating with their change management processes is actually
56:35
a huge accelerator for the deployment in a lot of cases
56:39
where you can say, oh, things are moving so slowly, it's
56:41
taking us months and months. But when you're actually on
56:44
board with them, you get that same force, but in the other
56:47
direction. And so we've had customers like,
56:51
you know, one of the 10 largest businesses in the United States.
56:54
We went from the, you know, basically first meeting with
56:58
them to live in about 3 months. Wow.
57:02
Wow, I think that's a good place to end it.
57:07
Demetrius, thanks so much for having me.
57:09
Yeah, this has been awesome. I really appreciate.
57:31
It one thing I didn't have on the list is.
57:36
Our, we just launched, we just announced year two of our APX
57:40
program, which is our program for new grads kind of combines
57:44
engineering, product management and entrepreneurship.
57:48
That's pretty fun. So I helped kind of conceive of
57:51
that program and get off the ground.
57:52
And we have 5 people in the first year, they're amazing.
57:57
So we're doubling down on it for the second.
57:59
Year and they stayed at the company so.
58:02
Yeah, they're, they're like 3 months in and it's a one year
58:05
program. We had it as 18 months and then
58:08
things were moving so fast that we were just like, OK, this can
58:12
only be 12 months. No way dude.
58:15
Wait, so this is somebody that got kind of gets to like
58:19
parachute into meetings and is no.
58:23
No, they're like a full time. The way it works is 6 months of
58:26
product management, six months of engineering.
58:29
Half the Apxs start on 1/2 start on the other and then they kind
58:32
of flip and you're working directly with customers,
58:35
building agents. And then you have kind of some
58:38
very large customers that you're working with where you're
58:40
filling that role, as well as some smaller customers where
58:44
you're kind of the end to end entrepreneur doing all the
58:47
coding, customer management, product management.
58:50
And it's a really good group. I think because it's so
58:54
multidisciplinary, it attracted a lot of people that want to be
58:56
founders and that are just awesome.
58:58
Yeah. And you get the chops, you get
59:00
to learn while you're doing it. Yeah.
59:02
How'd you think of that? So our two Co founders Brett and
59:07
Clay were in the 1st and 3rd class of Google's APM program
59:11
back in like 2002 or 2003. Yeah, didn't Brett create
59:16
something with Gmail or some some?
59:19
I think his Co founder created Gmail at a previous startup and
59:23
he Co created Google Maps. That's it.
59:25
Yeah, Maps. He was the one with maps, right?
59:28
There's the he's told not enough podcasts.
59:32
I probably don't need to because everyone asks him, but basically
59:35
rewrote the whole front end, which was the I guess
59:38
complicated part in a weekend after lots of different
59:42
explorations. And I think is proud of the, you
59:45
know, reduction in the binary size that they got out of that.
59:49
Nice. So then you saw that they did it
59:51
and you were like, why don't we do that?
59:53
So it's so Brett and Clay were Class 1 and Class 3 and just
59:59
remember it really fondly. And then I actually started my
1:00:02
career at Google too, about 10 years later, also in the same
1:00:05
program. And we have 10 or 11 people in
1:00:08
Sierra who started as Apms at Google.
1:00:12
So we had a lot of appreciation for that program, a lot of
1:00:14
excitement, but then we also wanted to kind of reimagine it
1:00:18
in the era of AI where you can be a bit more of a generalist.
1:00:23
You want to be super technical, but you also want to put your
1:00:25
product hat on and think. And we thought about what are
1:00:28
some of the most successful people in these roles?
1:00:30
It's future entrepreneurs that are going to go, you know, start
1:00:33
their own company 24510 years from now, in Clay's case, 18
1:00:38
years from now. And so we wanted to create a
1:00:41
program that we thought kind of honored both of those
1:00:44
influences. And that's why we have kind of
1:00:46
APX. It's kind of just the
1:00:48
multidisciplinary intersection of a lot of different things.
1:00:51
A. Lot of times you get
1:00:54
entrepreneurial folks that have so much ownership that they like
1:01:00
have a lot of opinion on how things should be done.
1:01:03
And then I've heard from friends who have not necessarily the
1:01:08
same type of program, but the similar things where they're
1:01:12
like, oh, we want like an entrepreneur type to come, but
1:01:16
then after they have them for a while, they're like, actually,
1:01:18
we don't want you to run the company.
1:01:21
So do you see, like how do you battle with that kind of, oh,
1:01:25
the extreme ownership? And I want to do it my way and
1:01:28
I'm very opinionated. And this is how we need to do it
1:01:31
because that entrepreneurial spirit versus I can't.
1:01:35
But we're also, we have the boss already.
1:01:38
You. You're not the boss.
1:01:40
Yeah. I think there are two things
1:01:41
that stand out a lot. 1 is kind of creating the systems and
1:01:45
conditions where people can have that level of autonomy and
1:01:48
ownership. It works pretty well at Sierra
1:01:50
because as an enterprise company, we're working with a
1:01:54
number of different customers. And so the each agent is kind of
1:01:59
a place where you can have that autonomy and ownership and
1:02:02
customer relationship building and user researching.
1:02:05
And so I feel like there's kind of AI wouldn't call it a sandbox
1:02:09
because it's in production, but there's a place where you can
1:02:11
really, you know, where autonomy is working as intended.
1:02:15
And then I think the other big piece is just having great
1:02:19
leadership in the company where even if you yourself feel like a
1:02:22
leader or someone with strong opinions, you have a level of
1:02:26
respect for the direction of the company that thought that goes
1:02:28
into it. And kind of certainly in my
1:02:31
case, and I think in each of the AP XS case view, our founders
1:02:35
view some of our early employees as role models that you kind of
1:02:37
want to model yourself off of, which takes the edge off of kind
1:02:42
of, you know, being more stubborn and those kinds of
1:02:45
things. Because if you're objective
1:02:47
about it, you know that you also need help, need to listen to
1:02:49
people. Yeah, especially at that point
1:02:52
in your career, I would hope. Yeah, yeah, I, I think.
1:02:57
The humility, especially with respect to customers and also
1:03:01
with respect to some people who have done it a few times before,
1:03:05
that's a big reason I decided to start my career at a big
1:03:07
company. You know, Google's full of some
1:03:10
of the smartest people, and you can find the right people that
1:03:13
you want to learn from. And I think one of the cool
1:03:15
things about Sierra is you have the pace and size of a startup,
1:03:20
but you also have the experience and accomplishments and wisdom
1:03:24
of, you know, if you were to take the people that you'd like
1:03:27
to work with at a company like Google.
1:03:29
And so that's been really fun. Yeah, and higher percent
1:03:33
likelihood of when you do get a mentor, it's a good one because
1:03:37
if there's a lot of people, there's also a probably a bigger
1:03:42
spectrum of what you can hit. Exactly.
1:03:44
Yeah, you have to be selective in a large organization here,
1:03:48
you probably will get lucky. Well, all right, let's let's
1:03:52
talk a little bit about is this. A good height by the way.
1:03:55
Yeah, that's perfect. I think it wait, keep talking.
1:03:58
Yeah. Can you hear me?
1:04:00
Yeah, I think it's perfect. OK.
1:04:01
All right.