Latest / Agentic Conversations (formally mlops.community) / AI Agent Development Tradeoffs You NEED to Know

Transcript

AI Agent Development Tradeoffs You NEED to Know · Agentic Conversations (formally mlops.community)

0:00
I think that agents collapse on a simpler structure as the
0:02
models get better. We actually had our first agent
0:04
frameworks all come out during that period and we got like auto
0:07
GPT and baby AGI and. None of them worked.
0:11
And they were enough to show promise.
0:12
And then we got Land Graph Cloud that's trying to become Datadog.
0:14
And then we've got AWS that's probably doing both.
0:17
I I still am not sure what it means to be an agent in the
0:21
cloud. One argument you could make
0:24
against these agent frameworks is that.
0:32
Here we go. Let's go.
0:33
This is particularly close to my heart because I started in tech
0:38
as an SDR and you are working on making that job obsolete.
0:44
Well, we don't have SDRS at 11 X.
0:46
So that's that's true really. Yeah, we, we actually we only
0:50
have account executives and other, you know, roles within
0:52
the revenue org, but we don't have.
0:54
And their calendars are both SDRS.
0:56
Yeah, Yeah, yeah. So for context, I'm tech Lead
1:00
and engineering manager at 11X. And 11X is a company that's
1:04
building digital workers. And our two workers today are
1:08
Alice and Julian. They are both in the revenue
1:10
org. So Alice is our AISDR or sales
1:14
development representative product.
1:16
And then there's Julian. Julian is our AI voice agent who
1:20
actually specializes in inbound sales and speed to lead.
1:24
If you're familiar with that concept and I have, you know,
1:28
I'll, I'll copy out by saying I've got some voice AI
1:30
experience at 11X, but most of my role is focused on Alice.
1:34
But my previous startup was a voice AI company.
1:37
And so we can talk a little bit about about Opkit, if that.
1:40
Yeah, well, we're going to Opkit.
1:42
OPKA was YC Summer 21 company. We built one of the first
1:45
vertical AI phone calling solutions for the healthcare
1:49
market. So we were automating calls to
1:52
insurance companies on behalf of healthcare providers and
1:55
provider groups mostly for verifying patients insurance
1:59
eligibility or for securing prior authorization or for
2:04
claims follow up. So, you know, a lot of the
2:06
operations that are relevant in the back office and billing
2:11
department of these provider groups are actually can only be
2:13
done over the phone. I was going to ask, they don't
2:15
have a website for that. Yeah, they, they have a website
2:18
short story is that they're, they're not that incentivized to
2:21
make it very easy. And one of the ways that they
2:23
make it hard is by only allowing you to do certain things over
2:27
the phone. And we had been working in
2:30
health insurance medical billing for about a year and a half when
2:35
ChatGPT and the GPT 3.5 came out.
2:38
And we had already had team members who operations folks who
2:41
are doing these calls for our customers as a sort of a
2:46
services component of our platform.
2:48
And it was like immediately we realized we're going to be able
2:51
to automate the call in to end using AI.
2:54
And so we just pivoted to purely doing the the voice AI solution
2:57
for provider groups. But what happened?
2:59
Why aren't you still doing that? We, I think we're a little early
3:01
on the technology technologies, I mean that's.
3:03
Summer 21. Yeah.
3:04
And I'm and I actually interesting, I watch the 11X
3:07
team work through a lot of the same engineering challenges that
3:10
we were working on while we were at at OPCAD, kind of doing it
3:14
from the the sidelines. I mean, they're, they're
3:16
definitely taking things to a new level and they're, they're
3:19
way beyond where we were previously and, and way more
3:22
sophisticated. Those guys are probably as sharp
3:24
as it gets when it comes to building voice agents, but it's
3:28
fine to see like the industry is still grappling with how to do
3:31
that well and actually voice agents are a really different
3:34
than regular text base or or multi modal agents.
3:38
I talk about this all the time. They're so it's a rich medium,
3:42
but they're when we type something out, if you and I both
3:46
type the same thing, it's pretty much understood what we're
3:51
trying to say. If we say the same thing, it can
3:55
be that I'm saying it with a certain tone and I mean one
3:58
thing and you're saying it and you mean something completely
4:02
different. Yeah.
4:03
And so how you pick that up with these voice agents is very
4:06
difficult. Yeah, Yeah.
4:07
Those inflections and those paralingual cues and obviously
4:12
make things a lot harder. And then doing that in real
4:14
time, yeah, with latency constraints is also really
4:17
challenging. Yeah, so but.
4:19
So then let's let's move over to Alice and what you're doing
4:22
there. Yeah, you probably know the SDR
4:24
world better than I do. I'm an engineer so I I was found
4:28
her previously and I've I've tried to do some outbound which
4:30
is why I have a little bit of appreciation for how painful.
4:32
You, you're on the other side of it, right?
4:34
You probably get hit up all the time.
4:36
Now it's that's all vendors coming at me.
4:39
And ironically, I'm, you know, I'm building like the the Canon.
4:41
Right. Imagine they're using Alice to
4:44
hit you up about that. I Yeah, I wonder.
4:47
If I could suss out or recognize and Alice read an e-mail, I've
4:50
fortunately I think they're like dynamic enough that I I wouldn't
4:53
notice. Now you talk about Alice being
4:55
an agent. What does that mean for you?
5:00
Why is it agentic as opposed to just a program?
5:05
Yeah, that's a really loaded question.
5:06
I'm getting to the definition of what an agent is in a hotly
5:10
debated topic. I think that, you know, agents
5:16
the the characteristics that they have are they can automate
5:19
complex and ambiguous tasks, tasks that really were not you
5:23
could not automate previously. So they're sort of unlocks a new
5:26
category of things that can be automated with software they use
5:30
tools they have sometimes they have memory and generally built
5:36
with one of these new agent frameworks, whether it's land
5:38
graph or the new Open AI Agent SDK.
5:43
And effectively there's A at the center of the agent, there is a
5:45
loop that involves a language model.
5:48
And in that loop, the the agent is planning about what to do and
5:53
then taking action, usually through a two call, then
5:56
observing the results of that action on their environment and
6:00
and then reasoning about whether it should continue to execute
6:02
and and take another action or whether it's in the execution.
6:06
And inside of Alice, this looks like I have this data, I'm
6:10
sending an e-mail or I'm scraping a website to find out
6:13
more about this person, and then I'm personalizing a message and
6:19
then I'm sending the e-mail. Yeah, that's essentially right.
6:21
So the this main campaign creation flow, which is the core
6:25
part of what Alice provides is starts with sourcing.
6:28
So helping you build a list of people that look like your ideal
6:31
customer profile. We have an agent that that
6:33
specifically helps with building the right audience.
6:36
Then there's a research on the individual leads that are in
6:41
your audience. So we'll essentially have a deep
6:43
research agent that like creates a really comprehensive report on
6:46
Demetrios and on Sherrod and anyone else who's enrolled in
6:48
your campaign using some of our data and also using, you know,
6:52
web web scraping and web search tools.
6:56
And then following research, there's sequence and message
6:58
generation, the sequences, the, the different messages that
7:03
occur or in the outreach process for this lead.
7:05
So you know, I'm going to contact Demetrios on day one,
7:08
day three, day 7. This is something the agent
7:11
decides. And then finally, the contents
7:14
of each of those messages. You know, what am I going to say
7:18
to Demetrios in each message? What's my follow up going to be?
7:20
You know, what value props am I going to emphasize?
7:23
What personalization am I going to call it?
7:24
I'm going to say like, Hey ML OPS, I saw you guys, you
7:27
recently broke what like 1,000,000 views on one of your
7:30
videos, so. They call that out and like
7:32
Congrats on the. Would say Congrats on like
7:34
breaking that milestone. You know, I bet you have a need
7:37
for, you know, more growth tools now that you have reached this
7:40
digital. Dude, you know what's so funny
7:42
is back in the day when I was doing the STR work, I thought it
7:46
was magic when a friend showed me a tool that gave you the tone
7:52
that you should speak to somebody in when they had this
7:56
list of folks. And so you could add your leads
7:59
and then it would say, Oh yeah, this person.
8:01
We know that judging by how they speak on the Internet, you can
8:07
talk to them in this type of tone.
8:09
Interesting. Yeah.
8:10
And now for me was like, wow, incredible.
8:12
Yeah, this is 10 steps further than that.
8:15
Because now I don't need to even know what kind of tone or what
8:19
to think about, and I can pull in relevant data on recent
8:22
things that have come up and I don't have to craft it.
8:26
And I think one of the product challenges that we have is, you
8:29
know, we're selling this into sales organizations, people like
8:33
you who actually know what they're doing, some of them a
8:35
lot, I mean basically all of them have previously been SDRS,
8:38
have opinions about how these things should work and what
8:42
balance we always need to strike is that we think that Alice can
8:46
do this very well. Alice is great at sourcing,
8:49
great at writing messages, great at research, knows how to pick
8:53
the right tone, knows what aspects of your product and
8:55
business to emphasize when reaching out to this particular
8:59
lead. And sometimes the user thinks
9:02
that they know best and wants to, you know, wants the
9:05
messaging to have a particular structure, wants it to only ever
9:09
reference, like this case study wants to target a specific type
9:14
of lead, even though, you know, maybe, you know, we don't think
9:17
that might be the best lead in terms of performance.
9:19
So we have to strike this balance between providing
9:22
control for the users and providing, you know, results.
9:27
And the way that we think we get results is by effectively
9:29
putting Alice on autopilot and letting her do what she does
9:32
best. And that's something comes up in
9:37
our, you know, product conversations and, and with
9:39
customers a lot. No, we talked about the
9:42
reliability of this and you mentioned, all right, well, we
9:45
chose to outsource the hosting of the agents in a way.
9:50
What were the trade-offs? How did you look at that
9:53
decision of we're going to go with and you're using lane
9:55
graph, right? So you said we're going to use
9:58
lane graph. Why and how did you come to that
10:01
decision? Yeah, that's a great question
10:04
and controversial recently, I forgot.
10:08
Yeah, there's. Why didn't you use pidantic AI?
10:11
That's so true. Pidantic we we're, we're mostly
10:15
TypeScript, TypeScript shops, so probably not the right fit for
10:18
us. But yeah, it's been a hotly
10:21
debated topic, the agent framework wars recently.
10:25
I'm sure your listeners know like the the blog post that
10:27
Harrison published in response to opening eyes new agent
10:31
framework and their their post, their guide to building agents
10:34
through a little bit of shade on the idea of workflows and graph
10:37
structures. Plangraph is the agent framework
10:41
that has graphs in its name. The name, they went all in on
10:44
the graph. Idea exactly and actually into
10:46
their credit. I think it's an extremely
10:48
flexible structure, land graph. Yeah, in in graphs generally for
10:54
representing the agent. And you know, with land graph,
10:58
everyone can build an agent that gets to production.
11:00
And I think that is in part possible because of the graph
11:04
structure that they chose. Now, a lot of other agent
11:07
frameworks are much simpler. They, you know, modeled around
11:12
chat and maybe are, they're sort of pure agents in the sense that
11:16
this is not a workflow. There are no predefined code
11:18
paths between nodes. There's really just one node and
11:21
it's looping through this recent act observe loop, you know, the
11:25
React agent that people are familiar with.
11:27
And that's a super powerful structure.
11:30
I think that as models get better, all agents start to
11:33
collapse on this simpler form factor.
11:37
But the reality is like, you know, models have limitations
11:40
today. Prompts aren't perfect.
11:42
We, our evals are not that good. Like something's very expensive,
11:46
you know, and you want to create structure in the graph that
11:50
ensures that like a certain step performs well and a graph, a
11:56
structure allows you to do that. Well, yeah.
11:57
And if you're going out and you're selling a product to
12:00
folks and you're claiming that it's going to work, you can't
12:03
have it work some of the time, right?
12:05
And so a graph gives you much more reliability.
12:08
And you thinking about it from that systems angle, I'm sure
12:10
that was one of the things that you realized right away was we
12:14
need this to be reliable. We recently rebuilt Alice from
12:19
scratch over like the last six or seven months.
12:23
We started in October of last year, so October 2024.
12:26
Alice for context was launched in January of 2023, so a lot had
12:31
changed. We actually had our first agent
12:34
frameworks all come out during that period and we got like auto
12:37
GPT and baby AGI and. None of them worked.
12:41
Or enough to show promise. You're like, wow, there's
12:43
something cool here. And then the Lang Chain people
12:46
are pretty forward thinking. So they launched Land Graph and
12:50
really what happened for us as we saw the way the landscape was
12:53
changing and then we saw some of the new agentic products that
12:56
were being released, specifically the Replit Agent,
12:59
left a really big impression on our team.
13:02
Are you familiar with the Replit Agents?
13:03
Yeah, but I don't know why it would be different than like
13:06
cursor or windsurf agents or Devon.
13:08
The Replit agent was the first of these tools of these coding
13:12
assistants to really be agentic. You know, cursor had tab, which
13:16
is the one of their main feature, which would use a
13:18
language model to predict where in the in your code you're going
13:23
to jump to next and what the next change that would be.
13:25
And you could sort of just press tab to to apply that they didn't
13:30
have an agent in the in the platform.
13:32
It was auto complete. It was.
13:33
Effectively really really good auto complete that made GitHub
13:36
Copilot look like a joke and to their credit, like all done on
13:41
like custom language models, some small language models
13:43
involved really impressive work that they did.
13:48
Meanwhile, Rapid was the first one that was really in my
13:51
opinion the first company to to bring this AI coding agent that
13:55
is looping through multiple steps calling tools to build an
13:59
entire project. The agent is your copilot and
14:02
it's actually helping you build the project and experience was
14:05
unlike any product experience we'd ever seen before.
14:08
And they did this way before Cursor Composer which is the you
14:11
know the cursor agent which is now the default mode within
14:15
cursor. And so you saw that and you
14:17
thought that's where we need to go.
14:19
Exactly. We, we saw that, we, we knew
14:22
that landscape had changed products, AI products were going
14:26
to look completely different and it had to be agents.
14:30
And like we just basically made the leap based on that
14:32
conviction. And it's interesting that you're
14:34
reassessing. I imagine you have to reassess
14:37
continuously because it moves so fast and the landscape and what
14:43
you're doing is 1 upping all the time.
14:48
Right. Yeah, keeping up with models is
14:50
really hard. Well, not just the models, but
14:52
like you're seeing, OK, now we're not in a baby AGI world,
14:56
we're in a lane graph world. So we could potentially do what
15:00
we're trying to do now a lot easier.
15:03
So obviously models are improving.
15:04
I've mentioned that I think that agents collapse on a simpler
15:06
structure as the models get better.
15:09
One argument you could make against these agent frameworks
15:13
or against more complex ones that allow for more complexity
15:15
like Landgraf, is that they actually they incentivize you to
15:19
create structure and scaffolding to accomplish short term
15:23
performance gains, right? Like I can add a few extra nodes
15:26
here and create a new loop within my agent and that ensures
15:30
that this step in my campaign creation flow now consistently
15:34
works. What happens is that you do this
15:37
for to expand the capabilities for agent in a lot of places.
15:40
Now you have a very complex graph.
15:42
Then a new model drops and you're like this new model.
15:47
Yeah, I could have accomplished all of what I just accomplished
15:50
using a new model in a much simpler structure.
15:52
And I could have just waited. You could have just waited.
15:55
Damn model came. Out.
15:56
And so you're doing this constant analysis of whether
15:59
there's premature optimization. You're like, is this is am I
16:03
making, is this investment in that my graph structure or in
16:06
the framework of my agent going to going to pay dividends before
16:11
a new model comes out that renders it obsolete.
16:13
And then there's clean up work to be done to simplify your
16:15
agent afterwards because actually you what you do is you
16:18
become sort of sclerotic where the agent is is like locked into
16:23
this structure that no longer really makes sense and prohibits
16:27
it from doing other things that you might.
16:29
Investors, do you look at, as you're saying, hey, we need to
16:31
be forward thinking in three months, we don't want to have to
16:34
go and rebuild this from scratch.
16:36
Yeah, that's a really good question.
16:39
And I imagine you can't tell the future, so obviously it's like.
16:41
I can't tell the future, barely keep up with the present.
16:44
We try to obviously stay on top of all the new model releases
16:47
and at minimum we will when we get access to a new model, we'll
16:52
have our team sort of bash it and like regular chat clients,
16:56
get a feel for how it behaves. Maybe use it in cursor a little
16:59
bit to see how it performs in an agent loop.
17:02
We will drop it into our agent and see what the performance
17:06
impact is. To be honest, it's a lot of
17:08
vibes today. We we have an agent that works
17:12
and you know a gentic product that's pretty successful
17:15
customers are happy with. Can you just lock it in there
17:18
then? I wish.
17:19
I wish we could, we could just say, you know, mission complete
17:22
and we're, we're done. But we, I think that there's a
17:26
lot more that we want Alice to do and our, our, our North Star
17:30
metrics are revenue generated for customers or pipeline
17:34
generated for customers meetings booked.
17:36
Some of these are hard to do attribution for by the way,
17:39
positive replies to to any of the outbound messages and
17:44
anything we can do to create left in those metrics justifies
17:48
like the investment on the engineering side.
17:51
So we're just kind of keeping an eye on those metrics and on some
17:54
of the evals that we've set up on, on the agent.
17:56
And when we feel like there's a model that changes the game, we
18:02
spend a little time investigating whether that is.
18:05
There any it's a time where you've been creating in creation
18:09
mode and you thought I could do this right now, but I think the
18:15
next model drop is probably going to make this all not worth
18:20
my time to invest in it. Yeah, that's that is a good
18:24
question. Or we could do it retroactively
18:29
and say what did change in the last model drop?
18:33
Yeah, we're still mostly on 3/5 and 37 sonnet.
18:40
We've obviously got access to some newer models from Open AI
18:43
Sense. I don't think that we've seen
18:48
like a big enough performance left from those models.
18:51
There's also a cost and latency consideration.
18:54
So it's a pretty complicated analysis at any given time to
18:57
understand whether we should upgrade.
19:00
We need to know is the performance.
19:03
First of all, we've got all these lagging performance
19:04
indicators. It's so it's hard to even know
19:06
what the performance left is. And once we know what the
19:08
performance left is, how does that relate to the, to the
19:12
increase in cost and whether and what does it, what does it mean
19:15
for product or end user, excuse me, in product user experience,
19:20
when you're using that with the agent or interacting with that,
19:24
does it feel slow when it's generating those messages?
19:27
And then meanwhile in the background, we are doing like
19:30
hundreds of thousands of emails per month.
19:35
What's our throughput, What's the speed impacts there?
19:39
So I wish we could say we're more systematic about it.
19:43
Maybe the best thing for like the ML OPS community to say here
19:46
is that we don't have it down to a science.
19:48
And it's a lot. It's a lot of, you know, vibe
19:50
decision making there. Yeah, it does.
19:52
It does feel like you're not the only one, Yeah.
19:56
I'm pretty sure that's the. State of the the industry right
20:00
now. Yeah, we're, we're not
20:02
researchers either. You know, we like, we, I'm
20:04
pretty proud of what we built with Alice and how, you know,
20:07
how far we are as a company and with 11X.
20:09
But we're really focused on commercializing AI.
20:13
And so we, we're happy to, to rely on the, on the labs and on
20:18
the other researchers to, to, to and the benchmarks to help guide
20:21
us. But we are very pragmatic and
20:24
focused on creating a better product.
20:27
And so that guides a lot of our decision making.
20:29
What did you decide on when it comes to Lane Graph again?
20:35
You told me that it's hosted. What does that even mean?
20:38
Because I'm not sure I'm clear with a hosted agent Versus what?
20:43
Yeah, yeah. And why?
20:46
That there's a lot of places we could go with that.
20:48
So just the main decision making criteria on Landgraf,
20:52
specifically in September of last year, Landgraf is one of
20:55
the best and most mature options for an agent framework.
20:58
And you know, you can argue whether that's still true, but
21:02
at the time, certainly it was and those guys are leader in the
21:05
AI dev tools and an agent space. So we really wanted to partner
21:07
with like a team that could educate us and that where it was
21:11
leading the way and had a solid adoption wasn't going to
21:14
disappear in six months like many agent frameworks.
21:17
That's their have and and probably well.
21:19
Yeah, you have to watch out for that.
21:20
That's a great axiom to be thinking about.
21:23
Yeah, yeah. We use Lang Chains products in a
21:26
lot of ways. We use their Landgraf agent
21:28
framework. We use Landgraf Cloud, which I
21:30
think is now called Landgraf platform.
21:33
This is their cloud hosting solution for AI agents.
21:37
We use Lang Smith for observability, which is coupled
21:40
with Landgraf Cloud. That gives us visibility into
21:44
the actual agent runs, and then we use some of their other SDKS
21:50
within the agent itself. I I still am not sure what it
21:54
means to be an agent in the cloud.
21:57
Yeah, yeah. But we could break down like
21:58
what the agent actually is. So I we talked about like the
22:01
agent is this loop, right? It's really an LLM and a loop.
22:06
And the LLM is saying is, is deciding what your reasoning
22:09
about its state and its environment.
22:11
So given like some input object which represents the state of
22:14
the world and a prompt which explains who it is and what its
22:19
objectives are, it decides or it reasons about what action it
22:23
should take. Then one of the actions and many
22:27
of the actions typically are tools that it can call.
22:29
So if it chooses to call a tool, then you have something that
22:32
executes the tool. Now the LLM isn't executing the
22:35
tool, it's just saying I would like to call this tool.
22:38
And then you've got maybe it's some case statement that says,
22:42
you know, if LLM return tool call get tool called type and
22:49
based on whatever type was called, you would then invoke a
22:52
function which is implemented in the in the code.
22:54
Or maybe a more modern way to do this was with MCP.
22:58
Now you have the, they have access to an have.
23:00
You had to go and upgrade shit because.
23:02
Of that we're I personally love MCP, very interested in it, have
23:05
built some side projects. We're not using MCP in
23:07
production today. No.
23:08
Smart, yeah. Understandable.
23:11
It's a very new technology. This would probably not be like
23:14
the most pragmatic thing to do, but super interesting and things
23:18
does solve a lot of problems related to tools.
23:20
We can get into that. So then it calls the tool.
23:23
So it calls the tool. Something executes this tool
23:26
That could be the same, let's call it a node process, because
23:30
we run TypeScript. So our agent is written
23:32
implemented in a node and this node process is running.
23:36
It calls the tool as a function. Function returns some results
23:40
then then you update your state to reflect this the new state of
23:45
the world now that the functions been called.
23:47
Then you repeat the Lib and eventually the LLM says yeah
23:50
I'm. I'm either stuck in need to and
23:53
should exit the process or I've completed the task and I should
23:56
exit. And so that whole process is
23:59
done on the link chain cloud. Yes.
24:01
So we use. So Lang chain provides us this
24:05
agent framework lane Graph, which was some create some nice
24:08
abstractions for defining the agent that I just described.
24:12
And what that ends up being is a a node or a JavaScript bundle
24:18
that runs with node in a container on Lang chain servers
24:23
or Lang graph servers. And there's a a.
24:27
There are two things that sit in front of us that Lang chain also
24:30
provides or Lang graph also provides.
24:32
The first is a is an API that allows us to call our agent from
24:37
either our back end or our front end.
24:39
We do. You're just interacting with an
24:40
API. Exactly.
24:41
So our agent becomes an API endpoint for us.
24:44
And as a side note, Lang chain has also created this protocol
24:49
called the Agent protocol, which is like a set of guidelines for
24:54
the for how that API should be structured.
24:58
What are the different endpoints that it should have?
25:00
What are the different entities and resources that it should
25:02
expose? Which Http://methods should be
25:05
used? What are the payloads?
25:07
That was pre MCP. It's not, it's not for tool
25:09
calling. It's really for like exposing
25:11
your agent. Yeah, You know, if I want to
25:13
deploy my agent as an API and have, I mean, Alice is a
25:16
constellation of agents. But if we wanted to take Alice
25:19
and turn her into an API endpoint that you could use
25:22
instead of using our dashboard, then one of the ways we could do
25:25
it is by implementing an API that adheres to this agent
25:29
protocol that Lang Chen has to has defined.
25:31
So they have this API and the API sits in front of a queue
25:36
basically every time I call this API endpoint.
25:38
I think an example endpoint would be like runs, creates,
25:42
like post runs create. Now I'm creating an agent run,
25:46
there's right at the front of the queue, something is holding
25:49
my connection. It actually depends on whether
25:51
I'm streaming, whether I want to stream the results back, or
25:54
whether I just want to fire off the run.
25:56
A simpler example is if I just want to fire off the run, I
25:59
don't care about streaming it back.
26:02
I, I call this endpoints the my, I pass the parameters like the
26:08
whatever the agent is that I like to call, whatever the
26:10
initial state is. And then a, you know, an advance
26:15
is placed onto this queue. The there's a consumer at the
26:18
other end of this queue that sits in front of the node
26:21
process. And for every queue event that
26:24
is received, it starts the node process and passes in that
26:27
state. And so the nice thing about
26:30
having that queue there is that if my, if I haven't like totally
26:35
scaled or if Lang chain in this case hasn't scaled up all of the
26:38
agents so that it can handle a peak load, none of the none of
26:44
my agent execution requests get dropped or lost.
26:47
They will all eventually get processed.
26:49
Yeah, just slower, but just be slower.
26:51
Yeah. And you can scale the queue or
26:54
Lang chain provides some ways for you to scale the queue.
26:56
If you are hosting this yourself, you could also have
26:58
your queue do some auto scaling or.
27:00
Well, let's talk a bit about that because I guess one of the
27:03
huge value props here is that you don't have to deal with that
27:07
scaling issue. You don't have to worry about.
27:09
Yeah, and but what are the trade-offs?
27:13
There's always going to be something that you're.
27:16
What always is there's no, there's no free lunch.
27:21
Yeah, there are trade-offs. I mean, so this is like a lot of
27:23
infrastructure we didn't have to build and it's infrastructure we
27:27
could have built. But you're with Alice is also
27:29
like a SAS tool. I mean, there's the agent, but
27:32
then there's the dashboard and then there's we have other
27:35
background processes that are happened.
27:37
So we're building a whole application and then we're
27:38
building this agent that needs to hook into the application.
27:41
And you rebuilt it in October when you already had something
27:45
out. So you can't just turn around
27:48
and spend a year and a half building out the infrastructure,
27:51
I imagine, so that you can get this agent going.
27:54
Yeah, there's a whole meta here, which is that our customer base
27:57
was growing, our old product was starting to crumble.
28:00
You were going fast. We had to go.
28:01
Fast go to market was not slowing down for us.
28:05
We had to get this thing out really quickly.
28:07
Yeah. And it was, it was quite the
28:08
crunch. It was, it was super painful.
28:10
But we're on the other side of it now, so I can laugh about it.
28:14
Somebody I heard somebody talk about that being type 2 fun.
28:17
Yeah, yeah, that's exactly right.
28:19
It was Type 2 fun. There's a lot of late nights.
28:22
My skin looks a lot better than it did in March or whatever,
28:26
February or something. You got some sun.
28:28
Thank you. Yeah, not as pasty.
28:31
I remember the first time that I met.
28:32
It's a little more pasty. The let's see the trade-offs, I
28:37
mean, so super advantageous for us to do this just to get to get
28:39
off the ground really quickly and to not have to worry about
28:41
building that infrastructure. The I think some of the
28:46
trade-offs were, you know, when we started using land graph
28:49
cloud, it was newer products. We did run into some, some
28:52
scaling issues at times. There were some limits on the,
28:55
the, the extent to which they could scale an individual what,
28:59
what they call deployments. So in fact, we actually had to
29:01
create lots of deployments. Those, those queues or those?
29:06
You can think about this as like, you know, if you're
29:08
deploying a service in Amazon, you like deploy UCS service.
29:14
If you are deploying an agent in Landgraf, you created a a
29:17
deployment and we had a lot of volume.
29:21
We had spiky volume and that led to us running into, you know,
29:27
connection time outs during those peaks.
29:30
And obviously one of the things we could do to make this, make
29:35
this better is to smooth out our, our request traffic, which
29:37
we have done. And that's we take full
29:41
responsibility for our query patterns.
29:44
But we are also, you know, the we are exceeding some of the
29:47
limits of, of what they could do.
29:49
They could the extent to which they could scale an individual
29:51
deployment. So the way we got around with
29:54
this is we created 10 different deployments and then we had our
29:57
all of our clients is round Robin across them and now it's
30:00
been fixed. So we're consolidated again and
30:03
then other challenges, we have this deployment pipeline that
30:08
deploys all of our back end services.
30:10
Our code is stored in a mono repo including the agent itself.
30:14
So usually at any given time, the the main, whatever the head
30:18
of main in our mono repo is, represents the state of the
30:21
world, what should be in production.
30:24
And our deployment system works for all of our back end
30:27
services. You we've got a separate set of
30:30
infrastructure now for our agents.
30:32
So if we want to also auto deploy and auto roll back our
30:35
agents, there's all of this additional deployment
30:38
infrastructure we need to create to support that.
30:41
And our deployment system gets way more complex.
30:45
And then last thing that comes to mind is on the observability
30:48
front. You know, agents are software
30:51
just like everything else. And we want to see the logs, we
30:55
want to see the traces. And we typically pipe all of our
30:59
our observability telemetry logs, traces, metrics to
31:03
Datadog. And this is different than
31:06
laying, which what's the service laying language?
31:09
Yeah. And this is this is another
31:11
small, small complaint about using Lang Lang chain.
31:15
The ecosystem is that they have Lang Chain, Lang Smith.
31:18
Everything is Lang graph. It's the year of the Lang.
31:20
Man, it's the year. It's not the year of the Lang.
31:22
Man, there's more. Yeah, ironic because it's hard
31:25
to remember. Yeah, and it's all about memory.
31:30
And then there's two sets of two sets of documentation for both
31:33
Python And TypeScript. So you're like, am I on the
31:36
right documentation? Page right now.
31:37
God, that's painful. Yeah.
31:40
Gotten better at navigating that.
31:41
I think they've gotten better at positioning there and.
31:43
Figuring the products, Yeah, call something different.
31:47
Yeah, the lanes have been exhausted.
31:49
Yeah, it's the calling card. We on the observability side,
31:53
Lang Smith offers a lot out-of-the-box, which is great,
31:55
but I'm a former infrastructure and observability engineer.
31:59
Like one of the core tenets of that role is like, all of the
32:03
telemetry should be in one place so that when something's going
32:05
wrong, the developer. You don't have to search.
32:08
You don't have to go looking for it.
32:09
And and if you're not, if you're a product engineer who doesn't
32:12
know what the whole observability stack is, you can
32:14
just like jump into that tool and it's got everything.
32:17
All of the logs are there. Yeah.
32:18
You don't need to remember, you know this, there's a customer
32:21
that's complaining or there's an outage.
32:23
Or, oh, there's. This is everything except for.
32:25
Except for that one one thing, which by the way, is where the
32:28
bug is, right? Like so getting some of our logs
32:31
and other telemetry into Datadog from Lang Smith was, is was not
32:35
that easy. And it's still a little, a
32:37
little bit painful. I think I know that they're
32:40
working on better integrations with with Datadog to make that
32:43
and make those pipes. That feel that seems relatively
32:48
trivial. I guess it's a huge problem for
32:52
you, but it's like there's nothing that's blocking them
32:55
from making that a easier process in six months.
32:59
You know it, it depends on how, where you think these agent
33:01
framework companies are going. Oh yeah, I think sort of there's
33:04
an incentive for Lang Chen. To own everything.
33:07
To, I mean they want to lock you into their platform and they,
33:11
they're all of their agent frameworks are open source.
33:14
A lot of their other tooling is open source.
33:16
So the the value, the way that they capture value is by getting
33:19
them who are you using their hosting solution and by you
33:22
using their observability solution.
33:23
So making the pipes easier so that I can get to Datadog may
33:28
not be like #1 priority, but they do want to make their
33:32
observability tool really good too.
33:35
So it kind of ends up becoming a question, yeah.
33:39
Do you think that agents are different enough to require a
33:43
different cloud? Will we will we end up using
33:47
dedicated clouds for agents? Or will we just deploy agents
33:51
like we deploy other software to existing clouds?
33:53
Yeah. Is it complex enough to warrant
33:57
its own cloud? I think that's I, I'm not sure
34:07
about that. I think we talked about the
34:09
infrastructure pieces that are required to to host an agent.
34:13
I think that you, you could imagine that all of open source
34:17
versions of all of those or a single open source solution that
34:21
combines those components, something like an elastic search
34:25
or a Kafka. That is, you know, everyone
34:28
knows how to work and everyone knows how to deploy and AWS is a
34:32
one click deploy for. So I, I don't think that that's
34:38
out of the realm of. I'm surprised AWS hasn't stolen
34:40
Lane Graph actually and run it as a service on AWS now that you
34:44
say that. They have, they have their own
34:46
agent framework and they have AWS Bedrock and they have
34:50
Bedrock agents. But their agent framework, it's
34:53
one of those ones where it was older, right?
34:56
And I don't know anybody who's using that.
34:58
I don't know anyone who's using bedrock agents.
35:01
No offense to the to the bedrock team doing great work.
35:04
We use Bedrock for in France. Bedrock has a has a partnership
35:09
with Anthropic where they can host some anthropic models.
35:11
So. But why bedrock as opposed to
35:14
just going to anthropic? Because of the reliability, we.
35:16
Originally switched from from anthropic to bedrock because of
35:20
rate limiting and we couldn't get high enough rate limits
35:22
through anthropic. You got to know someone, you got
35:25
to have an Angel. And so Bedrock gave us high rate
35:30
limits. Now I think there there's a
35:32
difference in cost between the two platforms.
35:35
And so this is like where you get your procurement team
35:38
involved because? Sure.
35:40
You don't want to give us these better rate limits.
35:42
Yeah, well, that's a great point.
35:46
So then. It's.
35:49
Just a ghost. Yeah, just a ghost.
35:52
There's a fitting for this room. It's.
35:55
Super cool space. It is cool.
35:58
Thought about getting a bunch of cigars.
36:00
Oh my God, that would be cool, but I wouldn't be able to like
36:02
step out of work to come do the podcast.
36:04
Yeah, no, or we don't have to smoke them.
36:06
We just chew on them and look cool with them.
36:08
Just have one like smoking in a. Tray Yeah, I bet they would be
36:13
very excited about us doing that.
36:18
We could do an after hours thing.
36:19
Yeah, well, what else man? There's a a few other things
36:23
right? Like I I like the trade-offs
36:25
with lane graph and hosting an agent there.
36:27
Lane, Lane Cloud. Lane Graph Cloud.
36:30
They missed an opportunity to confuse the shit out of people
36:33
more and just call it lane cloud.
36:35
I think lane cloud would have worked.
36:37
That's the next iteration. You actually, you know, what I
36:40
would love to talk about is you as a observability guy.
36:46
How do you see observability being different now with agents,
36:49
if at all? And it can, it can be totally
36:52
cool. If you say nothing, it's same,
36:55
same. It's still software.
36:56
No, no, I think this is a rich question.
36:59
We might have to noodle on it a little bit.
37:03
Yeah. So my background and
37:04
observability is I, I was, I was second infrastructure engineer
37:07
at BRACS and had like an amazing mentor who who grew me a lot of
37:11
Braxt, built a lot of cool production infrastructure there
37:14
and then at some point established the observability
37:18
team at Braxton. So we had a couple of folks who
37:21
are specifically focused on developer tools related to
37:25
instance and understanding how systems are behaving in
37:28
production. In practice, that's like we, we
37:31
set up Datadog, we set up auto instrumentation for all of our
37:34
services. We had like 30 or 40 different
37:36
micro services running in our Kubernetes cluster.
37:39
All of them are sort of uniformly and automatically
37:42
instrumented. We had a instant process and and
37:47
tool that you would use for for kicking those off and for
37:49
resolving them and and some other observability tools as
37:54
well for the things like fraud detection and a client side.
37:57
Observability is actually interesting area of area too.
38:02
That means like, like capturing errors that occur on the client
38:06
as opposed to on the server. Things that you know, for
38:09
example, there's like exception that occurs in your in your web
38:12
app and it's happens in the user's browser.
38:16
How do you send that back to a server somewhere so that someone
38:19
can look at it, someone knows that something went wrong?
38:22
Yeah, there are a lot of tools for doing this today.
38:25
Like I think Log Rocket was one of the first ones.
38:28
Post hoc has a session recording feature and it's it's pretty
38:31
cool with all the browser Dom events.
38:34
You can reconstruct exactly what happened on the on their
38:38
experience. So it's they're not actually
38:39
recording their your screen, but they might as well be because
38:42
you look at the recordings. It's the exact same thing that
38:44
happened. I think observability with
38:48
agents could be really interesting.
38:49
One is I think that just agents, so there's there's sort of two
38:52
separate things we can talk about here. 1 is like what does
38:55
it mean? How do we observe agents as more
38:57
of our production systems become agents?
39:00
Like how do we keep track of what's happening and make sure
39:02
that they're performing reliably and and reconstruct things after
39:07
the fact and and then the other one is like, what is
39:11
observability for just regular software look like when a world
39:15
of AI agents to tackle the latter 1.
39:18
I think agents have a lot of potential to automate the sort
39:22
of production infrastructure or site reliability and process.
39:25
I mean, so much of what I do in my day-to-day is someone says
39:29
there's an issue, you know, customer reports an issue or
39:32
customer success person reports an issue.
39:34
I get like an organization ID, maybe a URL, and then I can
39:38
roughly, I guess it happened within the last hour or
39:41
something like that. Maybe I get a loom or a session
39:45
requirement. That's crazy.
39:48
And then I take the whatever pieces of it, whatever clues
39:51
I've got and take them to Datadog or to another
39:54
observability tool and or, or maybe our, you know, I can query
39:58
our, our production access to our database.
40:00
So if I can look up a record, if it seems like a record might be
40:02
in a bad data state and I have an identifier, I can query for
40:05
that in our post res database. And in Datadog I can plug in
40:10
like a trace ID or the organization ID and get the logs
40:13
related to that request or org and.
40:17
And I start piercing together. You're probably asking people on
40:20
Slack. It's funny that you're
40:22
mentioning this because we had Willem in here yesterday and
40:24
he's working on AISDR. Yeah, no, sorry.
40:28
That's what you're working on. He's working on AISRE and.
40:33
I built. I built one over the.
40:35
Really. Yeah, no way.
40:36
I won a hackathon building an AI.
40:38
Yeah. Yeah, that's awesome.
40:40
It's so funny. He's so he's doing that.
40:42
He built the knowledge graph on all the stuff that's happening
40:46
in Slack, all the stuff that's happening in the logs, all the
40:49
stuff that's happening in your environment, and how do you
40:52
connect all of that and know what's valuable?
40:55
Tell the agent. Sometimes he was saying that the
40:57
agent will just go and do its thing and come back with
40:59
nothing, but it will tell you, hey, I searched these places and
41:04
so then you don't have to go and search.
41:06
Them. Yeah, that's super valuable.
41:07
Exactly. You have to really trust the
41:09
agents doing its job well, especially if it comes back with
41:12
nothing. Yeah.
41:13
But you're like, Are you sure about that?
41:15
I don't believe you. Yeah, I'm gonna go check.
41:17
In agents like our agents just, they want to please, you know,
41:20
language models want to please, so they have.
41:22
It's probably rare that they come back with nothing.
41:23
Products like that have a lot of potential.
41:26
It's a it's a clear set of tools that you would integrate with
41:30
clear set of data sources. And you're just correlating this
41:34
commit with this error blog line with, you know, this bad record
41:38
in the database over and over again and then summarizing what
41:41
happens, creating a linear ticket out of that report.
41:45
You're responding to either the customer or the internal user.
41:47
And agents can can automate a lot of that process.
41:50
You know, so we see them automating like the code, the
41:52
the process of writing the code and shipping it.
41:54
Yeah, we don't really see them automating the the production,
41:56
what's happening in production? Yeah, root cause analysis of
41:59
what's going on, why did that fail and and even preemptively
42:04
saying you sure you want to put that kind of scale on that kind
42:08
of load on? Yeah, yeah.
42:09
That's that's another piece actually, AI code review, which
42:12
just happened somewhere in between those two things and
42:14
there are some cool tools there. I, I know someone Meryl Lutsky
42:18
from, from graphite is a one of the leading AI code review tools
42:22
and we use graphite and love it. It pretty consistently catches
42:25
things. Now the balance you need to
42:27
strike there from a probably respective is how do you make
42:30
sure that you are catching things and providing useful
42:32
suggestions without being noisy, because nothing devastate more
42:36
than the noise. That alert fatigue.
42:38
And I know a friend that plugged in an LLM to their GitHub
42:42
instance and then they turned it off like 6 hours later and they
42:45
were like, this is not going to work.
42:47
Yeah, yeah. And so really recognizing what's
42:50
valuable and what's not, that's a huge thing.
42:53
Now when it comes to the observability of the agents,
42:56
yeah. Yeah, I think so.
43:00
I think that agents are kind of complex and scary and new.
43:04
It's important to remember that agents are just like software.
43:07
They're just regular software programs.
43:09
And so observability in a lot of ways is the same.
43:13
You know, you should you should be instrumenting them with
43:16
metrics, logs and traces. You can do this using open
43:20
source libraries. Well link chain and link graph
43:24
have a lot of the stuff out-of-the-box.
43:26
If you use that you get that in like Smith.
43:27
But open telemetry is the standard that has been used for
43:31
most software monitoring tools thus far.
43:36
And you can use a lot of those tools to instrument your your
43:38
agent. Yeah, depending on the agent
43:41
framework that you use, they might already have hooks for
43:44
like emitting open telemetry compliance traces at each step
43:50
or or similar things for for metrics and logging.
43:54
Otherwise you might have to do a little bit of your own build a
43:56
little bit of your own middleware or maybe wrap the
43:58
agent framework so that it it instruments it.
44:01
But all of those tools work just as well.
44:03
Now if you use a cloud solution like land graph, you are
44:08
restricted in terms of what other things processes you can
44:12
run alongside your agent. And that is is something that
44:17
you should consider when you're thinking about observability for
44:19
the agent. Because for example, a very
44:21
normal way to collect metrics, logs and traces with traditional
44:26
software would be to have a sidecar container that runs like
44:29
a an agent process like a stats D agent or Datadock has dog
44:34
stats DI think, which is their flavour.
44:38
You can't do that in a cloud environment that you don't have
44:41
a lot of flexibility or control over.
44:42
Interesting. So you might only be able to do
44:44
that in something like AWS or GCP or Azure.
44:48
So something to consider when you consider when you're
44:51
thinking about where to host your agent is what options does
44:54
this give me in terms of observability and getting them
44:56
those just metrics out of them it?
44:57
Restricts your architecture decisions.
45:00
Like you can't just go and have a blank canvas.
45:03
Yeah, you go and trawl in the lines.
45:05
Yeah, and none of the agent clouds are as sophisticated as
45:07
them as the big clouds, so. Are you linking up these
45:11
observability metrics with the evals or are you fully
45:14
separating them or are they kind of mixed and matched?
45:17
How do you look at those? Yeah, evals.
45:23
I think a lot of companies, most companies are not doing evals
45:26
today. Everyone's interested in evals.
45:28
But it's all vibes. But it's all vibes.
45:31
There's a lot, I mean a lot of vendors are talking about how
45:34
they how they have support for evals.
45:36
Usually those features kind of take the same shape.
45:38
It's like you can curate a data set from some of your production
45:42
traces and then you can run your agent in times for in items in
45:50
the data set and it will tell you, well each time you run your
45:55
agent, it will run a set of heuristics against the output.
45:59
And for each of those heuristics you get like a score or a label.
46:01
So a common one is to detect hallucinations.
46:06
We actually use this in our message writer.
46:08
So we have an eval that actually automatically runs on some
46:13
percentage of production. Nice.
46:16
Traces for the agent that generates the messages.
46:18
And this is also done with Lang Chung.
46:23
This one is done. We we've implemented and
46:26
Lanchain offers this capability. This one we just, we were trying
46:29
out another tool called Arise, which is another LLM or AI
46:33
observability tool. And they also have support for
46:35
evals. And so this one we implemented
46:37
using Arise. I don't have strong opinions for
46:39
the oh, this, this is using their core platform.
46:43
Although Phoenix is really interesting and I think I've,
46:47
I've been interested in Phoenix for a while as a way to deploy
46:50
sort of like production grade observability tool locally for
46:56
all of my developers on my team. So you know, we're running these
46:59
agents. We, we can run our entire
47:01
application locally using a single command, which is suite
47:05
starts up a bunch of services, starts up the client, starts up
47:07
the agent server, which is built on my graph.
47:11
What would be really nice is to also start up this dev server
47:15
that allows you to view all of the agent traces that are
47:19
happening on the on your local agent and maybe even iterate on
47:22
some of the prompts through that server.
47:24
I think Phoenix might allow us to do something.
47:26
Like nice. OK, cool.
47:27
Much better local development experience for building the
47:29
agent. All right, so sorry I got you
47:32
distracted with. Evals in production is the main
47:35
one we have running or one of the ones we have running today
47:38
is detecting hallucinations and messages.
47:40
Obviously super awkward if you say like hey Demetrius, Congrats
47:44
on passing 10,000. Subscribers and you're like, it
47:48
was a million, it was a million. My ego is hurt.
47:51
Exactly. I actually that might generate a
47:53
response, yeah. Yeah, not the kind of response
47:56
you want probably, but actually I was telling you this earlier.
48:00
Like I imagine a lot of the stuff that AI generates or Alice
48:04
generates specifically is a lot better than the human stuff
48:08
where I was saying some random shit in my that's so funny.
48:14
Yeah, we and we get scrutinized a lot on our messages too
48:17
because you know, users will log in and they'll be like, you
48:20
know, I saw this slightly weird message that you sent to 1
48:23
prospect and maybe the prospects LinkedIn says they work on like
48:27
Intuit and our data set, you know, our e-mail says like
48:31
Intuit Accelerate or something like that because.
48:34
So it's a little bit off. Yeah, it's not.
48:36
A little. Off or it's like when you're
48:38
spelling is it desert or desert? And yeah, it's the same when you
48:43
look at it and you don't know how to spell like me.
48:45
And so you're like, yeah, I said dessert, right?
48:49
There's one other thing that's really funny about this is we
48:51
often we try to show sources in our in the research alongside
48:57
the message that we generate when we do it in the product.
49:00
And often times we'll have customers say like why?
49:04
Why'd you say like we, you know, maybe the message says we helped
49:07
a similar client in the healthcare services industry
49:11
save, you know, $50,000 in the last year using this solution.
49:16
And our, you know, our customer will say, where'd you get that?
49:19
That like doesn't sound true. And then we pull up the
49:21
research. And sure enough, there's a case.
49:23
Study on their website that says that so many.
49:26
Yeah, it's, there's a lot of that that happens in the
49:29
day-to-day and it can be really delightful when it does happen.
49:33
But we have also, you know, we're held to a high standard
49:36
for that messaging. I think certain customers are
49:40
more savvy when it comes to sales and understand that a lot
49:42
of times sales reps will bend the truth or or say things to
49:46
provoke a response. And Alice can do that.
49:48
If you want her to you, you can nudge her.
49:50
Be a bit more sassy, yes? We effectively have Sassy mode
49:54
if you want to enable that. We call it brand voice.
49:57
Oh, interesting. But, but also, yeah, we want to,
50:02
we generally like want to be factual.
50:05
Everything that Alice should say should be based in fact.
50:06
And it's it's a product design problem to make sure that the
50:11
the sources are alongside the messaging so that you can
50:14
correlate. You know, if a user wants to
50:16
double check something, they can do that easily.
50:19
So then back to the evals, you're saying, all right, we are
50:22
doing evals with Lang Smith. We're also doing evals just kind
50:28
of on vibes and figuring it out as it's coming out.
50:32
And you have this cycle of whatever 20% did you say?
50:37
So and so we do this in a rise right now and jury's out on
50:42
which, you know, in our next episode we can talk about which
50:45
of these eval platforms I like better.
50:46
Interesting, so you're doing it both with Lang Smith and Or does
50:50
Lang Smith only do observability?
50:51
Lang Smith has an eval offering and we've used an earlier
50:55
version of it. We don't use it right now and
51:00
our team has been experimenting with Arise, which is another LM
51:02
observability platform and they have also an eval offering which
51:05
is been playing around. That's fascinating.
51:07
So you have evals in Langsmith, but you're like, we're just
51:12
going to stick to the observability piece of it.
51:15
We we tried an earlier version of it and weren't satisfied.
51:18
I felt like the experience of setting it up was a little too
51:20
hard. And you know, we, our team is
51:23
very curious about AI tooling and we're always kind of
51:28
shopping around for whatever the latest and greatest is.
51:31
So we, we test drive a lot of products and for evals right now
51:34
we're test driving arise very cool.
51:37
This eval we sample like 1% of our total.
51:42
Yeah, we have too much volume for, because we, we're running
51:46
an LLM in our eval, so LLM as a judge.
51:52
And so like it could be expensive if we ran it for 100%
51:54
of our traces. We, we only do it on like 1%.
51:57
But what's cool is it gives us this, this number between zero
52:00
and one, which represents, you know, that the average across
52:04
all of the traces in a period of, of whether they have
52:08
hallucinated or not. So as we, if that number is
52:12
close to 1, it means like we're not hallucinating.
52:14
If the number is close to 0, it means we are generally
52:17
hallucinating. Exactly.
52:20
You're burning that, right? This SDR is writing emails from
52:22
their satellite connection, you know, from deep Playa.
52:26
Yeah. So that's what we.
52:31
That's a number we watched to sort of as a bellwether of
52:35
whether or not like a recent change to the prompt has, has
52:38
made it less reliable or something like that.
52:41
And what the research or what the eval is doing is it's
52:43
comparing the inputs, which include this research report
52:46
which we've assembled and the outputs, which is which of the
52:50
e-mail that we've written basically saying, hey, is there
52:52
anything in the e-mail that doesn't match the research?
52:55
OK, I have a very clear picture of how you're running your
52:58
agents. It's you're hosting it with Lane
53:01
Graph. You've got Lang Smith as the
53:04
observability. You're exposing an API in a way
53:07
for that agent that you can then build services off of.
53:11
You are relaying whatever that output is.
53:15
You're taking 1% of that and feeding it into a rise right now
53:18
and saying we've got these evals and you're matching it up with
53:23
this is what the research says and this is what you said.
53:27
How accurate is it? Right.
53:29
That's effectively how the eval works.
53:31
Yeah, we, we send, send all of our traces to arise in the same
53:35
way that we send them to Lang Smith.
53:37
These are the agent traces. And you know, we, we run a bunch
53:40
of agents for Alice. One of these agents is the
53:43
message writer. And the input for the message
53:45
writer is the research report, some demographic info and the
53:49
output is the message. And so for the traces that are
53:53
message writer traces, we for 1% of those we run this
53:58
hallucination eval. Wait, but then what does?
54:01
Because you've got Datadog, right?
54:02
And Datadog is your observability, like the system
54:06
observability. What is Lang Smith doing then?
54:09
Lang Smith. So I mean some of these things
54:11
are duplicated. I mean we have Lang Smith is
54:13
allows you to do logs, metrics and traces too.
54:16
For the agent, for the for. Only what's deployed in Lang
54:19
graph cloud. I see.
54:21
Which is our agent now. It's nice to get all of that
54:24
data into or into Data Dog if possible 'cause that's where it
54:27
is for all of our services. Yeah, I remember you saying
54:30
that. No, that makes a lot of sense
54:32
because it's like data dog can't cover what's happening in Ling
54:35
Ling cloud. Exactly.
54:38
I mean, there's, there's two walled gardens right now.
54:40
Yeah. Interesting.
54:42
I think the Lang Chain team, I don't, I don't want to publicly
54:45
promise anything on behalf of them, but I'm pretty sure that
54:47
they're working on some kind of integration that.
54:48
Allows they understand. Pipe more of our our data to to
54:51
Lang to data. But they're not going to let you
54:53
put a data dog instance in Lang Cloud.
54:56
I don't think that their ambition is to become a general
54:58
observability platform like to have all our back end services
55:02
send their logs to Lang Draft Cloud that would that's probably
55:06
outside of their scope, yeah. Yeah, that makes sense.
55:10
Yeah. So then why would you want
55:12
Datadog in there? It would be better if you just
55:15
took all of that from Lane Graph and just said, hey, we're
55:20
throwing all the Lane Graph traces and whatever, whatever
55:24
your cloud is doing, we're throwing it into Datadog.
55:27
Yeah. And that I mean that would leave
55:28
them. Their value add would then be
55:30
like running the actual agent hosting that API in queue that
55:34
sit in front of the agents. Which as we talked about
55:37
earlier, less, yeah, it may or may not be needed.
55:41
Yeah. So everyone's being squeezed
55:43
right now. You've got Data Dog, which is
55:45
trying to become Landgraf Cloud without the hosting, I guess.
55:48
And then we've got Landgraf Cloud that's trying to become
55:50
Data Dog. And then we've got AWS is
55:51
probably doing both and so they're all kind of trying to
55:55
carve out more of, of the of the, the tooling and
55:59
infrastructure that needs to surround the agent and own it.
56:03
But there's definitely a lot of overlap between these different.
56:05
Tools, yeah. And then you have Arise that's
56:09
doing the Arise. Is sort of a Lang Smith
56:11
competitor. Exactly, and they're really
56:15
focused on LLM observability specifically.
56:17
So just more of a better specifically with with Lang
56:20
Smith and Datadog. Although Datadog has their own
56:23
LLM observability suite. Yeah, but.
56:25
Arise is an API that you're hitting or you're sending data
56:28
to. It's like a cloud, a hosted
56:30
version also. Arise is open telemetry
56:33
compliance, so. Oh, so yeah, OK.
56:37
In the you know we can just instrument our agents using some
56:40
open source libraries and all the data flows into arise.
56:44
Interesting. Yeah, dude.
56:45
Well, this has been awesome, man.
56:46
This is. I appreciate you being so
56:48
forthcoming. We'll see if your PR team allows
56:51
half of the shit that we just talked about to get out into the
56:54
world. I.
56:55
Think the technical stuff is very good?
56:59
No, this is fun. We could have gone on for
57:00
longer. Yeah, I think we could.
57:02
Well, we'll continue the conversation tonight.
57:04
I'm looking. Forward.
57:05
To it.