Latest / Agentic Conversations (formally mlops.community) / MCP Servers Are Becoming the UI for AI Agents

Transcript

MCP Servers Are Becoming the UI for AI Agents · Agentic Conversations (formally mlops.community)

0:00
All right, so at the beginning of April I went to New York for
0:03
the MCP Dev Summit, and while I was there I got the chance to
0:07
record a few podcasts with attendees that were at the
0:11
event. This is one of those
0:12
conversations. Hope you enjoy.
0:14
Gateways are very interesting, I think because they're solving
0:18
like a very similar issue where hey, there's now this new way to
0:22
do things. It's called MCP.
0:24
It's how you connect external services to your AI and with
0:30
great power comes great responsibility.
0:31
You're a security product. Focus on the security aspect,
0:34
don't also focus on like productivity.
0:37
Naseem Al Naji is the Co founder of MCP Cat, a platform that
0:42
helps developers debug and understand how AI agents
0:46
interact with their MCP server. Wait, wait.
0:49
Before you jump forward. Yeah, the goal idea.
0:52
Yeah, explain more of that because that feels like
0:54
something novel. I haven't really.
0:55
Heard Yeah, yeah, yeah. When you're building MCP server,
0:59
you're actually a the disconnected from your end
1:02
customer, right? You might have your MCP server,
1:05
but the person using that is interacting through it through
1:09
clawed or cursor or a clawed desktop, and those companies own
1:16
a big part of the experience. Oh, this guy says there's a
1:19
problem. I don't know why.
1:20
Oh, he's using Cursor? What version is my machine?
1:22
Dude, Yeah. We're on my.
1:24
Machine. Yeah, exactly.
1:26
We're on my machine. Yeah, well, I like this.
1:28
Let's pull on that thread a little more, because.
1:41
It was a very interesting, Yeah, I think that they're very
1:43
valuable. I'm surprised by how many there
1:46
are, but I guess that's like good, you know, good, good
1:50
competition. I think a lot of people pivoted
1:52
into the space and the most logical first step is a gateway,
2:01
and especially if you're a security company already and
2:04
you're doing like, package scanning.
2:06
Yeah, OK. What else should we scan?
2:09
MCP servers. Yeah, I think also it's a
2:14
repeated model because when LLMS came out, LLM proxies were, I
2:21
mean everybody, I mean, so many established companies use LLM
2:24
proxies. So it's kind of like the next
2:25
step. Oh, OK.
2:26
Well, like let's just get another proxy.
2:27
Yeah, we should have MCP proxies and LLM proxies, though we need
2:31
our gateway to check the cost. Yeah, yeah.
2:36
Yeah. Actually now that I think about
2:37
it, is it not the same thing? Like they just proxy they just
2:45
are the gateway and then if it goes to an MCP server through
2:48
the LLM then OK I. Mean it's it's very similar.
2:54
I think that proxy is very good because it can catch what people
2:59
are chatting with LLMS and if it doesn't like something it will
3:03
like kill the user session, right?
3:06
So you can actually like filter out what's getting sent to an 8
3:09
to an LLM. You can also, you know, block
3:14
all network traffic to other LMS. Gateways are very
3:17
interesting, I think because they're solving like a very
3:20
similar issue where, hey, there's now this new way to do
3:24
things. It's called MCP.
3:26
It's how you connect external services to your AI.
3:30
And with great power comes great responsibility.
3:34
So here's a ton of like gateway solutions.
3:37
Yeah, totally. And I I guess a lot of the
3:39
gateway value prop is hit our gateway and you don't have to
3:44
load up 5000 servers, you load up our gateway.
3:48
It's the one server to rule them all and we will figure out the
3:51
most efficient way to grab whatever tool you need from
3:55
whatever server you. Yeah, I personally don't I, I
3:58
personally wouldn't use them for that.
4:00
No, like I, I, I, I think having as a customer of an MCP, right,
4:07
I would prefer that the MC PS I use are like the ones I use kind
4:12
of like it's OK if they're proxied through a gateway, but I
4:15
don't want the gateway to do the routing for me.
4:18
I think personally, I think it's, I mean, I think that LMS
4:24
are good enough or the, the MCP clients are good enough at
4:27
picking the right tools. At least that's what we see at
4:29
MCP Cat like that. I think I do see some gateways
4:34
try to say like, hey, we'll give you all the tools and use our
4:39
gateway to to like navigate them.
4:41
And I it just feels like an unnecessary extra stab.
4:43
Like it's an anti pattern. Well, I think it's like, I think
4:47
like it's important to focus the solutions, right?
4:49
If you're a security product, focus on the security aspect.
4:52
Don't also focus on like productivity.
4:55
I started a cybersecurity company 8 years ago and I sold
4:59
it. So I think like that's, that's
5:03
like a more important, it's more important to understand your
5:06
ICP. And so like, I think as like a
5:09
user, I'd, I'd want just like, hey, this is like how we do
5:11
security and usability is handled by different solutions.
5:17
But that's my personal. Yeah.
5:18
Well, I like this. Let's pull on that thread a
5:20
little more because you're not going to have five different
5:23
gateways, right? Like so if there is a usability
5:26
feature and it doesn't have the security, you're not going to
5:30
like go and then also have a security gateway and then you
5:34
like have two gateways that it's going through.
5:36
I, I think like what I, what I just mean is like if I'm going
5:39
to use a gateway, I don't want it to be opinionated on which
5:43
version of a customer's, on which version of MCP server I
5:47
use. Like I want to pick and choose
5:48
the MCP servers that are loaded up into it, if that makes sense.
5:53
Yeah, yeah, that's like a like, yeah, I see that because what
5:56
if, you know, a new version comes out and it enables
5:59
different tools, I guess, for example, Web Flow is one of our
6:03
customers. Like if I want to use like a new
6:06
update of their MCP server, I don't want my gateway to be the
6:10
reason we can't use it. Yeah, yeah.
6:13
Yeah, especially because it's like you're constantly as that
6:15
gateway having to stay up to date with whatever the external
6:21
servers are doing. And which you, you kind of in a
6:24
way, if you're doing the security thing, you also have to
6:26
do, you have to scan it and make sure there's no malicious stuff.
6:29
But I see what you mean. It could be a potential blocker.
6:33
The thing that this makes me think about is how you're seeing
6:37
success with folks that are using MCP.
6:41
Yeah, yeah, totally. So I guess for context, if
6:44
people don't know what MCP CAT is, we are a like product
6:48
analytics layer for MCP servers. So if you're making an MCP
6:53
server and you want to make sure it's good, you use us to tell
6:58
you that, right? So what we do is we get feedback
7:04
from the agents using your MCP servers and share that with you
7:08
in a way that's statistical. We find their goals with your
7:11
MCP server and we help you understand if those goals are
7:13
succeeding or failing. Are agents hallucinating on
7:17
certain tool calls? And so you know, you, you see a
7:21
lot of the, the problems you will have with MCPA.
7:24
Big reason is that most of those MCP servers are half baked
7:27
hackathon projects that never got any real maturity from the
7:32
organization. Like like the main way they hear
7:34
about problems are through other issues.
7:37
That's that's not how we find problems in mature software
7:40
today. So how do we fix that And and so
7:43
wait? Wait, before you jump forward
7:46
the goal idea. Yeah, explain more of that,
7:49
because that feels like something novel I haven't really
7:51
heard. Yeah, yeah, yeah.
7:52
So when you're building MCP server, you're actually a bit
7:58
disconnected from your end customer, right?
8:02
You might have your MCP server, but the person using that is
8:06
interacting through it through Claude or Cursor or Claude
8:10
Desktop, and those companies own a big part of the experience.
8:16
But at the end of the day, if your MCP server cannot enable
8:19
your customer, then they're having a bad experience, right?
8:24
And so because there's these third parties in the way, you
8:27
actually don't know what your user is prompting at the like to
8:30
get to the like the MCP server. So if I want to use like Web
8:34
Flows MCP server to update my blog on my website, it's going
8:39
through cloud code and Web Flows MCP Server actually has no idea
8:44
that that's, that was my goal. Or if I wanted to translate it
8:46
into Spanish and have a Spanish version of it.
8:49
They have no idea that that's the goal.
8:51
They just see, oh, it's calling this tool.
8:53
Yeah. And So what we do is we actually
8:56
extract the user's overall goal with the MCP server.
9:02
And we do that by augmenting any MCP server's tool list with
9:09
required fields that tell the agent, hey, tell us a little bit
9:12
about why you're calling this tool and how it fits into your
9:15
overall goal. It's kind of like a micro micro
9:17
user interview and. So the agent has to say that
9:22
when it sends a request. Yeah, yeah.
9:24
And that's cool. And so that's a lot of data.
9:27
That's like a ton of tool calls, right.
9:28
So how do you stitch that in together into a reasonable way?
9:32
And so we'll like take all of those, stitch them together and
9:35
you have like a play by play of what the agent was doing, why it
9:39
was doing it. And you can even see extract
9:44
like higher level goals with your MCP server like what is
9:48
What are the top 10 use cases for this MCP server?
9:51
Does it not add to the cost? Yeah, it does.
9:54
It adds like a fraction of pennies to the cost, right.
9:59
I think that what's crazy is we have over 100 million events
10:03
logged every month and no complaints from customers
10:07
because that's just like the way people are paying for these.
10:11
I think tokens are actually quite abundant and so many
10:17
tokens are used in the process of even deciding to make a tool
10:21
call that we add .05% to that cost and so.
10:26
And the value you get from those .05 tokens or that that cost
10:31
increase is so much more. If I can understand how my
10:34
customers are using my MCP server and why, yeah, that's
10:38
going to enable me to build a better MCP.
10:40
Server, yeah and it's also opt in like if you're you could ask
10:43
like you can flip a switch in MCP cat and say also ask my
10:48
users if they're cool with it. And so it's not doesn't even
10:51
have to be by default enabled for your MCP server.
10:54
You can sort of let users decide if they are giving you user like
11:00
usage of data like what you do in traditional software.
11:03
And you're also tracking where it's coming from.
11:05
And I say this because one of the big use cases I heard
11:08
yesterday is internal MCP servers in large companies.
11:14
And so if I am building an MCP server and I have no visibility
11:19
into it, I don't know who's using it in the company, what
11:23
they're using it for. But then if I do know that, hey,
11:26
it's coming from legal and they're using it for this
11:29
reason, is that like something you can do to like get that
11:34
traceability as to where in the? Yeah, totally.
11:38
So you can have, what you're talking about is sort of
11:42
identifying the session of a user and who they are and where
11:45
they're coming from, right? Like we will get you information
11:48
that it is, you know, Kevin Smith, who belongs at this
11:53
company and is part of this team and this organization and
11:58
they're using cloud code, right? All of this stuff.
12:01
So, you know, when an issue happens, you might actually be
12:05
able to tell through our products that, oh, this is
12:08
affecting Cursor users only because Cursor interprets the
12:12
protocol differently than other clients do.
12:14
And so otherwise you, you would have no idea what like, oh, this
12:17
guy says there's a problem, I don't know why.
12:19
Oh, he's using Cursor? What version?
12:21
Is my machine, dude. Yeah, we're some my machine.
12:25
Yeah, exactly. We're some my machine.
12:27
I don't know. So you can get that in depth
12:32
analytics and like telemetry. Yeah, yeah.
12:35
I mean, I think you need it to, you need it to run a mature MCP
12:39
server at the end of the day. Like, I mean, it's, this is like
12:43
par for the course for web analytics.
12:45
You need these answers in order to know how to fix problems
12:50
before your customer even gets angry, right?
12:53
And for example, at MCP Cat, we have tons of analytics where I
12:57
get paged if the page loads slowly and I'll message our
13:01
customer like, hey, sorry about the like delay, like the slow
13:04
that slow downs, we're on it. And one of our customers, she
13:09
was like, oh, I, I was like, I was going to message you and you
13:14
miss messaged me before I even could message you because.
13:17
You can then set up events so if X happens then we do Y and
13:22
doesn't. It can be very deterministic,
13:24
like if we're seeing a spike in latency, tell the agent to reply
13:30
or tell or tell the end user and message them so.
13:35
Yes, yeah, you can even, Yeah, if a tool call is slow, the the
13:39
response could say, hey, sorry for the slow response, apologize
13:44
to the user on our behalf if you are cool with that.
13:47
And the LM will say, hey, this this company's MCP is slow Today
13:52
they wanted to apologize. Wild man.
13:55
I mean, you don't have to do that, but it's possible.
13:57
It is possible. Yeah, you can do a lot of stuff
13:59
and it is true, like for a mature product you need that
14:04
kind of analytics. I don't know anybody who's not
14:07
running something like Post Hog in their products or a variation
14:12
of Post Hog. And yes, Post Hog is very
14:14
startupy stuff and Post Hog and all these like launched darkly.
14:18
You have a lot of ways to update features.
14:23
Do you within MCP Cat also have that route where it's like, hey,
14:26
we're going to AB test some stuff.
14:29
We're going to do like a little bit of a roll out feature flags
14:31
type of idea. Yeah, very interesting.
14:33
We don't currently do that. Like I think, you know, feature
14:38
management is probably an interesting space, but I don't
14:42
think MCP servers need that yet. I don't see very many people
14:47
wanting to AB test their MCP. And so, you know, I think we are
14:53
already quite early to the analytics game.
14:56
And so I think like just sharing like what users are up to and
15:03
when they're running into problems is the core focus of
15:06
what we're doing. And maybe one day we'll do like
15:10
feature management, but I think there's also like a ton of
15:14
solutions out there that are not unique to MCP that might do it
15:17
bad for it. So I mean I think like we solve
15:20
problems unique to MCP and probably stay.
15:24
Back to that focus idea we were talking about just a few minutes
15:27
ago. Yeah.
15:28
I mean, I think what's really interesting is what we see makes
15:34
MCP servers better. Like has nothing to do with our
15:36
product. It's it's purely just like it
15:39
will flag when issues are happening and the solutions them
15:42
are quite creative. For example, one of the biggest
15:47
gaps I see in MCP servers is many people do not use errors on
15:54
tool calls as a way to guide an agent into the right way of
15:59
doing things. So like, you might hit a web
16:01
page and it errors and says at 500, OK, sure, you're a human,
16:05
there's nothing you can do to fix that.
16:08
But what's nicer for agent experiences is when they mess
16:13
up, the tool call tells them, hey, you called this error,
16:19
Maybe you were trying to call this or this or this instead, go
16:22
do that. And that makes agents a lot more
16:25
recoverable when they make mistakes.
16:28
And so there's tons of different techniques that we've seen
16:30
customers use to actually improve the success rates of
16:36
their MCP server. And so this is just when you
16:38
have errors, you're guiding them in different directions and
16:42
these are hard coded or this is something that you're telling an
16:46
agent to make sure when you're creating, if you know there's an
16:52
error, you give three other options.
16:55
I think it's more just a opportunity.
16:57
There's no, I, I don't have like whether it's hard coded or not,
17:01
right? I think that's kind of up to how
17:03
much time you guys want to spend on it, right?
17:05
But the fact of the matter is that errors propose a unique
17:09
opportunity to agents to kind of tell them what they're doing
17:13
wrong with your MCP server and of course, correct.
17:17
And if you're not doing that, if you're just saying an error
17:19
happened, sorry, you know. You're good luck.
17:21
Good luck. Try again.
17:22
Try again. It's not constructive.
17:25
The agent doesn't know what to do, so why wouldn't you use that
17:27
as an opportunity to to guide them?
17:29
It is so wild because as you're saying this, this is like the
17:32
first time that I can think of where you get that ability to
17:37
like reroute in flight and you get the the knowledge and that
17:42
it the intelligence that the agent has to understand the
17:45
error codes and then redirect its intention.
17:50
Yes, exactly. And actually, not that many
17:52
companies do this just because they haven't thought about it
17:55
or, you know, previously Error codes are something you might
17:59
like, not want to show your user.
18:01
Like what if it's, you know, too cryptic, you know, or like they
18:05
don't know what the like the parameters.
18:07
Like they don't need to know the parameters of the API that
18:09
detailed like you mean you want to hide it.
18:11
But actually with agents, you, you want to be clear and
18:14
commutative with them, kind of the same way we are with humans.
18:17
Like if you filled out a form to purchase a flight and you forgot
18:21
your last name, you'd see, hey, your last name is required.
18:24
That's a better thing to tell a human.
18:27
So why wouldn't you tell an agent like, Hey you got the
18:31
parameters wrong, you didn't just get them wrong, you forgot
18:34
the last name, so fill that it like retry your request with
18:38
last name like as an example of a proper way.
18:43
Easy. Fix.
18:44
But if you don't think about it, this is one of those ones where
18:47
it's like no or not that much overhead is needed to put this
18:52
into practice and you'll probably see such huge gains.
18:56
From yes, yes, and and we've seen customers have huge gains
18:59
from that. Yeah, for sure.
19:01
All right. What other tricks you got
19:03
successful in CP servers? So this this is probably unique
19:08
to, you know, different use cases.
19:11
We have a logging provider as a customer and so and MCP could
19:17
query the logs, right? But those logs could be like, if
19:23
you, if it's like a broad query, you'll get thousands and
19:27
thousands of results. And you don't want that to send
19:30
to the MCP or you don't want that to go into the context
19:34
window. Like that would fill out the
19:35
context window and it's not useful.
19:37
Something that we've seen some MCP servers do is they'll have
19:41
as a parameter to tool calls, token limits.
19:44
So when they're making the the tool call, the LLM can say I
19:49
only want you to return 100 tokens, so don't send me more
19:53
than 100 tokens because I don't need that many.
19:55
Does this make sense? Yeah, yeah.
19:57
Yeah, OK, so I'm pinging an MCP server and then what I get back
20:02
is very within. A certain bounds that I define.
20:08
So it's like it can't be more than you you set the parameters,
20:12
it can't be more than X tokens. It can't.
20:16
I'm sure you can set a lot of other parameters too, but is the
20:19
token length the one that gives you the most bang for the buck?
20:22
It's it's unique, right? Because normally you think of
20:24
your, you think of your tool calls mapping 1 to 1 to API
20:28
parameters, right? You're thinking like, OK, I want
20:31
you to search for my users in this country.
20:35
The agent might not know that you have millions of users, and
20:38
so when it does that query, it might get millions of results.
20:43
And it might try and give you back all of those.
20:45
Results and maybe some some AP is have limits that you can set
20:50
on results. But sort of what's what's nice
20:55
is to give the agent the agent has more context on how many
20:57
results it wants or expects to get its problem solved.
21:01
So having a token limit on every tool call that could be lengthy
21:07
is a really nice way to let the agent decide how many tokens
21:10
it's filling into the context window with a tool.
21:12
And so when the agent comes and submits whatever his request is,
21:16
it is always saying. That's one of the parameters
21:20
that it's throwing in. Yes, exactly.
21:22
Exactly for for lengthy tool calls.
21:24
Yeah. So, yeah, it doesn't need to be
21:27
every time, but when you know that there are certain tool
21:30
calls that can get a little bit. Yeah.
21:32
Unruly, you say? All right, cool.
21:35
Make sure to submit how many you want as one of these parameters.
21:38
Yeah, exactly. In that form that you make them
21:40
fill out before they are hitting it, that's that is super cool
21:47
and it saves a lot of money I'm sure.
21:48
I, I think it saves a lot of money and, and you know, people
21:53
talk about MC PS I mean, the goal is to save money and be
21:56
more reliable and more efficient, right?
21:58
And so this is just one way that agents can sort of not get
22:02
surprised with massive context window.
22:05
Yeah, because then it it ends up and 1st it fills it up and then
22:08
it ends up not being useful. The other fun one that I have
22:11
for you is something that we do automatically, but any, any
22:15
anybody with an MCP server can do right?
22:18
But we'll do it automatically for for customers is you add an
22:24
extra tool to your tool list that says tell me what we're
22:29
missing to get your job done. OK, so you know, let's say I is
22:35
there an MCP server you use much like that you like to use?
22:40
Of course, play right. OK, play right?
22:42
Yeah. So let's say you asked an agent
22:47
to do something with Playwright and that might be involving a
22:53
Chrome extension. Like you wanted to also click a
22:55
Chrome extension, right? But Playwright's tools don't let
22:58
you access Chrome extensions. Playwright could expose a tool
23:03
on their tool list that says if you don't have what you need,
23:07
report it. So report missing function.
23:09
It's like give us feedback. Yeah, yeah.
23:11
So instead your agent will report to Playwright.
23:14
Hey, the user wants to access Chrome extensions, but I can't
23:18
do that with the current available set of tools.
23:21
And that gets sent straight to the developer team of playwright
23:23
and they can say, oh, OK, well, this has been asked, you know,
23:28
1000 times. Maybe we should do it by now,
23:31
right? And then they'll they'll know
23:33
that you're trying to do something that can't be done and
23:35
they never even had to talk to you.
23:37
Or it just automatically creates an issue and starts trying to
23:40
create that right away? Slave, if they have that.
23:45
The developer gets the PR, they don't get the feature.
23:48
Request pad is wild. Yeah, couldn't agree more.
23:51
That is so logical and it makes so much sense.
23:56
And again, it's something that if I didn't think about it, I
24:00
probably wouldn't do it. But now that I'm thinking about
24:03
it, it's just adding an extra tool.
24:05
Yeah, just one extra tool. And you get so much insight.
24:08
Yeah. I mean, I, I, it's, it's
24:10
business critical. And so when you see these
24:12
discussions about, oh, MCPS suck, well, candidly, they've
24:16
been out a year and no one has put in tons of effort into
24:22
making them better, right? And they are now and they're
24:25
getting better and they're, and some of them are great.
24:27
Like I, I love some of them so. Have you seen like good
24:33
practices on how to build tools and how to properly like
24:38
abstract the tool calls? Because the classic gripe use
24:44
you hear is, well, if you just map it one to one with your API,
24:48
then it's a horrible experience and you've got a million tools
24:51
and you load up all those tools into the context window and it
24:53
blows everything up. And so then I've heard, well,
24:55
you want to abstract it away and maybe you just have three tools
24:59
that you can go to 1st and that will bucket it and you then you
25:03
can tree search down on the different tools or it's a clear
25:07
workflow. You know the intention, so you
25:10
don't need to have each minuscule detailed tool if you
25:16
know the intention. Yeah, yeah, I so I think that
25:22
the discussion when when people are talking about mapping tool
25:27
calls 1 to 1 to AP is and, and extracting them out into like
25:30
more usable things. Like I think there's obvious
25:32
gains, right? Like you could turn instead of
25:37
making multiple tool calls for the same thing.
25:40
Yeah, you're saving round trips If you're able to combine three
25:42
tool calls into like what they actually do together, that's
25:45
like obvious. But I think the hard thing is
25:47
giving people recommendations for well, like, what is that?
25:51
Like how do I know what those are?
25:53
And what I like to tell people is base it off your UI.
25:57
Like do you have AUI today? Like your UI is probably using
26:01
20 API calls to solve a human need, right?
26:05
And so that can be a good place of inspiration for where you
26:11
would normally combine tool calls you can actually expose
26:15
like. Oh, well, this page of our
26:17
product is like a review page and it uses like 10 different
26:23
API calls, but we can just tell the agent, oh, you want like us
26:26
to do that for you? Like here's like we'll do what
26:29
like just expose one tool call called review and it will like
26:33
do that. And it does everything in behind
26:36
the scenes that, yeah, it's like kind of taking inspiration from
26:40
your UI because you've done a lot of thinking on, on what the
26:42
human wants, and he just kind of want the same thing sometimes.
26:46
Yeah. You don't have to overcomplicate
26:47
it. Yeah.
26:48
Yeah. I'm thinking about how you
26:50
probably have a lot of extra stuff in your UI that you need
26:56
to do because a human is doing it, but you potentially don't
26:59
need as much, but you still know the outcome.
27:02
Yeah. So at the end of the day, you
27:03
know the outcome. And if you know that outcome,
27:07
that can be the tool that you're exposing.
27:09
And then along the way, if you need to get information from the
27:12
human or the agent, then you get it.
27:14
And that is part of that form or those those pieces of extra
27:18
information that you want to get from the agent.
27:21
Exactly. Yeah, I think like the UI
27:25
example probably has some things that aren't even relevant to an
27:29
agent. So you don't like, you know, you
27:30
would cut those out. But there are hidden gems in
27:33
there of like core, like use cases that you could put into
27:38
one nice tool call and maybe combine like 3 or 4.
27:42
And then your agent has less tools because once you hit over
27:45
30, we see like most agents start to crumble in terms of
27:49
like success rates. And so you kind of want to have
27:53
less than 30. And if you have tons of API
27:55
calls, you can condense them down into like more appropriate
27:58
use cases. And if the use cases are not
28:01
what the agent wants, you can always expand your tool list
28:05
because the use cases aren't what what it wants.
28:08
Have you seen much success with just having like 2 tools which
28:12
is like search tools and then execute?
28:15
Yeah, yeah, yeah. Cloudflare does this right.
28:18
Yeah, yeah, yeah. We don't have any.
28:22
I think like we've only seen Cloudfare do it.
28:24
We don't see none of our customers have done that.
28:29
I think it's, it's interesting, I think it could work, but I
28:35
think that it was a solution. It's like changing your product
28:39
for, you know, a solution that needs to happen more on the
28:43
downstream. And so I think it's like less
28:47
necessary now because Claude and some more intelligent agents are
28:53
doing searching on the tool calls already.
28:56
And so like before then I thought maybe like, OK, if you
28:59
have tons of tools, sure, that makes sense.
29:01
Like Cloudflare has so many different services.
29:04
I I'm sure like that they need it that way.
29:07
But now Claude can actually search across tools for what it
29:11
needs, and I don't think you really need those as much
29:15
anymore. What about going back to this
29:17
idea of a goal? Yeah, yeah.
29:20
Have you thought about how granular you have MCP servers?
29:26
Like do you have one overarching MCP server or do you break it
29:30
up? And you say, well I know a clear
29:33
set of goals and so I'm going to have one MCP server that's for
29:36
those goals. And maybe that means I make 5
29:39
different servers. Yeah.
29:40
I mean, I think if you're a large organization, you need
29:43
separate servers just because developers are deploying and you
29:48
don't want like their deployments to break other
29:50
services, deployments and maybe certain teams move at a
29:54
different pace. And so I think you'll always
29:56
want multiple MCP servers if you have like multiple products.
30:00
And that way you can like enable or disable them, like the user
30:04
can choose to like disable or enable them.
30:08
I think that Cloudflare's solution is very interesting in
30:12
that they probably have tons of MCP servers that they're like
30:16
routing the execute calls to. So I'm pretty sure that what
30:20
Cloudflare is doing is it's actually an agent behind the
30:23
search and execute and it's using, it has like, it has this
30:27
like very Cloudflare specific knowledge in the hood.
30:30
I think it's very cool. Yeah.
30:33
Yeah. That's something that like as
30:36
you said, the goals idea earlier, it made me think about.
30:39
OK, Well, how granular do we want our goals to be?
30:43
And I didn't even think about how you want teams to own
30:46
certain servers and you want to have that type of flexibility
30:51
within the servers. Yeah.
30:53
And the deployments and all that good stuff that micro services
30:56
bring to the table. So it's.
31:00
Well, yeah, good things. I I mean, I think like yes, like
31:05
micro services will like are definitely enabled that will
31:08
enable large, large companies. I don't know that like a smaller
31:11
team would want them per SE, but yeah.
31:13
Yeah. And and maybe you're like you're
31:15
saying like your product is not so big like Cloudflare where you
31:21
need 50 million different tools. I will say what's really
31:27
interesting is last year we saw it's like kind of a shift in
31:33
organizations. We saw companies, their MCP
31:37
servers were all one team and like companies had like an AI
31:41
team to do the cool new stuff that is coming out.
31:46
And so you'd see that 01 team of five built the MCP servers for
31:52
like all of Cloudflare or all of these companies, right?
31:56
And then what ended up happening is their MCPS took off and they
32:01
realize, oh, wait, OK. Like organizationally, this MCP
32:05
server is an interface for this product team's product, but the
32:11
AI team made it and is managing it.
32:13
And so now we've seen MCP servers shift in responsibility
32:17
away from AI dedicated teams to product dedicated teams that now
32:22
just need to think about AI for their product as a like first
32:27
principal. And so now we've seen the
32:29
ownership of MCP servers shift internally into the dedicated
32:33
team. Yeah, it's getting closer to the
32:35
source. Yeah, yeah.
32:37
And so we'll talk to some teams now and we'll say like, hey,
32:39
like, are you guys managing this MCP?
32:41
And they'll be like, Oh no, that's actually those guys.
32:44
They manage that MCP. And that's where you get more
32:46
granularity and it breaks up into different.
32:49
Yeah. And so now what we see in
32:51
organizations is like there's a platform developer experience
32:54
team that's in charge of making sure all the MCP servers kind of
32:58
use the same properties. What product teams are owning
33:02
their own MCP servers as almost like they they're like UI for
33:06
agents. And I imagine data teams have
33:09
their MCP servers to sync in with the databases or whatever,
33:13
so you let the teams who know the most about the product that
33:20
they're exposing to the agents be the owner of the MCP server.
33:24
But I do like the you have a platform team in a way to make
33:29
sure that things are a little more standardized.
33:31
Yes, exactly right. Like they're all TypeScript MCP
33:34
servers, and they all use this form of authentication for the
33:38
user and like, 'cause there are things that you want to
33:40
standardize the developer like that or like the user experience
33:43
on while enabling product teams to handle their core business
33:47
logic. For sure, yeah.
33:48
Wow OK, makes complete sense. What?
33:52
What else is there to chat about?
33:54
Yeah. I think something that I think
33:59
we came to this conference for is mainly there are protocol
34:08
level discussions happening that I think affect user analytics
34:16
pretty significantly. And So what I mean by that is
34:21
the protocol is developing ways to make running MCP servers
34:27
easier for developers, but there's consequences to
34:32
understanding a user's journey. And So what I mean is like, as a
34:38
company, you want to know that the user came from Claude code
34:42
and asked these like 10 different like, like tried these
34:47
different cool calls in that session.
34:49
But what does that mean? And how do you know that that's
34:52
a user session when they have 5 Claude codes and all five of
34:56
them have hit your MCP server and they're asking questions to
35:01
your tool to your MCP server? With different goals.
35:04
With different goals and then maybe even one of them spawns
35:07
sub agents, right and and so I think that there are a lot of
35:13
protocol conversations happening to understand that on like a
35:16
network level or like a authentication level, like how
35:20
do you know they're all authenticated the right way?
35:22
But we wanted to see how much of the protocol is actually
35:25
thinking about this usability and traceability for knowing
35:31
that the experience is good. And so there is conversations
35:35
happening actually like right now where they're like talking
35:38
about MCP version 2 for the SD KS.
35:41
I think like someone from anthropics giving that talk and
35:48
it has real implications for how that's done on a protocol level.
35:51
And I'm I'm curious to see what what they come up with.
35:54
They're not thinking about it as the product analytics point of
35:59
view. Yeah, I I think rightfully so.
36:03
They're thinking about it more from a networking and security
36:07
standpoint. Very deep technical.
36:09
How can we make this the best technical experience?
36:13
Yes. And like infrastructure
36:15
experience, etcetera. And that's totally where we
36:18
should that the protocol should be focused on.
36:21
I just think that there are repercussions to product
36:24
analytics that we just want to make sure that whatever they're
36:26
solving also makes session tracking easier and and.
36:32
But are you afraid that it's going to totally forget about
36:36
now? I don't think so, because I
36:38
think even for a good Cloud Code experience, it needs to know
36:45
that the ways it's using the MCP server are consistent with the
36:51
session itself. And so I don't think it will
36:53
like be completely discarded. But I do think it's useful for
36:57
companies to know that AMCP tool call came after these tool calls
37:03
and was spawned by a sub agent with different goals.
37:07
And yeah, I think like if you want to make a good experience,
37:11
you need to know the experiences structure.
37:16
Because right now you can't tell if it was spawned by a sub agent
37:19
to then execute. Yeah, right now you cannot tell
37:22
that it was spawned by a sub agent.
37:24
You can tell that it's a different like goal and purpose
37:28
and session from the same user from the same user, but you
37:32
can't tell that it's. A But you know, no, that it was
37:33
like, all right, we hit this MCP server, we got this information,
37:36
now we realized we need more and so we spawned sub agents and
37:41
then we hit it again. Yeah, to to a company even using
37:46
MCP Cat, right, it would just look like that's a new goal and
37:51
new tool calls were made for the goal, which is not like
37:54
necessarily wrong, right, But they tie into overarching.
37:57
Goals of of, Yeah, what that intention is, yeah.
38:01
And also the same. So tracking intention becomes
38:04
more complicated if these network primitives change a lot.
38:08
Or or or. Don't like really think about
38:11
user analytics at the entire level, Yeah.
38:13
OK, yeah, man, that is so wild. That is so.
38:17
It is so cool that you're working on this and like, it is
38:21
very novel in that I've seen a lot of MCP gateways, haven't
38:26
seen MCP analytics. I'm glad to be a breath of fresh
38:31
air. I mean, like I think the
38:32
gateways are solving a good problem, but I, I would say I
38:36
don't know anyone else doing what we're doing so.
38:38
It feels like people would resonate with it pretty easy,
38:41
like if they have more than three MCP servers set up, Have
38:46
you found that it's, it's quite easy for people to understand it
38:48
because you're effectively creating a new, like a whole
38:53
new, what do they call it? Like the blue ocean, You've got
38:56
the blue ocean strategy. You're you're creating a new
38:59
product category. Yeah, I mean, that's a very nice
39:04
visionary way to put it. I got the, got the hair.
39:10
We just do hard things that nobody else is willing to do.
39:13
I. Mean it's a very like I think
39:17
that it depends on the organization, right?
39:20
If they are, if they are viewing MCP as a core strategy, it
39:25
clicks. If they are not yet and they're
39:27
still experimenting, it doesn't click like they are like, I
39:31
don't know, getting our users, our users emailing us that it's
39:35
broken is fine right now, right? But it changed.
39:38
It's like it's a timeline thing for most people we talked to.
39:41
It's it's like, yeah, we need this.
39:43
It's just when we we need it. Is it easier to talk with
39:48
enterprises? Because like we were saying, you
39:51
know, they have a lot of these internal MCP servers.
39:54
Oh yeah, I think internal MCP servers are, are are like less
39:57
of a value add for us. Like like, well, they already
40:00
have like they could talk to their employees more readily on,
40:03
on what's wrong, what's broken? I mean, we're better for like
40:08
like if you have a customer facing MCP server, I mean,
40:10
that's like a like you can't let that experience be bad, right?
40:14
That's like. You know, one day AAI native
40:18
product could come around that has an incredible MCP experience
40:22
and an incredible agent experience and your sucks.
40:25
So you need you, you need the data to make it better.
40:30
And so, yeah, I mean, bigger companies for sure are are
40:33
better fits for for our products.
40:36
But you know anyone that it's a core part of their strategy.
40:39
It's. It makes sense.
40:40
It feels like you are very orthogonal to the whole movement
40:47
of agent experience that I think is gaining a lot of popularity
40:51
and. Yeah, it is a thing.
40:52
Yeah, a lot of us want to know like how can I use a product
40:56
that my agent is going to be able to grok really easy?
41:00
Yeah, yeah, It's so interesting because it it's kind of like now
41:05
when I'm picking a vendor, I kind of want to know how good it
41:09
is with Claude, right? And there's no, there's no way
41:13
for me to find that out. Like there's no registry where I
41:16
could see like oh this MCP. Actually, yeah, there is.
41:19
I found one. Oh, really?
41:21
I think it's 2027 dot dev. They give a talk at one of our
41:26
virtual events. It's a number dot dev.
41:28
Yeah, yeah, 2027 dot dev and they have a whole, they did a
41:35
bunch of research on agent usability products and so
41:40
there's all these different products that are like the best
41:44
ones, you know, an agent can use it within the 1st 4 seconds.
41:49
The worst ones, it takes like 10 minutes.
41:51
Oh, that's cool. So and they have the whole
41:53
spectrum and how different products are are shaping up
41:57
against. It wow.
41:58
Very cool. Yeah, yeah.
41:59
So, but I I understand what you're saying, because if your
42:01
product is not open source, it's very hard for you to just tell
42:05
Claude to go and test it out. Yeah, completely.
42:07
I mean, like I need to sign up for an enterprise account and to
42:11
get access to your MCP server. I mean, I don't know that's,
42:16
that's bullshit. I don't want to do that.
42:18
Especially if you're testing out like the agent usability of it.
42:23
Yeah, exactly. Like I I haven't even decided.
42:25
I it's, it does a good job, right?
42:28
And so, yeah, I think it's, it's very interesting.
42:31
I'll check out this, this registry.
42:33
I mean, you like it, you know, I don't know if we need another
42:35
registry, but. Well, it's not, it's not a
42:37
registry, it's just research that.
42:39
Was oh, it's just. Research, so it's not like, oh,
42:41
hey, here's the registry used it use us and then yeah, yeah, it's
42:45
just, hey, we tested all these with these coding agents.
42:48
Here's what we saw and they did it with different coding agents.
42:50
So it's not like bias and they're they gave a whole
42:54
report. On it I see.
42:55
Well, what I was thinking is, is something that is a bit more are
42:59
like, I mean, it's something we can do honestly, but it's a like
43:05
a review system for agents and these MTP servers where the
43:08
agents report to the MCP like the like the agents report to a
43:12
reviewing service like hey, like this server experience with all
43:16
the tools I use and all the schools I use were like X and I.
43:20
Mediocre at best. Yeah.
43:23
Imagine you have the like comments on there too.
43:25
It's like Yelp, except yeah, it's like the mole book of the
43:30
Yelp mole book. And now we look at like Yelp for
43:34
what services we want to use because agents are willing to
43:37
actually take the time and probably review all of them.
43:40
I think it's just like, what's the benefit to the user?
43:42
I'm not sure. Well, if it's a more reliable
43:45
way of using a tool, like if an agent comes to me and says, hey,
43:49
we can use railway or Versailles, but you know,
43:53
railway is really cool. It does all these things.
43:55
And my confidence score that I'm going to be able to properly
44:00
execute is X through. So my confidence score that I'm
44:04
going to be able to properly execute is Y because I read on,
44:08
you know, the MO book Yelp. I do wonder if this is something
44:12
that Cursor or Claude would do for their own tools.
44:15
Well, not no, not necessarily like.
44:17
So like, let's say I'm Cursor as a company, and Cursor can have
44:22
its Cursor agents do public reviews of MCPS if their users
44:28
like adopt them. And then, you know, oh, for
44:31
Cursor, these MCPS perform the best.
44:34
And so like, you know, that makes me like, that's something
44:38
Cursor could do, right? And then I know which ones are
44:40
best with Cursor. Maybe Anthropic also has its own
44:42
reviewing system. Yeah, and it almost like biases
44:45
towards using those tools because it knows it uses them
44:47
the best. Yeah, Yeah, exactly.
44:49
I saw. Did you see?
44:50
I thought it was a super novel, but I don't think they've done
44:52
anything with it. Because a lot of ways that I
44:55
find out about new tools is I get suggested in my coding
44:58
sessions like hey I want to do XYZ and it says well you can do
45:01
that with these different tools and AMP I was having ads be put
45:09
in so you could pay so that you would be suggested in those
45:13
moments. I hate that, but I, I mean like,
45:16
I get it. I, I, I think it's like a fair
45:18
thing. I think it's just.
45:20
I don't think it went anywhere. Yeah, I'm sure.
45:22
Yeah, yeah, yeah. I think like I want to make sure
45:27
that the recommendations are unbiased, right.
45:29
And so if there's like a maybe like central reviewing system
45:34
where like I as a user of a agent can like have AMCP server
45:39
that like after every session, like reviews every MCP server it
45:44
used for how good it was and that all goes somewhere and
45:47
everybody uses this and it all goes to the same place, then
45:50
it's like, it is like a Yelp for, for aging experiences.
45:54
Yeah, I imagine you would have it in like the Google sponsored,
45:59
these first two links are sponsored and they have this
46:03
review that you're talking about and then the next ones are are
46:06
internal. Yeah, SEO surfaced ones, Yeah.
46:09
That'll be cool. But that is man, the agent
46:13
usability on it. I I do like that and the whole
46:16
Asian experience to have an agent be able to go.
46:21
I mean, it, it goes back to what you're saying with the what are
46:24
we doing or what are we not doing that you want us to be
46:27
doing so that the experience can be better it.
46:30
Needs to be better, right? I think we have been as an
46:33
industry, like kind of lazy, like we have been relying on the
46:37
model labs, like we've been relying on Anthropic and open AI
46:41
to just make AI smarter when there's so many things we can be
46:44
doing to just make their jobs easier, right?
46:47
Like to make like the models job easier.
46:51
So I think that it's a natural like we've seen in the last like
46:57
6 years, like models come to here and no effort from like
47:02
existing software to move them there.
47:04
And so now I think we're shifting into a world where
47:07
obviously the models will improve, but we're also
47:09
improving interfaces for them to do a better and do better,
47:12
right? So I think MCP is like the
47:14
biggest SAP we've done to to get them there.