Latest / Agentic Conversations (formally mlops.community) / MCP Goes Stateless

Transcript

MCP Goes Stateless · Agentic Conversations (formally mlops.community)

0:00
Now we're in a state where people rely on the SDKS, where
0:02
the SDKS have hundreds of millions of downloads a year or
0:06
a month, and that gives us a different lever that we didn't
0:11
have 12 months ago. And you need to be aware of
0:13
these and then use them to your advantage and and be mindful
0:17
around the limitations. It does feel like a maturing
0:20
and. It does start this maturing,
0:21
yeah. Yeah, it's, it's something that
0:24
you're like, oh, very clear value prop.
0:28
There I need to do the minimal thing that I I know I can extend
0:31
so I can solve future problems additively rather than subtract
0:36
like with a subtraction or removing things.
0:38
What do you think people don't understand about tasks?
0:54
All right, probably a good place to start this one, dude.
0:57
And what I'm most stoked about is hearing the context around
1:00
the new release. What is it?
1:01
What was the inspiration behind it?
1:03
Yeah. And like the, I think the new
1:04
release is honestly probably one of the biggest release we have
1:08
ever done around MCP. It has a one big change to how
1:12
we are using MCP in in the sense that when we originally
1:18
developed MCP over the last year, the the protocol was
1:22
somewhat stateful. So you needed to keep you know a
1:25
session open between the server and the client.
1:28
And that did not work very well for some of their particularly
1:31
big hyper scalers that now use MCP pretty much everywhere.
1:34
That being the Microsoft of the world, the Google's of the
1:36
world, the anthropics of the world.
1:38
And so this release is really about moving away from the
1:43
statefulness of the protocol to a full day stateless protocol.
1:46
And we have worked really, really hard to make this work in
1:50
the, in the way MCP operates. And I've put a lot of work into
1:54
this and this probably the biggest change we've ever done
1:57
to the protocol. So I'm I'm super hyped about it.
1:59
So no more handshakes. What can you go a little bit
2:03
deeper into this inspiration? Because you probably saw this
2:06
coming up and then there was a point where you were like, we
2:09
need to figure out what's going on here.
2:11
Yeah, like the the original version of the protocol, right.
2:15
It was like all standard IO. It was like it was a very
2:18
natural thing that you have this notion of, you know, a program
2:21
that you can always interact with that's this MCP server.
2:25
And then as we went over trying to do this over HTTP, we tried
2:28
to retain this because we fundamentally believed that
2:32
agents are somewhat stateful. But it turned out operationally
2:36
over the year we learned that there are other methods how you
2:38
can still get that without having to rely on a stateful
2:42
protocol or a stateful transport.
2:45
And the cracks kind of showed, particularly when we saw again,
2:49
the really big hyperscaler tried to do this with like at like
2:54
massive scale, like thousands of 10s of thousands of servers
2:56
serving like millions of requests.
2:58
And then you're like, you're getting into really hard
3:00
difficulty problems of like, how do you scale this?
3:03
How do you, you know, move the state around?
3:05
There's a particular problem where, you know, you hit one
3:07
server and you kind of want to have a load balance to hit the
3:10
next server. But if you have some for state
3:12
in this one server, you kind of need to communicate this to the
3:14
other server. That's hard to do.
3:16
And you can do it and, but it's painful.
3:18
It's, it causes latencies and it causes CPU usage And it's just
3:23
like very unpleasant to serve. And so a lot of the hyper scaler
3:27
really wants stateless workloads, but that they can go
3:30
to 1 server, make a request, take the request, go to the next
3:34
server and do you know something else?
3:37
And everything you know can go through a load balancer,
3:40
everything can, you know, some of the things can be cached.
3:43
Just the standard things we kind of know how to do.
3:46
We kind of went there and we again overtime figured out what
3:50
are the pieces of MTP that needs to be stateful, what are the
3:52
pieces that don't need be stateful?
3:53
And we decided the transport part doesn't need to be.
3:56
And so it's, I think it's a very natural, you know, progression
4:01
and evolution of the protocol. Maybe you could have seen some
4:04
of that bit earlier, but I think now I think it's a great time
4:06
and we have worked really hard with again some of the the
4:09
industry giants honestly to like make this work for them and also
4:12
make it work for us. It does feel like a maturing
4:15
and. This is, yeah, it certainly is
4:16
maturing, Yeah. Yeah, it's, it's something that
4:18
you're like, oh, very clear value prop there.
4:23
Are there any trade-offs that you had to grapple with when you
4:26
were deciding this? Yeah, some of the hardest part
4:30
is that some things inside the protocol are somewhat inherently
4:34
stateful. So for example, if you send a
4:37
tool request and you want to do an elicitation, so you ask the
4:41
user for explicitly for input, that is kind of bound to that
4:45
tool call. In the original specification,
4:49
it was actually could at any time send this elicitation even
4:53
if there's no tool call. So we have to make constraints
4:55
of like now you need to do it within a true call, which is we
4:58
we did a survey of all guitar proposals we could find and
5:02
basically 99.9% use it exactly the way we are a tool call.
5:08
But it means that when you do this, you, you still need a way
5:13
for like there's some form of state being hold on a server
5:16
while this true call is ongoing and you asking for input from
5:18
the user. And so we needed to build a
5:21
little bit of a machinery on top of it of this like statefulness
5:25
approach as statelessness approach, which we call multi
5:28
round trip responses, which or requests which require you to
5:35
basically go there, do a tool call, get an hesitation back,
5:39
put all of this together to call plus solicitation result and
5:42
send it again. So you're basically appending to
5:46
the context and you're transporting the whole context
5:49
over the wire. And that just allows the like
5:53
another server can now send like response without having, you
5:58
know, while having the full context.
5:59
So there was some of that machinery we needed to build
6:02
into that is less that is a bit more unique to MCP because we
6:07
have this like there's inherent statefulness in some of the
6:10
semantics that we that are in the protocol.
6:13
Well, it does feel like these agents and the assumption at the
6:17
beginning of this needs to be stateful because agents
6:20
potentially have these unique characteristics and that got
6:24
disproven in a way, if I'm understanding you correctly.
6:28
Can you talk a little bit more about how that got disproven?
6:32
I I think that it's less about it got disproven as a as a
6:35
concept. I think it what we learned is
6:37
that it doesn't have to be on a transport level.
6:40
So it doesn't have to be on the HTTP channel that the state is
6:44
open. So for example, the that there
6:47
is state between a tool call and elicitation.
6:50
When you make a tool call and you ask the user for input,
6:52
there is some form of state. You know the tool call waits for
6:55
that elicitation to to be done and get this additional input.
6:59
So there's some form of state. But what we have now done is we
7:02
have taken all this state that is required and basically move
7:07
it over the wire over and over again.
7:09
So effectively making the the transport doesn't have to be
7:11
open all the time, but the data still encapsulates that kind of
7:17
state. And So what we have learned is
7:18
just to be more efficient about encapsulating and how we encode
7:22
the state. And we have learned areas where
7:25
we don't need that state. So for example, we have learned
7:28
that people always want to ask for user input during a tool
7:32
call and never outside of that. And that's some of the things
7:35
that we have learned and that learning over the years that has
7:38
led to this like new stateless mode that we call like multi
7:41
round trip responses, replies. And I think that's especially
7:45
the thing fundamentally, there's still statefulness in agents, I
7:48
think. I do like that you're starting
7:50
to see the patterns and you're starting to help MCP protocol
7:55
work together with the patterns, as opposed to all of us figure
8:00
out how MCP works and then create patterns around it.
8:03
Exactly. A second part of that learning,
8:06
for example, is and we have learned that when you do need
8:10
some form of session of like, oh, you will call, you know, you
8:14
call the tool to add something to a basket, only then you have
8:17
a new tool that has like a basket, you know, I don't know,
8:20
list my basket or whatever. We have learned that nowadays a
8:24
year and a half and part of that is the models just got better.
8:28
We is that you can just give the model us like an ID that it just
8:32
carries around between the tool calls.
8:35
And so it's like an implicit session.
8:37
You don't have to put it on the protocol level anymore because
8:39
the the model will just perfectly happily take a like an
8:44
ID that you give it and then move it into every other tool
8:47
call. So you can piece them together
8:49
that they belong to each other. And we have proven this was
8:53
basically all major models and all of them do this now.
8:55
Well, a year and a half ago, it would have been a very different
8:59
story. So part of it is also that the
9:01
models just get better. Yeah, and they're happy to carry
9:04
that ID. Yeah, they're happy to carry
9:06
that ID now, yeah. And that makes it much easier on
9:09
your side. I can see how that will help
9:11
with the the whole designing. You go, OK, well if this is true
9:14
then that makes our lives a little bit easier and you feel
9:17
more confident removing some of these pieces.
9:20
Yeah, exactly. And now paint me the picture of
9:24
someone who is currently using the MCP version and wants to go
9:31
to the new version. What does that look like?
9:33
Luckily it looks very, somewhat, very easy to you as long as you
9:37
use one of the SDKS. So we put a lot of effort at the
9:40
moment into changing the SDK such that the the lift is
9:45
somewhat minimal, at least minimal with the help of a
9:48
model. And they're the TypeScript SDK
9:51
and the Python SDK, for example, they come out with a version 2
9:54
that will support this. And the changes between version
9:57
1 and version 2 are very easily done by like an Opus model or
10:01
something like that. And so it's maybe a minor
10:03
update, but it's not a hard lift.
10:06
It's not a fundamental change. A lot of the hard pieces are
10:09
abstracted away by by the SDKS if you're not.
10:16
But if you're running your own SDK, the lift is significantly
10:18
higher. And I would highly say suggest
10:21
that you going to go and use some of the SDKS and all of
10:26
these SDKS currently kind of work, work for version 2.
10:29
I think the C# SDK also works towards the version 2.
10:32
Not sure about the Go SDK. I think they stick to a want to
10:34
still to that 10. But most of them again, have put
10:39
a lot of thought and effort into making sure the upgrade is very,
10:43
very minimal and basically one shot about with the model that
10:46
was like the base requirement that it's not a hard lift.
10:49
So it should be fairly easy. And then similarly, if you have
10:52
existing soft, if you have existing servers, they should be
10:57
able to, to, to continue to work, right?
11:00
The protocol has versions. There's a, there's a lot of
11:04
sessions and a lot of documentation and work in the
11:07
SDKS to make sure they fall gracefully back to the version
11:14
that your server is going to use.
11:15
So it should just continue to to use.
11:18
This can be in a way of forcing function to use those SDKS.
11:22
If you had been thinking about bringing over some ideas and
11:26
it's weekly, Yeah, just use it, make your life easier.
11:30
You don't need to impress anyone, but exactly the other
11:36
PSA is don't forget to pour everything to Rust before you
11:40
change it to the SDK. But apparently.
11:44
Always, always. Everything runs better after
11:47
it's been ported to Rust, apparently from the.
11:49
Research I but you just just give it to to Fable and it will
11:53
just do all of it for you. Exactly.
11:55
That's what I've heard, and it's so wild to think like, oh, take
11:58
it, port it to Rust, and then you can port it back to the same
12:01
language and it will perform better.
12:02
So yeah, maybe. I don't know, I have to try.
12:06
Yeah, yeah, there there's the rumor going around.
12:09
So we need some hard details and science backed proof on this.
12:14
But until I see that I'm still going to be perpetuating the
12:17
rumor. Anyway, there's a few things
12:21
that I want to go into that are more theoretical and less based
12:27
in the protocol. And that's like what you've
12:30
learned. You've been manning the ship and
12:32
there's a great maintainer team that is also working on this,
12:35
but there's a lot of precedence for open source projects over
12:40
the years. Are there things that you feel
12:43
like the MCP protocol can learn from all of our past open source
12:49
winnings, and this one comes specifically, I asked the
12:52
community for questions and we got some incredible questions.
12:56
This one's coming from a question from the community.
12:58
They're wondering like, what are you going to take from HTTP and
13:03
all these other protocols that we've seen in the wild?
13:06
I think there, I think there's always something to learn.
13:09
I think you can always take a look and sometimes you learn
13:13
things you want to repeat and sometimes you learn things you
13:16
do not want to repeat. Well, they don't apply to you.
13:19
It's less often that they're wrong.
13:21
It's mostly that they might not apply to you.
13:22
And so they're, I think there's a few things.
13:26
The, the first thing is on the standards, some of the big
13:30
challenges is the, the standards, most successful
13:34
standards work in a way that they take a long time to be
13:40
billed and they're mostly consensus driven.
13:42
That's like HTTP, some of that, a lot of the authorization work
13:48
things that come out of the ITF in general, for example, are
13:50
very consensus based. Consensus based mechanisms are
13:54
very, very slow. They they take a long time, but
14:00
at the same time they often, depending how you do it, they
14:04
result in usually solutions that are quite good.
14:08
They're not perfect, but they're quite good.
14:11
HTP 1.1, HP 10, interesting enough, did not work well. 1.1
14:18
is the one everyone's using, right?
14:20
And I think we keep forgetting that quickly. 2.0 is also quite
14:24
well done. But then even they already had
14:27
shortcut Cummings and now they're three, right?
14:29
And you can learn some of the carefulness and considerations
14:34
and being sometimes more on the conservative side and saying no
14:38
to things because you know that two things can happen.
14:42
First of all, you can change things and add things later
14:45
that's happened to, you know, HTTP 1.1.
14:48
I'm sure during the process of HTTP, two people have known that
14:52
they will be shortcomings that they are currently putting into
14:54
the protocol and yet they continue because it is good
14:59
enough and it is within the time frame that they have and they
15:03
know the version 3 periods later might, might change it.
15:06
And that's OK. Standards are not there to solve
15:09
everything. That's they're there to solve
15:11
most problems. And so I think there's a lot of
15:15
lessons to learn in that regard. There's a lot of lessons to
15:18
learn in like how you run a successful open source project
15:20
and like in a governance and like a general sense.
15:23
And there are many, many ways to do this.
15:25
You can run it in a BDFL style like like the Linux project.
15:29
You can run it in a steering committee style like the Python
15:32
community. And they all have their
15:33
drawbacks and benefits. And I think what we have learned
15:37
here is that I think for us like having a strong core to maintain
15:42
our foundation. So a set of like a more of a
15:45
steering committee type and plus the BDFL on top has worked for
15:50
the most part quite well. I think there's things that we
15:54
will change this year to make it even more democratic and allow
15:57
more as the project is maturing and the structures of maturing
16:02
and like being able to be a bit more, be even a bit more
16:07
democratic. So it's always like a matter of
16:10
like you look at these projects and like, okay, you know, an ITF
16:12
project chooses democracy over everything else and consensus.
16:16
And as a result, it's slow. A project like, I know, take
16:22
something like and like a like a famous new open source project
16:25
like Ghosty, like with a, with a clear one leading person was
16:28
Mitchell, right? He's very fast and in certain
16:31
ways because of just one person making a decision, right.
16:34
And then there are mixes like the Python projects and you
16:36
choose the thing for the stage you're in.
16:38
And I think that's the, there's a good lesson to be learnt
16:41
there. And so for us, the interesting
16:43
part is finding that middle ground of like, we want to be
16:46
fast. We want to be somewhat
16:48
democratic, but we will also want to move.
16:50
We also want to be considerate and be somewhat conservative,
16:54
but at the same time, you know, we also work in a, in a, in a
16:58
slightly different environment because we're working in
17:00
software where things can change quicker.
17:03
And for example, if you like pre creating like a hardware
17:06
standard or a telecommunications standard where someone puts the
17:09
standard on software onto a box, puts the box 200 kilometers into
17:13
the north of Sweden and it's going to sit there for the next
17:16
10 years, you have a very different set of problems than
17:18
if you build MCP, for example. I like that juxtaposition
17:24
because in the real world, yeah, the atoms take a whole different
17:28
route for the standards. And it also is cool to hear you
17:31
say like, you're choosing this path right now because it is
17:36
what you feel the most optimal route.
17:38
But in the future, once things stabilize even more than they
17:44
already are, you're open to taking on new ways and trying
17:47
those out and seeing like a new pair of pants.
17:49
Do they fit? Do they do they look good in
17:50
them? All right, cool.
17:52
I, I, I have a very clear picture in my mind of like where
17:56
this going to go in a year or two years and how this needs to
17:58
evolve on a governing perspective.
18:00
Because it's pretty clear that I think this model is, it's is a
18:04
is a temporary artefact in time for the current situation the
18:08
protocol is in. But that will not be the
18:11
situation of protocol as in a year from now and then.
18:14
It requires different governance.
18:16
And I did want to hit on something else that you
18:18
mentioned, which I have found to be true, especially back in the
18:23
ML OPS days, is the sooner you can get something out there and
18:27
get folks using it and testing it, the sooner you can realize
18:31
where you had gaps in your thinking.
18:34
And so it is like that speed that you're looking for is a
18:39
feature, not necessarily a bug in that regard.
18:42
That makes sense. I I agree with some constraints,
18:45
right? The constraint is like how
18:47
flexible is the environment. And again, I think the
18:50
environment in which MCP is at is exceptionally flexible
18:54
because people around AI do two things.
18:58
They first of all, assume an exceptionally high velocity of
19:01
change and they're using models that help them to retain that
19:06
high velocity. So some of the burdensome work
19:09
is less burden burdensome. And so those are things that are
19:14
cultural pieces that you need to take into account when you
19:17
design the, the speed that you're you're going at.
19:21
And this is, for example, not true for programming languages
19:25
per SE. In programming languages you
19:29
know, particularly older programming languages or certain
19:31
niche programming languages, you might have less adoption of AI
19:35
or you might have, you might not want to change your code
19:39
constantly because of changes and because you just want to get
19:45
stuff done. And so there's, there's
19:50
carefulness to be considered here.
19:52
And even we, I think, have limits, right?
19:54
Like when we did the protocol changes and now what we're
19:57
doing, the SDK changes in particular, we're paying a lot
20:00
of attention that the delta is small enough to be done because
20:05
we don't want to jump into the problems that let's say like a
20:10
Python 2 to Python 3. Change had or there's there's
20:15
other aspects or in IPV 4 or IPV 6, there's a bunch of these type
20:20
of things where they are very cumbersome and difficult to do
20:24
and you need to be careful that the delta is small enough that
20:28
that the net benefit outweighs the friction.
20:31
Are there breaking changes that you feel like may happen in the
20:35
future? I never know.
20:37
I think we are getting into a shape where it's a lot of the
20:40
things are additive. The the problem at the moment, I
20:43
think we're going to see now is that the complexity of the
20:46
protocol will increase. And by virtue of we're trying to
20:52
we're probably at like an 80% solution space like 8020 where
20:55
80% of the problems people have in that space of soft and 20%
20:59
are not. And if you, it turns out if you
21:01
push the boundary beyond these 80%, the last 20% are actually
21:04
like 80, like 80% more complexity added, right And so.
21:08
80 times it's. Disproportionately more
21:10
complexity, right? We're adding caching, we're
21:13
adding caching scopes, things that look very similar to things
21:15
that exist in HTTP, at which point you need to ask yourself,
21:19
OK, at which point you're just going to rely on HTTP, right?
21:22
And, and we have made some choices a year and a half ago
21:25
that make this a bit more tricky.
21:26
And so there are there are decisions to be made down the
21:30
road, but most importantly, it is it means added complexity.
21:33
When we originally designed the protocol, we were mostly
21:37
concerned and mostly obsessed with adoption because that's the
21:40
thing that makes or breaks the protocol.
21:42
And the key principle for the very first version of the first
21:45
six months of MCP was that it's very simple to implement to the
21:50
point where we constantly tested that if you just give it an SDK
21:55
or if you just give it even the specification, a model will
21:58
write you happily an SDK in that language.
22:01
I mean, that was true 6 done in the first six months.
22:04
I don't think that is necessarily true anymore because
22:07
the complexity has increased. But that's OK because now we're
22:11
in a different state. Now we're in a state where
22:13
people rely on the SDKS, where the SDKS have hundreds of
22:16
millions of downloads a year or a month, and that gives us a
22:21
different lever that we didn't have 12 months ago.
22:24
And you need to be aware of these and then use them to your
22:27
advantage and may and be mindful around the limitation.
22:30
Hundreds of millions of downloads a month.
22:32
There is unreal just as a stat that you kind of glossed over.
22:36
And that is a great segue into I think this philosophical battle
22:43
that I'm sure your going through day by day as you're looking at
22:47
what to add as an extension versus bring on as a core.
22:50
And one of the questions that the community asked me too to
22:53
ask you, well it comes from Manvika and she was asking where
22:59
do you draw the line between something that is core versus
23:02
extension, especially with the idea of simplicity in like that
23:06
you want to touch on? There's a few things I think.
23:10
I think it's a very good question.
23:12
It's an aspect that we very rarely explain, but I think like
23:17
there's some very thought, there's some thoughtful thinking
23:19
behind this. What the, the, the principle
23:23
that need to apply to core are it needs to be composable.
23:28
It needs to be so that for example, like a, a resource in
23:32
MCP is a fairly general like remote file system approach that
23:37
perfectly composes nicely with tools as an execution approach
23:42
and so on. Then that's one aspect of that.
23:45
The second aspect of that is it needs to solve like serve all
23:49
clients and all servers at the same time.
23:52
I give you an example. MCP apps is a great thing, but
23:57
it only works in surfaces that I have browser, a browser
24:01
rendering in it, which is very constrained, right?
24:03
Because it an MCP app renders HTML, right?
24:07
And so a terminal will never be able to and look at an MCP app
24:11
and that's OK, but it means that it's not part of core because
24:15
you're leaving out a large chunk of the MCP client clients out
24:19
there. And so there's these kind of
24:21
consideration that we are having that is, that is for what
24:25
belongs in the core. The second thing that belongs in
24:28
the core is concepts that we are very confident will stay very
24:34
stable. So one of the best example of
24:37
this is where we maybe made a, maybe not a mistake, but like we
24:41
were, it was a sequencing issue. It's MCP tasks, MTP tasks is
24:46
this idea of long running tasks for long running tool calls,
24:50
which allows you effectively to do agent to agent communication
24:53
over MCP, which I think is a massive unlock.
24:55
I think it's a huge thing that will be super good, super widely
24:59
used in the long run. And we'll have actually in the
25:03
long run a lot of client support.
25:04
But the first iteration of it had issues and it was part of
25:09
the as an experimental change, the core.
25:12
And it turned out we were much better separating it from the
25:15
core into its own extension where it can be developed at its
25:19
own pace independently from the release cycle of the core in
25:23
there. Like basically iterates fast
25:26
enough to the point where we are back being able to move it into
25:29
the core because then we have confidence that it works the way
25:32
it works. And the extension is already the
25:35
now MCP task extension is significantly different from the
25:41
original experimental core piece that we have now moved out.
25:46
And so we want to basically just like let it cook, like let it
25:49
cook in the extension space, let it let people iterate, let
25:52
people play around with it. And once we were confidence in
25:55
the design, we've got to put it back into the core because that
25:58
is one of the pieces that does fit nicely into the like,
26:03
compose like it composes nicely with the rest of their the
26:06
primitives. Do you feel like that's a nice
26:08
pattern? Let it be an extension, kind of
26:11
prove out its value, and then merge it into core.
26:16
Do you see that? Being potential core pieces, yes
26:19
there are pieces like MCP apps or skills of MCP that will
26:24
always stay extension. Yeah.
26:25
But then there's other pieces that potentially could be core,
26:28
but instead of doing it like you did with tasks, you can let it
26:34
stand on its own 2 legs 1st and then go.
26:38
That makes a ton of sense. So I want to go into a few more
26:43
of these questions that are coming through.
26:44
By far, the most asked question is about the progressive
26:49
disclosure. Can you talk to us a bit about
26:51
that? Yeah, progressive disclosure I
26:53
think is an interesting case. So what people, and I'm curious
26:56
what your take is, what people actually mean by that, because
26:58
that first and foremost is like, what do people, what do people
27:02
actually think about what, what does this mean?
27:05
Progressive disclosure. Most of the time I've seen it
27:08
being mentioned, it refers to this notion of making sure that
27:13
the model learns about specific of a tool only when it needs it.
27:18
Because otherwise if you put all the tools into into the context
27:25
window, you get context bloat very quickly.
27:28
And so it's, it's less that the people are worried about like
27:32
the one progressive disclosure. They just think it's one
27:34
solution to the problem of context bloat.
27:36
Exactly. I wholeheartedly agree with
27:39
that. Right.
27:40
And so context bloat is context bloat is a, is a, in my mind an
27:48
interesting problem because there's a separation in my mind,
27:52
what does the protocol do versus what does the client do.
27:56
The protocol transports information.
27:58
It transports information about the tools available.
28:01
The problem we have seen is that naively most client implementers
28:05
take whatever comes over the protocol and just slap it into
28:08
the context window. And that is not necessarily at
28:15
the right approach. What you should do is you take
28:18
the protocol as an informative piece of like these are the
28:22
tools available, but I then choose techniques on my side, on
28:26
the client side that are avoiding this context bloat and
28:31
the two mechanisms that we found exceptionally useful and
28:34
basically use everywhere Cloud code does this and basically all
28:39
anthropic products do this is using tool search.
28:42
So like instead of putting the tools into the context window
28:46
right away, we are, we are putting it into effectively an
28:51
index. And the model has trained, it's
28:55
been trained to like search for certain tools when it thinks it
28:59
probably needs a tool. And that works quite well.
29:04
And it basically reduces your token usage as you would expect,
29:08
right? It's the exact, it's basically
29:09
effectively some form of progressive disclosure or
29:12
discovery from the model side. The second part of that is
29:16
programmatic tool calling, which is like how you combine or
29:19
compose tools together. Normally the traditional old
29:23
school way is the model runs inference, it spits something
29:26
out, it tells you to recall a tool.
29:28
You write, you call the tool, you give back the true results
29:31
to the model. Now the model says, oh, thank
29:33
you for the input. I'll call this other tool.
29:36
And So what the model effectively doing is it's
29:38
composing tools together in a quite inefficient manner because
29:42
it doesn't bear inference. And what you can do instead is
29:44
you can have the model optimistically write a piece of
29:48
code that says, take this now a JavaScript block that says call
29:52
this tool, take the output, parse it into Jason, put this
29:55
over there into this other tool, and then give me the output,
29:57
right? And so you have two tool calls
29:59
put together in a script and you execute that.
30:01
And that saves tools as well. And these two things to combine
30:06
are very, very, very effective like tool search plus and
30:09
programmatic tool calling to basically get rid of tool to
30:13
bloat. And so that's a long winded way
30:17
to say that I think 80% of the problems are client implemented
30:21
problems that they're not doing a good enough job skills issue.
30:25
That being said, I think there is an argument that we're slowly
30:28
coming around where it might be useful in certain situations
30:31
where the server offers, where the server authors knows better
30:35
about the shape of the problem. So it might wanna offer some
30:39
form of search or some form of progressive discovery, our
30:43
disclosure. And so we're thinking about some
30:45
of the primitives, but fundamentally it's to some
30:49
degree a client problem. Very proud of myself because in
30:54
my head, that's kind of what I was thinking.
30:57
I especially on those two where it's like, yeah, well, tool
30:59
search, I think Cloudflare was the first to popularize that,
31:02
right? I can't remember who.
31:04
Exactly, they did something called code mode which is what
31:07
the other word for programmatic. We did it around the same time I
31:11
think Cloudflare did the popular blog post about it.
31:14
Yeah, I knew one of those. I remember seeing that and
31:17
thinking, yeah, that makes sense, right?
31:19
And especially because I had been hearing murmurs in the
31:22
community of folks saying, well, you want to just bring the
31:25
abstraction layer up So you don't give them 90 tools and say
31:29
here is get Slack ID, here is send Slack message.
31:33
It's like you want to just figure out what's the intent and
31:36
create that from the intent as opposed to a million tools.
31:41
And then you got to choose one by one.
31:43
You have one tool that can cascade down.
31:45
And the search is, it's kind of obvious because that in a way is
31:50
very much progressive disclosure, just a different way
31:54
of doing progressive disclosure, right?
31:56
And we have seen, without going into too much details, we have,
31:59
we have seen a tool search plus programmatic tool calling can be
32:04
as good if not even better than for example, CLIS.
32:07
Yeah, I believe it. I believe it 100%.
32:10
And you know, I think the reason that folks felt like they were
32:15
getting this bloat in the beginning was because there was
32:19
so many community built servers that just went up so fast and
32:24
you would go and try and use them, but it was, they were
32:28
built so poorly without any idea of how to do any of this.
32:32
And so again, it goes back to that maturity.
32:35
If I used an MCP server that was built poorly back in the day, of
32:40
course I'm going to be complaining.
32:41
And GitHub was the notorious one who did this the worst because
32:44
they just threw all the tools into the and it was.
32:47
So that was what you're saying? Yeah.
32:50
And then part of that is like the the classic is like take an
32:54
open API spec and like transport it like transform it one to one
32:57
to MCP. And that's that's fundamentally
33:00
an anti pattern because it just leads to poor and to to very
33:05
poorly performing MCP servers that also have the problem that
33:08
the tools look similar enough that the model gets very easily
33:10
very confused about which tool to use.
33:12
So it's a bit of a, it's a bit of a you 100% right.
33:16
Like they're it was amplified by poor servers.
33:19
It was mostly amplified by poor client, yeah.
33:22
Yeah, it got it. You just didn't realize it until
33:26
later that, oh, this is a skills issue.
33:28
This is not the protocol necessarily that's causing these
33:31
problems. It's just that folks who didn't
33:33
really go deep into the protocol and figure out how to build it
33:37
properly or leverage what is there like this search.
33:41
And part of it is also learning honestly, like everybody learns
33:44
on it, right? Like when MCP came out, like how
33:46
many tools would you where you're using in a session like
33:48
10, right? Six months later, 12 months
33:51
later, MCP led to pro to a proliferation in the ecosystem,
33:55
right? And so some of the solutions are
33:58
needed because of MCP being part of the amplifier of tools.
34:02
And so it's a bit of a chicken egg type of problems.
34:05
I'm not worried. I'm not.
34:06
I don't. I'm less worried about that.
34:08
I think the client author didn't do it right from the beginning.
34:11
Completely normal. We all learn, we all do
34:13
iterative work. I think what I'm what I want to
34:17
make sure is I think a lot of people don't realize that the
34:19
situation gets significantly better and most clients don't
34:21
have this issue. At least most major clients
34:24
don't have this issue anymore. Speaking of which, we talk a lot
34:26
about like this forward compatibility.
34:29
You know, in software you have backwards compatibility.
34:32
That's a thing, obviously, but I've heard people talk about
34:35
forward compatibility, especially as you're building
34:39
and knowing, all right, there's certain things that I can build
34:42
this right now or I can wait six months and the models probably
34:46
going to be able to do it on its own.
34:48
And the classic examples of this are back in the day when folks
34:52
would try and do all these hacky moves to make the context window
34:55
bigger. Or I know people in certain
35:00
e-commerce stores who built their own MCP because they
35:02
didn't have that. And it's like, wow, if you just
35:04
waited a few months, MCP came out, it became popular.
35:08
So as you're looking at ways that things could now be forward
35:14
compatible, or as you're thinking through like what's
35:16
going to get better, where do you plan on, OK, this can get
35:22
better. Maybe you can build it right
35:24
now, or maybe you can wait six months, and at the rate of
35:27
change, it wouldn't be a far stretch to say that this is
35:31
going to be done. It's such a good question.
35:34
I think there's a lot of these kind of scenarios where you need
35:36
to be so careful and some of them you will get right, some of
35:38
them you will get wrong, right. And like the session piece, we
35:41
kind of got wrong, I guess in retrospective, right, or there
35:44
was a temporary thing. But another example is like,
35:47
people often ask us to do something what we, what they
35:50
originally called primitive grouping, which is like ADC,
35:54
allow tools to be grouped, right?
35:57
And that's super sensible. It's a very good use case.
36:00
There's a lot of use cases around this.
36:02
But if you think forward, you're like, OK, first of all, there's
36:06
two things to this. First of all, the models would
36:07
get really good and people will not select tools as a group in a
36:12
session anymore. Six months ago, 12 months ago,
36:15
you have to select tools in, you know, UI.
36:19
Nobody does this anymore. Nobody expects us to do this
36:21
anymore. And so these things will just go
36:23
away to some degree, right. At the same time, the models
36:26
will get better to reason based on the tool description about
36:28
the grouping that's there in the 1st place that you don't have to
36:31
spell it out as a, as a human or that the group might not be
36:36
relevant to the the problem that the user that the user has.
36:39
And if the model might find the different grouping mechanism.
36:42
I mean, in the second part of that is like a group is, is
36:46
nothing but like a tree with a with a like with a depth of 1
36:49
effectively. So there's a generalization
36:51
there that you need to be very mindful when you design
36:54
protocols. You need to think about like,
36:55
let's just generalize. Does it solve a problem?
36:58
It's never is is often the case is yes, but the problem is like,
37:02
is it solving all my problems in the future?
37:04
And the answer has in some sense, no.
37:06
And I need to that's the the hard one.
37:08
And then you need to go back and like, no, let's wait, let's see
37:12
if the model gets better. Let's see if how the world looks
37:15
like in six months. Let's see how we're, you know,
37:19
if we find a more general mechanism, even though it takes
37:22
a three months more and it's painful for the people wait for
37:24
the solution in the meantime. And I understand that, but
37:27
that's the that's the kind of role that you will wear
37:29
forward-looking is super important.
37:32
And I think these things are super interesting, but there are
37:37
like things that are pretty clear that there are no horizon,
37:39
right? Like we know that the task, the
37:42
length of a task that a model does is increasing, right?
37:46
It's kind of like linear increasing.
37:47
And we know this and we have that.
37:49
That hasn't stopped really since the years this increase.
37:53
So we know that like more long running tasks will just
37:57
increasingly more important because we will increasingly
38:00
less interact with a model in our chat environment because the
38:04
model will like, because the task the model will do will be
38:06
more asynchronously how we interact with humans and they go
38:09
away and do something and come back.
38:11
And so in that scenario, you're like, OK, I will need something
38:13
like tasks, right? Like I will need something like
38:15
that. I need to design it minimal and
38:17
make sure it can be extended because I need to look in the
38:21
future that I cannot predict, right?
38:23
Nobody knows what's happening in six months from now in the AI
38:25
field. So I will not know it either.
38:29
So do all I know is I need to do the minimal thing that I know I
38:32
can extend so I can solve future problems additively rather than
38:38
subtract like with a subtraction or removing things?
38:41
What do you think people don't understand about tasks?
38:45
I think Task is a poor name. It's a poor name and a good
38:49
name. It's a good name because it's
38:50
accurate and what it's trying to do.
38:51
It's poor. Maybe it does agent to agent
38:54
communication effectively and I think there's some open
39:00
questions there around. Is this the right format where
39:03
there's some shapes to be experimenting around with it.
39:06
But I think people really don't understand that what it
39:09
effectively allows you to do is have an agent talk to another
39:13
agent that that's effectively what it's doing.
39:15
I think it's doing it quite effectively.
39:19
But again, in the long tradition of MCP, we are in poor at naming
39:25
things. So we we shall continue in our
39:29
old tradition of continuing to name things, probably.
39:32
Dude, you know, so I'm not going to comment on the naming of MCP
39:37
because that's your business and I will let you have your fun
39:40
with it. But I will say the folks who
39:43
name things the best by far in my experience are the damn
39:48
researchers. Follow me on this one.
39:50
Hallucinations with models, right?
39:53
And then you have, what else do you have?
39:55
Catastrophic forgetting of models.
39:59
Give me a better name. Maybe we need a bit more drama.
40:02
Exactly. Catastrophically forgotten that
40:06
10. Mug Yeah, yeah, maybe, maybe,
40:10
maybe should go in. Next time you have a big naming
40:14
conundrum on your hands, just go walk around the office and find
40:18
a few of the researchers and say, like, what would you call
40:21
this? I love it.
40:24
I love it. I I should do it.
40:25
That's a good idea. We are, we are certainly.
40:28
We are not. We are good at a lot of things.
40:29
Naming is not one of them. That's for.
40:31
Sure, but it is hard dude. It is so.
40:33
I It's hard. Vibe with you 100% on that
40:35
because you're like yeah, that sounds good.
40:37
And then somebody comes up and they're like well does that mean
40:38
this? And you're like, that is not at
40:40
all what I thought it was going to mean.
40:42
How did? You get that where you start and
40:44
where you end up. These are two different things.
40:45
And the name has does not change while the expectation has.
40:49
So there's a. There's a fundamental issue
40:51
there as well. Oh well.
40:53
All right. So I want to shift towards Jason
40:57
and token usage a little bit because a question came through
41:00
about Jason, how verbose it is, how many tokens it uses.
41:06
Do you feel like there's a world and of course this may and is
41:09
probably stepping outside of MCP world potentially, but is there
41:15
a world where Jason is not there or there's something more
41:19
efficient? I, I do not know, I think the so
41:22
there's two things from the on the what is what is MCP using as
41:27
a transport mechanism? And that's independent from what
41:30
you give the model, right? You can take a Jason BLOB and if
41:33
the model accepts something else than Jason, you can transform
41:36
them. As long as it's a bi directional
41:37
mapping, which it should be, it should be fine.
41:41
So what do you get from MCP? Must must not necessarily what
41:44
you give to the model, what you give to the model.
41:47
I do I do not know what the what the most recent research about
41:50
this is. I do think Jason is has a lot of
41:55
benefits. It is actually quite, quite
42:02
compact by itself. Like people forgot the world
42:06
where XML was the big thing that was verbose in comparison.
42:10
So I think Jason's actually quite compact.
42:14
I do think that I'm that I'm usually looking at things about
42:19
less of a like token usage, but like how much tokens did I need
42:23
to get a task done like that I that I wanted to do.
42:28
And I do think like, while tool usage might be now, might be not
42:31
the perfect, most efficient thing to do as, as we do it now,
42:35
I think it's probably very good enough, given that the that the
42:39
that the cost per task done is, is, is decreasing as the models
42:44
are getting better, right? Like, and so I'm less worried
42:48
about it. I don't honestly really see it
42:50
as a, as a massive issue. And I think actually that you
42:55
what, what, what you can foresee a world where instead of like
43:00
using Jason as a tool calling mechanism, you just write code
43:04
that you execute, that the model just writes code that you
43:06
execute. That's another idea.
43:08
And that failed. And then your effective format
43:11
becomes a JavaScript or whatever you might choose.
43:14
And that is also maybe not the most efficient thing, but it is
43:17
the thing that the the model is really pretty good at because it
43:20
turns out there's a lot of Jason in the world.
43:23
And so again, this is a good research question.
43:28
I think it's I did does not feel to me like a particularly high
43:32
priority in the savings you're getting from being smart about
43:35
your context selection, about being smart about the way you
43:40
execute things, the things using tool searches and tool and
43:43
programmatic tool calling. I think those in general are
43:46
probably bigger savings in my mind intuitively than a specific
43:50
formatting questions to be honest.
43:52
And I do like this idea of hey, just because the transport layer
43:55
uses it doesn't mean you can't use something else.
43:57
Oh yeah, yeah, yeah. But this is the thing, right?
43:59
MCP like that's maybe one of the biggest misunderstandings for
44:03
them to be people think is like that's what the model interact
44:05
with. That's not true.
44:06
It's what the application around interact with.
44:09
It's a pure it's a very old school program, a talk to
44:16
program B and some form of information.
44:19
And that information does not need to be the same thing that
44:23
goes into the true calling. It turns out that the format
44:28
that we chose is clearly the same format that like an open AI
44:33
or Anthropic API would expect to do.
44:35
Which is very convenient for people to do, but it does not
44:38
stop you to experiment with other things.
44:42
So anyway. Let's hit on observability
44:46
because this question came through and I, I really liked it
44:49
as far as like as agents start using multiple MCP servers and
44:54
then they're chaining the different tool calls, debugging
44:59
becomes difficult. You had mentioned earlier about
45:01
the ID session that potentially feels like it could be a little
45:05
bit of traces happening, but incorrect tool selection might
45:09
be there. Also, you have MCP server
45:12
failures. Do you see a world where your
45:16
standardized tracing or is this again something that you're like
45:19
that shouldn't be on us, that needs to be something else?
45:24
Two things that first of all, I think in a, in a more general
45:28
tracing mechanics, I don't think that's an MCP problem.
45:31
I think because there are aspects within the execution of
45:36
a of an of a model that are not MCP that you still want to
45:40
trace. And so those would be different
45:42
things. But at the same time, within
45:46
MCP, there's open telemetry standards that he should be
45:50
using. And so there are some of the
45:52
things that that we can do and, and should do.
45:54
But again, they're on the MCP layer.
45:56
And the MCP layer is just part of the general execution of, of
46:01
like a model loop and not the full loop.
46:03
And for that, you know, different model providers offer
46:08
different abilities to introspect this or write trans
46:11
transcriptions or something like that.
46:14
And that really depends. I if I'm not sure if there will
46:16
be an open centered, I don't know if there there should be.
46:18
If anything, I think should be something out of something like
46:21
an open telemetry to some degree.
46:24
I'm not sure if that's enough, if there might be more
46:27
additional information needed. There's always a challenge here,
46:31
particularly when you're dealing with raw unstructured text,
46:33
which effectively models are, is being careful about privacy and
46:40
data retention implications while being useful at the same
46:44
time, right. And the, the most useful tracing
46:49
you can do is trace everything, right?
46:51
That, that, that would of course include things that you probably
46:54
don't want to trace for the sake of your users.
46:57
And so there's trade-offs to be done.
46:59
Again, on the on the MCP side, there's a lot of open telemetry
47:02
work around. The open telemetry people have
47:05
some form of standards around how you trace MCP servers and
47:08
and a lot of the the the typical server and client SDKS do have
47:14
traceability functionality in it.
47:16
On the general thing, I do not know if there should be an open
47:19
standard or not. It's something that's not top of
47:22
my mind. It could be.
47:24
Yeah, it's outside of your scope.
47:26
You you just need to focus on naming properly and that's going
47:31
to get you way farther. So when will MCP be done?
47:38
Is there ever a finished state? How's the vision and mission set
47:42
for that type of thing or is it continuously going to go?
47:46
Do you feel like you're going to be able to put your feet up and
47:50
relax at any point if it gets to certain place?
47:55
I, I think there's, there's a world where the core part of it
47:58
will be kind of done. I, I think it's actually not
48:01
that far away. I think they're the, there's
48:03
things we need to do, but the general core pieces seem to get
48:07
into a reasonable good shape there.
48:09
It feels to me it might relate similar to how you see a lot of
48:17
standards, like for example, our authorization, others happen
48:21
where there will be a a small use case that someone has and
48:26
they will want to build an extension and to the protocol
48:30
and then they will make this a thing.
48:33
And then that's an important thing to do.
48:35
So there's a lot of there's a lot of like in the last 20% of
48:39
use cases, there's a lot there that still needs to be done.
48:43
Those will likely be done during extension.
48:46
And I think that has I do not see that to slow down.
48:48
There's so many open questions like how do you deal around
48:51
guardrails? How do you deal with the
48:53
compliance problems and regulator regulatory problems.
48:57
There's, there's is a extension called Interceptors that is
49:03
proposed by by the financial interest group at the MCP
49:06
project, which is mostly like consists of the Bloomberg, the
49:10
Capital One of the world, who all have very interesting, very
49:15
valid problems. And I think that will these kind
49:19
of senses will continue. There will be, you know, you
49:21
can, you can afford, I do not know, but like, for example, you
49:25
can think about like an extension that will do something
49:27
like A to UI and MCP where you are allowed to do the I
49:32
components that can be rendered in a terminal.
49:35
But there's a lot of these use cases that's awesome that I
49:37
think can happen. But I think they will happen in
49:40
an an extension type of way where the core hopefully should
49:44
be stable. That's the goal.
49:46
Like protocols like MCP thrive from from stability.
49:53
And most importantly, protocols like MCP have a budget of how
49:57
willing the ecosystem is to change.
50:01
And that budget is pretty much directly correlated with the
50:04
amount of attention the project has.
50:06
And the project has a lot of attention in year 1.
50:08
It has more some attention in year 2.
50:11
It will have less attention in Year 3 because it will just be a
50:14
standard part of the stack, and that's fine.
50:16
And that means that people do not want to change the center
50:19
part of the stack because they're thinking about something
50:21
else. Incredible, man.
50:22
Well, this has been great chatting with you.
50:23
I will say that you're going to be giving talks at the MCP Dev
50:28
Summit that we're doing right. You're coming to Japan.
50:31
I'm coming to Japan, I'm coming to Amsterdam, I think the week
50:33
after a few days later and then I'm going to be in October and
50:38
some some Jose at the the big Asian comp.
50:40
Well, it's. Going to be there batting.
50:41
With you, I'm going to be at all of those.
50:43
I'm going to be. I'll try and corral you when
50:45
we're there and. Are you going to go to even
50:47
more? Are you going to go to the the
50:49
Seoul one of the? Nairobi one, I am going to
50:51
Korea. I'm very excited for that
50:53
because I've never been to Korea.
50:54
I'm pretty sad I miss out of Korea.
50:55
I really like, I really like myself some some Korean food and
50:58
like watching some some E sports life and stuff like that.
51:03
I want to watch robots. Literally just saw as I was
51:05
boarding the plane to Lisbon that robot soccer has become
51:09
very popular now because of the World Cup, but Korea's not
51:13
playing anymore in it, so they're enjoying robot football,
51:17
so that's pretty good. Yeah, I've been to Korea before.
51:20
It's a great place, man. I'm I'm hope you're going to
51:22
enjoy it. Yeah, that should be fun and
51:24
we'll hear how people are using MCP over there.
51:25
So that should be fun. And then the big release is
51:29
coming out. So folks, let us know what you
51:31
think and we will. The 28th of of July comes the
51:36
release and then was that the the version true of all the
51:40
SDKS, which I think is the the thing that for developers matter
51:43
the most. Yeah, I really hope that
51:46
everybody's going to upgrade their MCP servers.
51:49
Give us feedback comes to the community.
51:51
We're we're actually I think a quite a friendly bunch.
51:54
We're busy, but we are friendly bunch and trying to listen to
51:57
everyone and understand what they're coming from.
51:59
So bring the problems, bring the ideas.
52:01
A lot of the things that are in this protocol revision are not
52:06
things that came out of my head. There are problems that other
52:08
people brought to us and that we're working on.
52:11
So yeah, bring the back the feedback, update your servers,
52:15
update your clients, and yeah, I'm super, super stoked and see
52:19
where the ecosystem is going then.
52:21
That's the beauty of open source.